An intelligent analysis method for multi-source unstructured data

Abstract

The present invention relates to an intelligent analysis method for multi-source unstructured data, including the following steps: Step S1: Construct an intelligent security acquisition network to obtain power multi-source unstructured data, and perform preliminary data filtering and cleaning processing; Step S2: Build a real-time data pipeline through Apache Kafka, continuously receive data streams from the intelligent security acquisition network, and integrate data extracted from external data sources to ensure data integrity and timeliness; Step S3: Store the multi-source unstructured data obtained by the real-time data pipeline, and use data virtualization technology to achieve seamless integration and unified access to different data sources; Step S4: Use the graph neural network GCN to model the complex network relationships of the integrated multi-source unstructured data to obtain the dynamic relationship characteristics between data; Step S5: Perform intelligent analysis by combining the integrated multi-source unstructured data and the dynamic relationship characteristics between data. The present invention significantly improves the intelligent level and decision support ability of multi-source unstructured data analysis in the power system.

Description

Technical Field

[0001] The present invention relates to the field of unstructured data processing, and particularly to an intelligent analysis method for multi-source unstructured data. Background Art

[0002] With the rapid development of the power industry, the power system has generated a large amount of multi-source unstructured data, including images, texts, time series data captured by sensors, etc. With the improvement of digital transformation, the volume and complexity of these data continue to grow, and traditional data processing and analysis methods are difficult to meet the requirements of real-time and accuracy.

[0003] Moreover, the unstructured data in the power system not only comes from various sources, but also has different structures and storage forms. Traditional centralized databases cannot meet the storage requirements of large-scale and multi-modal data. Moreover, for data analysis, traditional static models are difficult to effectively represent the dynamic relationships between components in the system. For the high complexity and multi-source nature of the data, a more in-depth and complex analysis model is needed to capture the hidden patterns and anomalies in the data. Summary of the Invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide an intelligent analysis method for multi-source unstructured data, which significantly improves the intelligent level and decision-making support ability of multi-source unstructured data analysis in the power system.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions:

[0006] An intelligent analysis method for multi-source unstructured data, comprising the following steps:

[0007] S1: Deploy edge terminals, construct an intelligent security acquisition network, obtain power multi-source unstructured data, and perform preliminary data filtering and cleaning processing;

[0008] S2: Build a real-time data pipeline through Apache Kafka, continuously receive data streams from the intelligent security acquisition network, and integrate data extracted from external data sources to ensure data integrity and timeliness;

[0009] S3: Store the multi-source unstructured data obtained by the real-time data pipeline, store the hot data in the high-performance cache Cassandra, store the cold data in HDFS, automatically adjust according to the access frequency, and use data virtualization technology to achieve seamless integration and unified access of different data sources, improving data query efficiency;

[0010] S4: Use the graph neural network GCN to perform complex network relationship modeling on the integrated multi-source unstructured data to obtain the dynamic relationship characteristics between data;

[0011] S5: Combine the integrated multi-source unstructured data and the dynamic relationship features between the data for intelligent analysis.

[0012] Further, S1 is specifically as follows:

[0013] Use ArcGIS to import geographical and population data, including topography, existing power grid facilities, public infrastructure, and population density; use spatial analysis tools to mark and classify substations, important public infrastructure, and population-dense areas;

[0014] Divide the area into several subnets according to geographical features and population distribution, and each subnet covers the key nodes of a specific area;

[0015] Deploy edge devices according to the data requirements of each subnet; create a multi-level topology structure, with data acquisition devices at the bottom layer, edge gateways in the middle layer, and a central management platform at the top layer;

[0016] Obtain the sensor data of each subnet, and preliminarily process different data types through the edge gateway, including data cleaning and filtering; use Matlab Simulink for device coverage simulation to optimize the location and quantity of edge gateways.

[0017] Further, use Matlab Simulink for device coverage simulation to optimize the location and quantity of edge gateways, specifically as follows:

[0018] Use the objective function F(x) to minimize the number of edge gateways N(x) and maximize the coverage area C(x):

[0019] ;

[0020] L(d)=L0 + 10ηlog10(d / d0)+X g ;

[0021] Among them, α, β are weight coefficients; L(d) is the signal loss at distance d; L0 is the path loss at the reference distance d0; η is the path loss exponent; X g is the attenuation change introduced by the environmental factor; x is the decision variable, that is, the layout scheme of the edge gateway;

[0022] Use the particle swarm optimization algorithm to optimize the selected location of device placement to ensure optimal coverage and the least use of edge gateways:

[0023] Initialize the particle set, where each particle represents an edge gateway layout scheme, and randomly assign it a speed and a position;

[0024] ;

[0025] s x (t + 1)=s x (t)+s x (t+1);

[0026] where: ω is the inertia weight; c1, c2 are the learning factors; r1, r2 are random numbers; p x is the historical local optimal position of particle x; g is the global optimal position; v x (t) and s x (t) are the velocity and position of particle x at the current iteration t respectively;

[0027] Adjust the velocity and position of the particle according to the distance of the particle to the optimal solution:

[0028] Evaluate the fitness of each particle using the objective function F(x), and update the individual optimal solution and the global optimal solution;

[0029] Continuously repeat updating the velocity, position, and fitness until the maximum number of iterations or the convergence condition is reached.

[0030] Furthermore, S2 is specifically:

[0031] Deploy several Kafka brokers to form a distributed cluster to ensure the high availability of the system, and configure ZooKeeper as the coordination and management service of the Kafka cluster;

[0032] Create different topics to classify the data streams received from the intelligent security acquisition network, including sensor_data, video_streams, usage_metrics; and set appropriate partitions and replication factors for each topic to improve data throughput and fault tolerance;

[0033] At the edge gateway, publish the data to the corresponding topic through the Kafka producer API;

[0034] Determine the external data sources to be integrated; develop the corresponding API clients using the HTTP client library;

[0035] Periodically call these APIs and obtain the latest data by setting up scheduled tasks;

[0036] And format and convert the data obtained from the external API to ensure consistency with the internal data;

[0037] Publish the converted external merged data to a suitable topic through a specific Kafka producer, ensuring integration with internal data and data tagging.

[0038] Furthermore, S3 specifically is:

[0039] Store hot data in the high-performance cache Cassandra, build a Cassandra cluster, configure the number of nodes and replication factor to ensure high data availability and load balancing, and optimize the write path to improve write efficiency through Batched Writes and Compaction strategies;

[0040] Store cold data in HDFS, deploy a Hadoop cluster, configure data nodes and NameNode to ensure prominent data redundancy and reliability through replica configuration;

[0041] Monitor data access frequency and migrate data between Cassandra and HDFS according to predefined threshold rules;

[0042] Use data virtualization technology to achieve seamless integration and unified access to different data sources, improving data query efficiency.

[0043] Furthermore, the data virtualization technology uses Denodo to achieve seamless integration and unified access to different data sources as follows: In Denodo, connect Cassandra and HDFS through built-in connectors to ensure that both types of data can be accessed; set connection parameters, including URL, username, and password to ensure security and permission management; and create virtual views in Denodo to integrate data from multiple sources and define logical relationships and mapping rules for data from different data sources to represent the relationships between different data entities, and use the union, join, and subquery functions in Denodo to build cross-source relationships.

[0044] Furthermore, S4 specifically is:

[0045] Take power stations, substations, and sensors in the power grid as nodes, and each node carries different characteristics;

[0046] Define edges based on topological structure, physical proximity, and functional association, and set edge weights;

[0047] Use the power grid topology and related data to construct a weighted undirected graph to form the adjacency matrix A of the power network;

[0048] The initial node feature H(0) includes static features and dynamic features;

[0049] Introduce a temporal convolutional layer and combine it with GCN to handle the prominent changes in power load patterns in time series data:

[0050] Use the temporal convolutional layer to convolve the time series data of each node and extract time-dependent features:

[0051] ;

[0052] Among them, X t is the input time series data, Conv1D is a one-dimensional convolution operation; kernel_size is the size of the convolution kernel; is the extracted time-dependent feature;

[0053] Use GCN to perform spatial aggregation on the extracted time features to capture the complex relationships between nodes:

[0054] ;

[0055] Among them, is the node feature matrix of the l-th layer, is the adjacency matrix with self-loops added; is 's degree matrix; is the inverse square root of the degree matrix; is the weight matrix of the l-th layer; σ is the activation function;

[0056] Use the time-dependent feature as the input and aggregate the time-dependent feature on each node of the graph through GCN to capture the complex network relationships between spaces.

[0057] Furthermore, S5 is specifically:

[0058] The H extracted by GCN represents the dynamic relationship features between nodes; the initial features extracted from images, text, and time series data are F image, F text , F time ;

[0059] Fuse the dynamic relationship features extracted by GCN with other initial features using the attention mechanism method:

[0060] F fused =FusionLayer(F image ,F text ,F time ,H);

[0061] FusionLayer is the fusion operation;

[0062] Based on the fused feature Ffused , select a suitable model according to the required analysis tasks for analysis. The tasks include equipment fault classification, power demand prediction, and identification of abnormal patterns.

[0063] Furthermore, the attention mechanism method is used for feature fusion, specifically as follows:

[0064] Perform a linear transformation on the input feature F to obtain the query Q, key K, and value V vectors. Assume there is a feature space with dimension d k :

[0065]

[0066] where the subscript a ∈ {image, text, time, H}, and image, text, time, and H correspond to image, text, time, and dynamic features respectively; , and are learnable weight matrices;

[0067] Calculate the attention weights using scaled dot-product attention:

[0068] ;

[0069] Concatenate the outputs of all attention heads and perform a linear transformation:

[0070] ;

[0071] where each head is the output of a single attention head, and W O is the linear transformation matrix of the output.

[0072] The present invention has the following beneficial effects:

[0073] 1. By constructing an intelligent security acquisition network, the present invention effectively realizes the intelligent security acquisition and edge processing of power multi-source unstructured data, and uses the particle swarm optimization algorithm to optimize the selected positions of equipment placement, ensuring optimal coverage and the least use of edge gateways, and effectively reducing the number of devices on the premise of ensuring comprehensive acquisition;

[0074] 2. The present invention realizes the efficient storage and management of multi-source unstructured data, combines data virtualization technology for unified access and query, not only increases the flexibility and scalability of the system, but also significantly improves the utilization efficiency of data, realizes fast real-time response and in-depth data analysis; and uses graph neural networks to extract complex network relationships from the integrated multi-source unstructured data, can effectively utilize the complex multi-dimensional data of the power industry, and extract the internal dynamic relationships, thereby supporting more intelligent and efficient power data analysis and decision-making;

[0075] 3. The present invention performs intelligent analysis by combining the integrated multi-source unstructured data and the dynamic relationship characteristics between the data, effectively capturing complex patterns in the power system and helping to make data-driven decisions. Description of the Drawings

[0076] Figure 1 This is the flowchart of the method of the present invention. Detailed Embodiment

[0077] The following further describes the present invention in detail with reference to the drawings and specific embodiments:

[0078] Reference Figure 1 , in this embodiment, an intelligent analysis method for multi-source unstructured data is provided, including the following steps:

[0079] S1: Deploy edge terminals, build an intelligent security acquisition network, obtain power multi-source unstructured data, and perform preliminary data filtering and cleaning processing;

[0080] S2: Build a real-time data pipeline through Apache Kafka, continuously receive data streams from the intelligent security acquisition network, and integrate data extracted from external data sources (such as weather APIs, power market dynamics, etc.) to ensure data integrity and timeliness;

[0081] S3: Store the multi-source unstructured data obtained from the real-time data pipeline, store the hot data in the high-performance cache Cassandra, store the cold data in HDFS, automatically adjust according to the access frequency, and use data virtualization technology (such as Denodo) to achieve seamless integration and unified access of different data sources, improving data query efficiency;

[0082] S4: Use the graph neural network GCN to model the complex network relationships of the integrated multi-source unstructured data and obtain the dynamic relationship characteristics between the data;

[0083] S5: Combine the integrated multi-source unstructured data and the dynamic relationship characteristics between the data for intelligent analysis.

[0084] In this embodiment, S1 is specifically:

[0085] Use ArcGIS to import geographical and population data, including terrain, existing power grid facilities (substations, lines), public infrastructure, and population density; use spatial analysis tools to mark and classify substations, important public infrastructure, and population-dense areas;

[0086] Divide the area into several subnets according to geographical features (such as rivers, mountains) and population distribution, and each subnet covers key nodes in a specific area, such as substations, public infrastructure, and population-dense areas;

[0087] Deploy edge devices according to the data requirements of each subnet (for example, in industrial areas: install high-precision energy consumption monitors and power quality monitoring equipment to obtain detailed electricity consumption data in real time. In residential areas: configure smart meters and environmental monitoring sensors to obtain usage loads and environmental parameters (such as temperature and humidity)); create a multi-level topology structure, with data collection devices at the bottom layer, edge gateways in the middle layer, and a central management platform at the top layer;

[0088] Obtain the sensor data of each subnet, and preliminarily process different data types through the edge gateway, including data cleaning and filtering; select appropriate sensing devices according to the data types, such as video monitoring, environmental data (temperature, humidity), electricity consumption, etc.;

[0089] Use Matlab Simulink to perform device coverage simulation to optimize the location and quantity of edge gateways.

[0090] In this embodiment, use Matlab Simulink to perform device coverage simulation to optimize the location and quantity of edge gateways, specifically as follows:

[0091] Use the objective function F(x) to minimize the number of edge gateways N(x) and maximize the coverage area C(x):

[0092] ;

[0093] L(d)=L0 + 10ηlog10(d / d0)+X g ;

[0094] Where α, β are weight coefficients; L(d) is the signal loss at distance d; L0 is the path loss at the reference distance d0; η is the path loss exponent; X g is the attenuation change introduced by the environmental factor; x is the decision variable, that is, the layout scheme of the edge gateway;

[0095] Use the particle swarm optimization algorithm to optimize the selected location of device placement to ensure optimal coverage and the least use of edge gateways:

[0096] Initialize the particle set, where each particle represents an edge gateway layout scheme, and randomly assign its speed and position;

[0097] ;

[0098] s x (t + 1)=s x (t)+s x (t+1);

[0099] where: ω is the inertia weight; c1, c2 are learning factors; r1, r2 are random numbers; p x is the historical local optimal position of particle x; g is the global optimal position; v x (t) and s x (t) are the velocity and position of particle x at the current iteration t respectively;

[0100] Adjust its velocity and position according to the distance of the particle to the optimal solution:

[0101] Evaluate the fitness of each particle using the objective function F(x), and update the individual optimal solution and the global optimal solution;

[0102] Continuously repeat the update of velocity, position and fitness until the maximum number of iterations or the convergence condition is reached.

[0103] In this embodiment, S2 is specifically:

[0104] Deploy several Kafka brokers to form a distributed cluster to ensure the high availability of the system, and configure ZooKeeper as the coordination and management service of the Kafka cluster;

[0105] Create different topics to classify the data streams received from the intelligent security collection network, including sensor_data, video_streams, usage_metrics; and set appropriate partitions and replication factors for each topic to improve data throughput and fault tolerance;

[0106] At the edge gateway, publish the data to the corresponding topic through the Kafka producer API;

[0107] Determine the external data sources to be integrated, such as weather forecast services (such as OpenWeather API), power market dynamics services, etc.;

[0108] Develop the corresponding API client using the HTTP client library (such as the requests library for Python, HttpClient for Java or axios for Node.js);

[0109] Call these APIs regularly by setting up a scheduled task and obtain the latest data;

[0110] And format and convert the data obtained from the external API to ensure consistency with the internal data;

[0111] Publish the converted external merged data to a suitable topic through a specific Kafka producer, ensuring integration with internal data and ensuring data tagging or association (such as timestamps or locations) for subsequent fusion processing.

[0112] In this embodiment, S3 specifically is:

[0113] Store hot data in the high-performance cache Cassandra, build a Cassandra cluster, configure the number of nodes and replication factor to ensure high data availability and load balancing, and optimize the write path. Through Batched Writes and Compaction strategies, improve the write efficiency;

[0114] Store cold data in HDFS, deploy a Hadoop cluster, configure data nodes and NameNode to ensure outstanding data redundancy and reliability through replica configuration;

[0115] Monitor data access frequency and migrate data between Cassandra and HDFS according to predefined threshold rules;

[0116] Use data virtualization technology to achieve seamless integration and unified access to different data sources, improving data query efficiency.

[0117] In this embodiment, the data virtualization technology adopts Denodo to achieve seamless integration and unified access to different data sources as follows: In Denodo, connect Cassandra and HDFS through built-in connectors to ensure that both types of data can be accessed; set connection parameters, including URL, username, password to ensure security and permission management; and create virtual views in Denodo to integrate data from multiple sources, and define logical relationships and mapping rules for data from different data sources to represent the relationships between different data entities, and use the union, join, and subquery functions in Denodo to build cross-source relationships.

[0118] In this embodiment, S4 specifically is:

[0119] Take power generation stations, substations, and sensors in the power grid as nodes, and each node carries different features (such as real-time voltage, current, power factor, historical electricity consumption);

[0120] The definition of edges is based on topological structure (such as connection lines), physical proximity (such as geographical location), and functional association (such as the same operation and maintenance area), and edge weights are set, such as transmission power loss, geographical distance, etc.;

[0121] Use the power grid topology and related data to build a weighted undirected graph to form the adjacency matrix A of the power network;

[0122] The initial node feature H(0) includes static features (device specifications) and dynamic features (real-time time features);

[0123] Introduce a temporal convolutional layer and combine it with GCN to handle the prominent changes in power load patterns in time series data:

[0124] Use the temporal convolutional layer to perform convolution on the time series data of each node to extract time-dependent features:

[0125] ;

[0126] Among them, X t is the input time series data, Conv1D is a one-dimensional convolution operation; kernel_size is the size of the convolution kernel; is the extracted time-dependent feature;

[0127] Use GCN to perform spatial aggregation on the extracted time features to capture the complex relationships between nodes:

[0128] ;

[0129] Among them, is the node feature matrix of the l-th layer, is the adjacency matrix with self-loops added; is 's degree matrix; is the inverse square root of the degree matrix; is the weight matrix of the l-th layer; σ is the activation function;

[0130] Take the time-dependent feature as the input and aggregate the time-dependent feature on each node of the graph through GCN, so as to capture the complex network relationships between spaces.

[0131] In this embodiment, S5 is specifically:

[0132] The H extracted by GCN represents the dynamic relationship features between nodes; the initial features extracted from images, texts, and time series data are F image, F text , F time ;

[0133] Use the attention mechanism method to fuse the dynamic relationship features extracted by GCN with other initial features:

[0134] F fused = FusionLayer(F image ,F text ,F time ,H);

[0135] The FusionLayer is a fusion operation;

[0136] Based on the fused feature F fused , select a suitable model (such as a deep neural network, random forest, etc.) according to the required analysis tasks for analysis. The tasks include equipment fault classification and power demand prediction; combine autoencoder detection and prediction to identify abnormal patterns.

[0137] In this embodiment, the attention mechanism method is used for feature fusion, specifically as follows:

[0138] Perform a linear transformation on the input feature F to obtain query Q, key K, and value V vectors. Assume there is a feature space with d k dimensions:

[0139]

[0140] where the subscript a ∈ {image, text, time, H}, and image, text, time, and H correspond to image, text, time, and dynamic features respectively; , and are learnable weight matrices;

[0141] Calculate the attention weights using scaled dot-product attention:

[0142] ;

[0143] Concatenate the outputs of all attention heads and perform a linear transformation:

[0144] ;

[0145] where each head is the output of a single attention head, and W O is the linear transformation matrix of the output

[0146] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0147] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device generate means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0148] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0150] As described above, it is only the preferred embodiment of the present invention, and it is not intended to limit the present invention in other forms. Any person skilled in the art may use the disclosed technical content to make changes or modifications into equivalent embodiments with equivalent changes. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical solution content of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. An intelligent analysis method for multi-source unstructured data, characterized in that, It includes the following steps: S1: Deploy edge terminals, build an intelligent security acquisition network, obtain multi-source unstructured power data, and perform preliminary data filtering and cleaning; S2: Build a real-time data pipeline through Apache Kafka, continuously receive data streams from the intelligent security acquisition network, and integrate data extracted from external data sources to ensure data integrity and timeliness; S3: Store the multi-source unstructured data obtained from the real-time data pipeline. Store the hot data in the high-performance cache Cassandra and the cold data in HDFS. Automatically adjust according to the access frequency, and use data virtualization technology to achieve seamless integration and unified access of different data sources, improving data query efficiency; S4: Use the graph neural network GCN to model complex network relationships for the integrated multi-source unstructured data to obtain dynamic relationship features between data; S5: Combine the integrated multi-source unstructured data and the dynamic relationship features between data for intelligent analysis; The specific content of S1 is as follows: Use ArcGIS to import geographical and population data, including topography, existing power grid facilities, public infrastructure, and population density; use spatial analysis tools to mark and classify substations, important public infrastructure, and densely populated areas; Divide the area into several subnets according to geographical features and population distribution, and each subnet covers the key nodes of a specific area; Deploy edge devices according to the data requirements of each subnet; create a multi-level topology structure, with data acquisition devices at the bottom layer, edge gateways in the middle layer, and a central management platform at the top layer; Obtain sensor data from each subnet, and preliminarily process different data types through the edge gateway, including data cleaning and filtering; use Matlab Simulink to perform device coverage simulation to optimize the location and quantity of edge gateways; The specific content of using Matlab Simulink to perform device coverage simulation and optimize the location and quantity of edge gateways is as follows: Use the objective function F(x) to minimize the number of edge gateways N(x) and maximize the coverage area C(x); ； ； where α and β are weight coefficients; L(d) is the signal loss at distance d; L0 is the path loss at the reference distance d0; η is the path loss exponent; X g is the attenuation change introduced by the environmental factor; x is the decision variable, i.e., the layout scheme of the edge gateway; Use the particle swarm optimization algorithm to optimize the selected location of device placement to ensure optimal coverage and the least use of edge gateways; Initialize the particle set, where each particle represents an edge gateway layout scheme, and randomly assign its speed and position; ; ; Where: ω is the inertia weight; c1, c2 are learning factors; r1, r2 are random numbers; p x is the historical local optimal position of particle x; g is the global optimal position; v x (t) and s x (t) are the velocity and position of particle x at the current iteration t respectively; Adjust its speed and position according to the distance of the particle to the optimal solution; Use the objective function F(x) to evaluate the fitness of each particle, and update the individual optimal solution and the global optimal solution; Continuously repeat the update of speed, position, and fitness until the maximum iteration number or convergence condition is reached; The specific content of S5 is as follows: The H extracted by GCN represents the dynamic relationship features between nodes; the initial features extracted from image, text, and time series data are F image, F text , F time ; Fuse the dynamic relationship features extracted by GCN and other initial features using the attention mechanism method; F fused =FusionLayer(F image ,F text ,F time ,H); FusionLayer is the fusion operation; Based on the fused feature F fused , select a suitable model according to the required analysis tasks for analysis. The tasks include equipment fault classification, power demand prediction, and identification of abnormal patterns.

2. An intelligent analysis method for multi-source unstructured data according to claim 1, characterized in that The specific content of S2 is as follows: Deploy several Kafka brokers to form a distributed cluster to ensure high system availability, and configure ZooKeeper as the coordination management service for the Kafka cluster; Create different topics to classify the data streams received from the intelligent security acquisition network, including sensor_data, video_streams, usage_metrics; and set appropriate partitioning and replication factors for each topic to improve data throughput and fault tolerance; At the edge gateway, publish the data to the corresponding topic through the Kafka producer API; Determine the external data sources to be integrated; develop the corresponding API clients using the HTTP client library; Periodically call these APIs and obtain the latest data by setting up scheduled tasks; And format and transform the data obtained from the external APIs to ensure consistency with the internal data; Publish the transformed external merged data to the appropriate topic through a specific Kafka producer to ensure integration with the internal data and ensure data tagging.

3. An intelligent analysis method for multi-source unstructured data according to claim 1, characterized in that The S3 specifically is: Store the hot data in the high-performance cache Cassandra, build a Cassandra cluster, configure the number of nodes and replication factor to ensure high data availability and load balancing, and optimize the write path. Through Batched Writes and Compaction strategies, improve the write efficiency; Store the cold data in HDFS, deploy a Hadoop cluster, configure the data nodes and NameNode to ensure prominent data redundancy and reliability through replica configuration; Monitor the data access frequency and migrate the data between Cassandra and HDFS according to the predefined threshold rules; Use data virtualization technology to achieve seamless integration and unified access of different data sources and improve data query efficiency.

4. An intelligent analysis method for multi-source unstructured data according to claim 3, characterized in that The data virtualization technology adopts Denodo to achieve seamless integration and unified access of different data sources. Specifically, in Denodo, connect Cassandra and HDFS through built-in connectors to ensure that both types of data can be accessed; set connection parameters, including URL, username, password to ensure security and permission management; and create virtual views in Denodo to integrate data from multiple sources, and define logical relationships and mapping rules for data from different data sources to represent the relationships between different data entities, and use the union, join, and subquery functions in Denodo to build cross-source relationships.

5. An intelligent analysis method for multi-source unstructured data according to claim 1, characterized in that The S4 specifically is: Regard the power generation stations, substations, and sensors in the power grid as nodes, and each node carries different characteristics; Define the edges based on the topological structure, physical proximity, and functional association, and set the edge weights; Use the power grid topology and related data to construct a weighted undirected graph to form the adjacency matrix A of the power network; The initial node feature H(0) includes static features and dynamic features; Introduce a temporal convolutional layer and combine it with GCN to process the changing power load patterns prominent in time series data: Use the temporal convolutional layer to perform convolution on the time series data of each node to extract the time dependencies Features: ； Among them, X t is the input time series data, Conv1D is a one-dimensional convolution operation; kernel_size is the size of the convolution kernel; is the extracted time-dependent feature; Use GCN to perform spatial aggregation on the extracted time features to capture the complex relationships between nodes: ； Among them, is the node feature matrix of the l-th layer, is the adjacency matrix with self-loops added; is the degree matrix of; is the inverse square root of the degree matrix; is the weight matrix of the l-th layer; σ is the activation function; Taking the time-dependent features as input, the time-dependent features are aggregated on each node of the graph through GCN, so as to capture the complex network relationships between spaces.

6. An intelligent analysis method for multi-source unstructured data according to claim 1, characterized in that The method of using the attention mechanism for feature fusion is as follows: Perform a linear transformation on the input feature F to obtain the query Q, key K, and value V vectors, assuming a feature space of dimension d k dimensions: ； where the subscript a ∈ {image, text, time, H}, and image, text, time, and H correspond to image, text, time, and dynamic features respectively; , and are learnable weight matrices; Calculating the attention weights using scaled dot-product attention: ； Concatenating the outputs of all attention heads and performing a linear transformation: ； where each head is the output of a single attention head, and W O is the linear transformation matrix of the output.

Images