Big data analysis and processing method and system based on deep learning

Abstract

The invention provides a big data analysis and processing method and system based on deep learning. The method comprises the following steps: collecting and processing real-time data streams containing structured data and unstructured data from a plurality of heterogeneous data sources to obtain a data stream set; performing instant mode recognition processing on the data stream set to obtain behavior mode information; based on the behavior pattern information, constructing a dynamic data graph capable of reflecting relevance between different data nodes; analyzing and processing the relationship between the nodes and the edges in the dynamic data graph to obtain influence factors among the nodes and predict a future trend; generating decision support information according to the influence factor and the prediction result; and feeding back decision support information to a real-time data source to form a closed-loop control mechanism. According to the technical scheme provided by the invention, data from different sources can be rapidly integrated, behavior mode information is identified, trend prediction is carried out based on the behavior mode information, and finally, timely and effective decision support is provided for services.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data analysis technology, and in particular, to a big data analysis and processing method and system based on deep learning. Background Art

[0002] With the rapid development of technologies such as the Internet and the Internet of Things, enterprises generate a large amount of data every day. These data include not only structured database records, but also unstructured information such as pictures, videos, and texts. For industries such as finance, retail, and manufacturing, how to extract valuable information from these massive and diverse data to support real-time decision-making has become a key challenge.

[0003] At present, there are some technical solutions for big data analysis and processing on the market, mainly including traditional statistical analysis methods and more advanced machine learning algorithms. In recent years, with the advancement of deep learning technology, many big data analysis tools and service platforms based on deep neural networks have emerged. Such systems usually use distributed computing frameworks to process large-scale data sets and use complex models such as convolutional neural networks (CNN) or recurrent neural networks (RNN) for feature extraction and pattern recognition.

[0004] Although existing big data analysis solutions have met the needs of enterprises to a certain extent, most existing systems are mainly designed for specific types of data (such as images or time series) and are difficult to flexibly deal with mixed data streams in multiple formats. Especially in the face of application scenarios with high real-time requirements, traditional methods have slow response speeds due to large amounts of calculations and cannot achieve instant feedback. Summary of the invention

[0005] The embodiments of the present application provide a big data analysis and processing method and system based on deep learning, so as to solve the problem of poor data analysis and processing effect in the prior art.

[0006] In a first aspect, the present application provides a method for analyzing and processing big data based on deep learning, including: Using a real-time data collection system, real-time data streams containing structured data and unstructured data from multiple heterogeneous data sources are collected and processed to obtain a data stream set; According to the distributed computing framework, real-time pattern recognition processing is performed on the data stream set to obtain behavior pattern information; Based on the behavior pattern information, a dynamic data graph capable of reflecting the correlation between different data nodes is constructed; Using a graph neural network model, the relationship between nodes and edges in the dynamic data graph is analyzed and processed to obtain the influence factors between nodes and predict future trends; generating decision support information based on the influencing factors and prediction results; The decision support information is fed back to the real-time data source to form a closed-loop control mechanism.

[0007] In a second aspect, the embodiment of the present application provides a big data analysis and processing system based on deep learning, including: A collection module is used to collect and process real-time data streams containing structured data and unstructured data from multiple heterogeneous data sources using a real-time data collection system to obtain a data stream set; An identification module, used for performing instant pattern recognition processing on the data stream set according to a distributed computing framework to obtain behavior pattern information; A construction module, used to construct a dynamic data graph that can reflect the correlation between different data nodes based on the behavior pattern information; An analysis module is used to analyze and process the relationship between nodes and edges in the dynamic data graph using a graph neural network model, obtain the influence factors between nodes and predict future trends; A generation module, used to generate decision support information according to the influencing factors and prediction results; The feedback module is used to feed back the decision support information to the real-time data source to form a closed-loop control mechanism.

[0008] In a third aspect, an embodiment of the present application provides a computing device, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a big data analysis and processing method based on deep learning as described in any one of the first aspects above.

[0009] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements a big data analysis and processing method based on deep learning as described in any one of the first aspects above.

[0010] In an embodiment of the present application, real-time data streams containing structured data and unstructured data from multiple heterogeneous data sources are collected and processed to obtain a data stream set; based on a distributed computing framework, the data stream set is subjected to instant pattern recognition processing to obtain behavior pattern information; based on the behavior pattern information, a dynamic data graph that can reflect the correlation between different data nodes is constructed; using a graph neural network model, the relationship between nodes and edges in the dynamic data graph is analyzed and processed to obtain the influencing factors between each node and predict future trends; based on the influencing factors and prediction results, decision support information is generated; the decision support information is fed back to the real-time data source to form a closed-loop control mechanism. The technical solution provided by the present application can quickly integrate data from different sources, identify behavior pattern information, and perform trend prediction based on this, ultimately providing timely and effective decision support for the business.

[0011] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of a big data analysis and processing method based on deep learning provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a big data analysis and processing system based on deep learning provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0015] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0017] Figure 1 A flowchart of a big data analysis and processing method based on deep learning is provided for an embodiment of the present application, such as Figure 1 As shown, the method includes: 101. Using a real-time data collection system, collect and process real-time data streams containing structured data and unstructured data from multiple heterogeneous data sources to obtain a data stream set; In this step, various sensors and other data collection devices (such as smart meters) are deployed to obtain real-time operation data from power plants, transmission lines, substations, etc. These data include structured information such as power production and voltage levels, and may also include unstructured data such as video surveillance images.

[0018] In the embodiment of the present application, State Grid Corporation uses advanced Internet of Things technology to connect smart devices across the country to achieve comprehensive monitoring of the power grid status.

[0019] 102. Performing real-time pattern recognition processing on the data stream set according to the distributed computing framework to obtain behavior pattern information; In this step, based on distributed computing platforms such as Hadoop or Spark, large amounts of incoming data streams are quickly processed, and machine learning algorithms are used to automatically detect abnormal operating modes or signs of potential failures.

[0020] In the embodiment of the present application, by learning from historical data, the system can quickly discover possible overload conditions in certain areas of the power grid and provide early warning.

[0021] This application takes into account that in the existing real-time data analysis technology, although it can process large-scale data streams relatively effectively, in the field of energy and power, especially in complex systems like the State Grid, there are problems such as a large amount of noise and missing values ​​in the original data due to the diverse data sources, which directly affects the accuracy of subsequent analysis. The stable operation of the power system depends on the rapid response to abnormal situations, so it is necessary to be able to complete the process from data collection to decision output in near real time. With the continuous expansion of the scale of the power grid and the increase in the proportion of new energy access, traditional methods are difficult to effectively capture the various complex patterns hidden behind the massive data. In order to solve the above problems, a more refined data processing process is proposed, aiming to improve the analysis accuracy and response speed through multi-level data cleaning and pattern mining.

[0022] The options are as follows: Optionally, the step 102 of "performing instant pattern recognition processing on the data stream set according to the distributed computing framework to obtain behavior pattern information" includes: cleaning and standardizing the data in the data stream set to obtain preprocessed data with noise eliminated and unified data format; utilizing a streaming processing engine to perform real-time analysis on the preprocessed data to obtain a preliminary feature representation; performing pattern mining processing on the preliminary feature representation to obtain pattern features; and based on the pattern features, performing high-level abstract processing on the pattern features to obtain behavior pattern information.

[0023] Cleaning: refers to the process of removing noise, redundant information and outliers from data.

[0024] Standardization: Converting data into a uniform format or dimension to facilitate comparison and analysis.

[0025] Stream processing engine: A technology designed specifically to process data that continuously flows into a system, such as Apache Flink or Spark Streaming.

[0026] Pattern mining: discovering regularities or recurring phenomena from large amounts of data.

[0027] High-level abstract processing: Further refinement and generalization of identified basic patterns to form a higher level of understanding.

[0028] First, clean the invalid entries in the data, fill in missing values, and correct obvious errors. Convert all data to the same format to ensure compatibility and consistency. Then use an efficient stream computing framework to process the cleaned data online. Extract key indicators and generate preliminary feature vector representations. Secondly, apply machine learning algorithms (such as clustering and classification) to explore the intrinsic connections between data. Identify representative behavior patterns and their related attributes. Finally, use domain knowledge or other advanced algorithms to comprehensively summarize the identified pattern features. Form behavior pattern information that can reflect complex business logic.

[0029] In the embodiment of the present application, it is assumed that a certain city power grid often encounters power shortage in some areas during the peak period in summer. By implementing the above process: First, we collected the city's electricity consumption records during the past five summers and cleaned them up as necessary. Secondly, we used the Flink stream processing framework to monitor the daily electricity consumption changes in the current season in real time. We further applied the FP-growth algorithm to find the pattern that "when the temperature exceeds 35°C and the humidity is greater than 78%, the electricity consumption in Area A surges between 8pm and 10pm." Finally, we used the LSTM model to predict that the potential electricity demand growth in Area A under similar conditions in the next week is about 15%.

[0030] Through such a complete data analysis process, not only can the factors that may lead to power supply shortages be accurately identified, but corresponding adjustments can also be made in advance, thereby greatly reducing the risk of power outages and improving the overall reliability and service level of the power grid. This directly responds to the data quality issues mentioned initially and the need for efficient pattern recognition.

[0031] 103. Based on the behavior pattern information, construct a dynamic data graph that can reflect the correlation between different data nodes; In this step, a visual network diagram reflecting the interaction between various power facilities is established based on the identified behavior patterns, which facilitates intuitive understanding of the operation status of the entire system.

[0032] In an embodiment of the present application, a map view covering the key nodes of the entire network is created to display the direction and intensity of energy flow between the various parts, helping managers to better plan resource allocation.

[0033] This application takes into account that in the field of energy and power, especially in large and complex systems like the State Grid, traditional static data analysis methods often fail to fully capture the dynamic interactions between the various components within the system. In order to overcome these problems, a method for constructing a dynamic data map is proposed, which can not only intuitively show the correlation between various entities, but also dynamically update these relationships over time, thereby providing deeper insights.

[0034] The options are as follows: Optionally, the "constructing a dynamic data graph that can reflect the correlation between different data nodes based on the behavior pattern information" in step 103 includes: performing entity object extraction processing on the behavior pattern information to obtain entity objects as nodes, and assigning a unique identifier to each node; identifying the relationship between the entity objects to obtain edges connecting different nodes to reflect the correlation between entities; recording the time series changes of the relationship between the nodes to obtain time series information reflecting the time dimension of the dynamic data graph; analyzing the importance and group characteristics of the nodes to obtain feature information that enhances the expression of the correlation between nodes in the dynamic data graph; constructing a dynamic data graph based on the entity objects of the nodes, the edges connecting different nodes, the time series information and the feature information of the expression of the correlation between nodes.

[0035] Entity object: refers to the basic elements in the data graph, which can represent specific individuals or things such as people, equipment, and events.

[0036] Identifier: A number or code used to uniquely identify each physical object.

[0037] Edge: A line segment connecting two nodes, indicating that some type of relationship exists between them.

[0038] Time series information: records the changes in the relationship between nodes over time, reflecting the dynamic nature of the graph.

[0039] Importance: A measure of the relative influence or criticality of a node in the entire network structure.

[0040] Group characteristics: describe the characteristics or attributes shared by a group of related nodes.

[0041] First, extract meaningful objects from the behavior pattern information as nodes in the graph, and assign a unique ID to each node. Analyze and determine the interaction between different entities, and establish corresponding links (edges) to display these relationships. Then track and record the state changes of each edge, including the time points of creation, update or deletion, to form continuous time series data. Secondly, use algorithms such as PageRank to evaluate the importance of each node; at the same time, clustering or other statistical methods are used to find groups of nodes with similar characteristics. Finally, combine all the above information to create a data graph model that contains both static structure and dynamic evolution.

[0042] In the embodiment of the present application, it is assumed that the power dispatching system of a certain province is to be optimized. In this example, behavioral data of all key facilities including power plants, substations, and transmission lines are collected from across the province. Key entities are extracted from these data through text mining technology and assigned unique ID numbers. Secondly, the electrical connections between these entities are analyzed to clarify which facilities have direct energy transmission paths. The time of occurrence of each energy flow event is marked with a timestamp to form a detailed interaction history. Further application of the PageRank algorithm found that several large substations located in the center of the network play a decisive role in the stability of the entire system. Finally, based on the above information, a dynamic data map is constructed, which not only shows the location and connection mode of each node in the current state, but also reflects the changes under different seasons and weather conditions.

[0043] By constructing such a dynamic data map, when an abnormality occurs in the power grid, the most affected area can be quickly identified and the path of problem propagation can be tracked. Secondly, maintenance plans can be reasonably arranged according to the importance of each node, giving priority to the safe operation of core facilities. By learning from historical data, possible risk points in the future can be predicted and preventive measures can be taken in advance. Finally, based on the development trend shown in the map, a scientific basis can be provided for future power grid expansion.

[0044] 104. Analyze and process the relationship between nodes and edges in the dynamic data graph using a graph neural network model to obtain the influence factors between nodes and predict future trends; In this step, graph neural network technology is used to deeply explore the complex connections between different components, quantify the influence between them, and predict possible changes in the future.

[0045] In the embodiment of the present application, by analyzing the historical interaction data of multiple substations in a specific area, it is predicted where maintenance and upgrades are most likely to be required in the next few months.

[0046] This application takes into account that in the field of energy and power, especially for complex systems like the State Grid, traditional data analysis methods often have difficulty capturing the complex dynamic relationships between nodes within the system (such as power stations, substations, transmission lines, etc.), as well as the trends of these relationships over time. In order to solve the above problems, a method is proposed to use a graph neural network (GNN) model to analyze dynamic data graphs and combine it with a recurrent neural network (RNN) for trend prediction. This method can better understand and predict the complex relationships between nodes in the power network and their trends over time.

[0047] The options are as follows: Optionally, in step 104, "the use of the graph neural network model to analyze and process the relationship between the nodes and edges in the dynamic data graph, obtain the influencing factors between the nodes and predict future trends" includes: using the dynamic data graph as input data, and defining the weight matrix of the nodes and edges in the dynamic data graph to obtain the graph neural network model; in each layer of the graph neural network model, exchanging and processing the information of the nodes in the dynamic data graph and the adjacent nodes of the nodes to update the feature representation of each node to obtain the updated node features; based on the updated node features, applying a pooling operation to the updated node features Aggregation processing is performed to obtain key information with reduced dimensionality; based on the key information with reduced dimensionality, a readout function is used in the last layer of the graph neural network model to summarize the final feature representations of all nodes to obtain a global representation of the dynamic data graph; using the global representation, the influencing factors between the nodes in the dynamic data graph are analyzed and processed through a regression algorithm to obtain influencing factors that quantify the degree of interaction between the nodes; based on the influencing factors, a graph neural network model extended by a recurrent neural network is used to predict the changing trend of the dynamic data graph to obtain a prediction result, which includes the future node states and their mutual influence.

[0048] Weight matrix: A matrix of numbers that describes the strength of connections between nodes in a graph.

[0049] Information exchange processing: refers to the message passing mechanism in graph neural networks, which allows adjacent nodes to share information and update their own feature representations.

[0050] Pooling: A dimensionality reduction technique used to reduce the number of features while retaining important information.

[0051] Readout function: used to extract all node features from the last layer of the graph neural network to form a global representation.

[0052] Regression algorithm: A statistical method used to estimate the relationship between two variables.

[0053] Graph neural network model extended by recurrent neural network: It combines the characteristics of graph neural network and recurrent neural network and is suitable for processing data with time dimension.

[0054] First, a weight value is assigned to each edge in the dynamic data graph. These weights reflect the strength of the association between nodes and are organized into a matrix form for use by the graph neural network. Secondly, in each layer of the graph neural network, each node exchanges information with its neighboring nodes, and then updates its own feature vector based on the received information. Then, a pooling operation, such as maximum pooling or average pooling, is applied to the node features after multiple rounds of information exchange to reduce the feature dimension and extract key information. The final feature representations of all nodes are further integrated using readout functions (such as summation, mean, etc.) to obtain a global representation of the entire graph. Furthermore, based on the global representation, a regression algorithm is used to calculate the degree of interaction between nodes, that is, the influence factor, which can be achieved through linear regression or other appropriate regression models. Finally, a graph neural network model that extends the recurrent neural network is used to predict the future node status and its mutual influence based on historical data. Here, the RNN part is particularly suitable for capturing temporal dependencies.

[0055] In the embodiment of the present application, it is assumed that the power system in a certain area needs to be optimized, especially focusing on the stability of the power grid under extreme weather conditions (such as heat waves). In this example, a dynamic data map containing nodes such as power stations, substations, and important transmission lines is constructed from historical data. By analyzing historical electricity consumption data, a weight matrix between nodes is defined, where the weight reflects the energy transmission intensity between nodes under different weather conditions. In the GNN model, each node exchanges information with its neighboring nodes and gradually updates the feature representation of the node, such as the load increase of certain nodes under high temperature conditions. Through the pooling operation, the dimension of the feature vector is reduced, making the model more concise and efficient. The global representation of the entire power grid is obtained using the readout function. Applying the linear regression algorithm, the influencing factors between the nodes are calculated, and it is found that some substations have a greater impact on the stability of the power grid under high temperature conditions. Combined with RNN, the load distribution of each node in the power grid and possible fault points under similar high temperature conditions in the future are predicted.

[0056] Through the application of the above methods, the operating status of the power grid under different environmental conditions can be predicted more accurately, and measures can be taken in advance to avoid potential failures. Based on the influencing factors between nodes, resources can be dispatched more reasonably to improve the overall efficiency and reliability of the power grid. The model has a strong time series processing capability, which can better adapt to the changing external environment and provide strong support for long-term planning.

[0057] 105. Generate decision support information based on the influencing factors and prediction results; In this step, based on the results of the previous steps and taking into account factors such as economic benefits and safety, specific optimization suggestions or emergency measures are proposed.

[0058] In an embodiment of the present application, a specific guidance plan is provided to the dispatch center on how to adjust the power supply strategy to cope with the upcoming demand peak.

[0059] 106. Feedback the decision support information to the real-time data source to form a closed-loop control mechanism.

[0060] In this step, the above analysis results are promptly sent back to the front-end equipment to guide on-site staff to take action. At the same time, new operation data will continue to be collected to continuously iterate and improve the entire process.

[0061] In the embodiment of the present application, once a sudden failure occurs somewhere, the central control system will immediately notify the nearest technical team to handle it, and simultaneously update the information of all relevant parties to ensure rapid restoration of services.

[0062] This application takes into account that in the field of energy and power, especially for large-scale complex systems such as the State Grid, the existing technology has the problem that traditional feature engineering methods may not be able to fully capture the complex interaction patterns between nodes when analyzing the degree of interaction between nodes (such as power plants, substations, etc.). Models established using fixed rules or simple statistical methods are often difficult to adapt to new or unseen data situations. For dynamically changing power grid structures, static models cannot well reflect the changes in the relationship between nodes over time, resulting in inaccurate prediction results. Therefore, in order to overcome these problems, an improved solution is proposed, that is, after generating a global representation through a graph neural network (GNN), a regression algorithm is used to further refine the degree of interaction between nodes. This not only improves the depth of understanding of complex network relationships, but also enhances the model's ability to predict future situations.

[0063] The options are as follows: Optionally, the global representation is used to analyze and process the influencing factors between the nodes in the dynamic data graph through a regression algorithm to obtain influencing factors that quantify the degree of interaction between the nodes, including: using the global representation generated by the graph neural network model to perform feature extraction on the relationship between the nodes in the dynamic data graph to obtain a global feature vector containing information on the interaction between the nodes; applying a regression algorithm to train the global feature vector to obtain a regression model that quantifies the degree of interaction between the nodes; based on the trained regression model, the newly acquired global feature vector is predicted to obtain a quantitative value or category label of the degree of interaction between the nodes as an influencing factor.

[0064] Global feature vector: A high-dimensional vector that represents the state of the entire dynamic data graph after being processed by the graph neural network.

[0065] Regression model: A statistical model used to predict continuous value outputs, such as linear regression, decision tree regression, etc.

[0066] Impact factor: A numerical indicator that measures the strength or nature of the interaction between nodes. It can be a specific quantitative value or a category label.

[0067] First, use additional feature extraction layers or techniques (such as autoencoders) to extract higher-level features from the global representation, which can better reflect the interaction patterns between nodes. Then use known influencing factors as labels and use appropriate regression algorithms (such as random forest regression, support vector regression, etc.) to train the model, using the global feature vector as input to learn how to accurately predict the degree of interaction between nodes. Finally, use the trained regression model to predict the global feature vector of new data and output the specific quantitative value or classification label of the degree of interaction between nodes.

[0068] In the embodiment of the present application, it is assumed that a large power network covering multiple cities is being studied. During the peak summer season, some substations frequently have problems due to overload. In order to improve this situation, the following steps are attempted: First, a GNN model was built to learn the relationship between nodes in the entire power network. Each node represents a substation or power plant, and the edges reflect information such as the power transmission lines and capacity constraints between them. Secondly, a global feature vector describing the state of the entire power network was obtained through the trained GNN model. This vector contains information about which areas are prone to overload and which paths are the most critical power transmission routes. Next, a regression algorithm (such as random forest regression) was selected and trained using a historical data set, with the goal of building a model that can convert the global feature vector into a prediction value of the intensity of interaction between specific substations. Finally, once the model is trained, it can be applied to predict possible problems in the future. For example, if a region is expected to experience extreme high temperature weather, the corresponding global feature vector can be input to estimate which substations are most likely to be affected and how the intensity of their interactions will change.

[0069] By introducing the concept of global eigenvectors, we can understand the working mechanism of the entire power network at a higher level. Combined with the regression model, it can not only more accurately predict the degree of interaction between nodes, but also help identify potential risk points in advance. Based on more accurate prediction results, power companies can allocate resources more effectively, ensure stable operation of the power grid, and reduce the probability of power outages.

[0070] This application takes into account that in the field of energy and power, especially in large-scale complex systems such as the State Grid, existing prediction methods often face the fact that the states of nodes (such as power stations, substations, etc.) in the power system and their mutual influences change over time and environmental conditions, and traditional static models are difficult to accurately capture this dynamic nature. The operating state of the power system is affected by many factors, including weather, power demand, equipment status, etc. There are complex time series dependencies between these factors, and existing methods may not be able to effectively handle such long sequence dependencies. Most prediction models only provide prediction results without evaluating the credibility of the prediction results, which may lead to decision-making errors in practical applications. In order to overcome the above problems, this application proposes a scheme based on influencing factors and a graph neural network (GNN) model extended by a recurrent neural network (RNN) to predict future node states and their mutual influences. This method can better process time series data and evaluate the credibility of prediction results, thereby improving the accuracy and reliability of predictions.

[0071] The options are as follows: Optionally, based on the influencing factors, a graph neural network model extended by a recurrent neural network is used to predict the changing trend of the historical dynamic data graph to obtain the states of future nodes and their mutual influence, including: based on the influencing factors, feature extraction is performed on the node states in the dynamic data graph and their mutual influence to obtain time series features; using the graph neural network model extended by a recurrent neural network, dynamic evolution simulation is performed on the time series features to obtain a dynamic evolution model; according to the dynamic evolution model, the dynamic data graph at future time points is predicted to obtain the states of future nodes and their mutual influence; the states of future nodes and their mutual influence are evaluated to obtain a credibility index of the prediction result, and the parameters of the graph neural network model extended by the recurrent neural network are adjusted as needed; and the prediction result is output.

[0072] Time series features: A series of data points extracted from historical data and arranged in chronological order, reflecting the changes of a variable over time.

[0073] Dynamic evolution model: A mathematical model that describes how the state of a system changes over time.

[0074] Credibility index: A quantitative indicator used to measure the reliability of prediction results, such as confidence interval, error rate, etc.

[0075] First, the time series features of node states and their mutual influence are extracted from historical data. These features will be provided as input to subsequent models for learning. Secondly, the time series features are processed using a model that combines RNN and GNN to simulate the evolution of node states and their interactions over time to form a dynamic evolution model. Based on the constructed dynamic evolution model, the node states and their mutual influence at future time points are predicted. The prediction results are further evaluated, and credibility indicators such as mean square error (MSE) and mean absolute error (MAE) are calculated. The model parameters are adjusted according to the evaluation results to optimize the model performance. Finally, the evaluated and optimized prediction results are output, including the states of future nodes and their mutual influence.

[0076] In the embodiment of the present application, it is assumed that the load conditions of each substation in a certain area in the next week and their mutual influence are predicted, especially in the upcoming heavy rainfall weather. First, the load conditions of each substation under different weather conditions are extracted from the power data of the past year, and the influencing factors are calculated. These data will be used to form time series features. Secondly, a RNN-extended GNN model is constructed, in which the GNN part learns the topological structure between substations, and the RNN part learns the time series pattern of load changes. By training the model, it can learn the relationship between load changes and weather conditions. Then, using the trained model, the upcoming heavy rainfall weather forecast data is input to predict the load changes of each substation in the next week, as well as the possibility of load transfer between substations. Further, by comparing the model prediction value with the actual load changes under similar weather conditions in the historical data, the error of the prediction result is calculated, and the credibility score of the prediction result is given. If the error is large, it means that the credibility of the prediction result is low. If the credibility of the prediction result is low, the parameters of the RNN-GNN model can be adjusted, such as increasing the number of training rounds, adjusting the learning rate, etc., to improve the prediction accuracy of the model. The final output is a forecast report, which includes the expected load changes of each substation in the next week and the possibility of load transfer between substations. The report also indicates the credibility score of the forecast results for the reference of dispatchers.

[0077] By combining the advantages of GNN and RNN, the dynamic changes of node states in the power system can be captured more accurately, thereby improving the accuracy of predictions. RNN can handle long sequence dependencies, allowing the model to better adapt to changes under different conditions and enhancing the generalization ability of the model. The prediction results are accompanied by credibility indicators, which can help decision makers better understand the reliability of the prediction results and make more informed decisions. Based on accurate prediction results, power companies can make scheduling plans in advance, optimize resource allocation, and ensure the stable operation of the power grid under extreme weather conditions.

[0078] This application takes into account that in the field of energy and power, especially for large-scale complex systems such as the State Grid, the existing technology has problems when processing graph structured data. Traditional graph neural network (GNN) methods usually use fixed aggregation strategies, such as average or maximum pooling, to merge the information of neighbor nodes. However, these methods ignore the actual situation that the importance of different neighbor nodes to the central node may be different. Standard GNN models often lack sufficient flexibility when dealing with different types of nodes and edges, and cannot adapt to the complex dependencies between nodes in the power system. In multi-layer GNNs, as the number of layers increases, the characteristics of the nodes may gradually converge, resulting in local information loss and affecting the performance of the model. In order to solve these problems, this application proposes a graph neural network model (Graph Attention Networks, GAT) based on an attention mechanism. By introducing the attention mechanism, different weights can be dynamically assigned to each neighbor node, so as to better capture the complex relationship between nodes and maintain the diversity of node features.

[0079] The options are as follows: Optionally, in each layer of the graph neural network model, nodes in the dynamic data graph and information of adjacent nodes to the nodes are exchanged to update feature representations of each node to obtain updated node features, including: Calculate the nodes in the dynamic data graph and with nodes Neighboring nodes The attention weight between : ; in, Is a node and nodes The attention weights between is the attention coefficient vector, Represents the concatenation operation of node feature vectors, that is, connecting two node feature vectors together to form a new vector. Is an activation function used to increase the nonlinearity in the graph neural network model. Represents the weight matrix, which is used to linearly transform node features. Applied to The node feature vector of the layer, and Respectively represent Node in layer and nodes The node feature vector of Representation Node The neighbor set of all nodes The set of adjacent nodes. Indicates Node in layer The characteristic vector of Indicates Node in layer The eigenvector of For nodes in the dynamic data graph and with nodes Neighboring nodes The information is exchanged to obtain the updated node The node feature vector of : ; in, Is a node exist The updated node feature vector of the layer, and They are The weight matrix and bias vector of the layer, is the activation function, Is a node exist The node feature vector of the layer, Is a node exist The node feature vector of the layer.

[0080] Attention weight: A value that measures the degree of attention a node pays to its neighboring nodes, reflecting the importance of the neighboring nodes' influence on the current node.

[0081] Attention coefficient vector: parameter vector used to calculate attention weights.

[0082] LeakyReLU: An activation function that solves the problem of complete suppression of the standard ReLU function in the negative interval and adds nonlinear characteristics.

[0083] Weight matrix: A matrix used to linearly transform node features.

[0084] Bias vector: A constant vector added to the weighted sum to increase the flexibility of the model.

[0085] Activation function: a function used to increase the nonlinearity of the model, such as Sigmoid, Tanh, ReLU, etc.

[0086] First, the attention mechanism is used to calculate the attention weight between node v and its neighbor node u This process involves concatenating the feature vectors of the two nodes and passing them through a learnable attention coefficient vector The LeakyReLU activation function is used for processing, and finally the attention weight of each neighbor node is normalized. , weighted sum of the feature vectors of neighboring nodes, combined with the feature vector of the current node itself, after linear transformation and activation function processing, the updated feature vector of node v in the next layer is obtained .

[0087] In the embodiment of the present application, it is assumed that the load conditions of each substation in a regional power network and their mutual influence are to be analyzed, especially during the high temperature period in summer.

[0088] Each substation is regarded as a node, and the power transmission lines between substations are regarded as edges to construct a graph structure. The initial feature vector of each node may include information such as the capacity of the substation and historical load data.

[0089] Assume there are two substations A and B, and an attention coefficient vector .

[0090] Assume that the feature vectors of nodes A and B are and , the weight matrix .

[0091] Calculating attention weights and : ; Assume the weight matrix , the bias vector , activation function .

[0092] Update the feature vector of node A: ; Through the attention mechanism, weights can be dynamically assigned according to the importance of neighbor nodes, making information transfer more efficient. The attention mechanism enables the model to flexibly adapt to the relationship between different types of nodes and edges, and better capture the complex dependencies in the power system. And in multi-layer GNNs, the attention mechanism helps to maintain the diversity of node features and prevent feature convergence, thereby improving the performance of the model.

[0093] LeakyReLU (LeakyRectifiedLinearUnit) is an improved activation function designed to solve the problem that the standard ReLU (RectifiedLinearUnit) completely suppresses the output when the input is negative. LeakyReLU allows a small amount of negative axis information to pass through, thereby maintaining the activity of the neuron and helping to alleviate the "dead ReLU" problem, that is, some neurons may always output zero, causing the gradient to disappear.

[0094] The specific expression of LeakyReLU is as follows: ; in, is the input value, is a positive number less than 1, usually set to 0.01. This means that when the input is positive, LeakyReLU behaves the same as the standard ReLU; when the input is negative, it does not completely zero the output, but multiplies it by a smaller slope .

[0095] In a neural network, Usually refers to the activation function. In different contexts, It can refer to many different activation functions, but the most common one is the Sigmoid function. The expression of the Sigmoid function is as follows: ; The Sigmoid function maps real numbers to Within the range, its output can be interpreted as a probability value. It has the characteristics of smoothness and derivability, and is suitable for situations where probability output is required, such as binary classification problems.

[0096] Figure 2 A structural diagram of a big data analysis and processing system based on deep learning is provided for the embodiment of the present application, such as Figure 2 As shown, the device comprises: The collection module 21 is used to collect and process real-time data streams containing structured data and unstructured data from multiple heterogeneous data sources using a real-time data collection system to obtain a data stream set; The identification module 22 is used to perform real-time pattern recognition processing on the data stream set according to the distributed computing framework to obtain behavior pattern information; A construction module 23, configured to construct a dynamic data graph capable of reflecting the correlation between different data nodes based on the behavior pattern information; An analysis module 24 is used to analyze and process the relationship between nodes and edges in the dynamic data graph using a graph neural network model, obtain the influence factors between nodes and predict future trends; A generating module 25, for generating decision support information according to the influencing factors and the prediction results; The feedback module 26 is used to feed back the decision support information to the real-time data source to form a closed-loop control mechanism.

[0097] Figure 2 The big data analysis and processing system based on deep learning can be executed Figure 1 The implementation principle and technical effect of the big data analysis and processing method based on deep learning described in the illustrated embodiment will not be repeated. The specific manner in which each module and unit performs operations in the big data analysis and processing system based on deep learning in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0098] In one possible design, Figure 2 A big data analysis and processing system based on deep learning in the illustrated embodiment can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32; The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0099] The processing component 32 is used to: collect and process real-time data streams containing structured data and unstructured data from multiple heterogeneous data sources to obtain a data stream set; perform real-time pattern recognition processing on the data stream set according to the distributed computing framework to obtain behavior pattern information; based on the behavior pattern information, construct a dynamic data map that can reflect the correlation between different data nodes; use the graph neural network model to analyze and process the relationship between nodes and edges in the dynamic data map to obtain the influencing factors between each node and predict future trends; generate decision support information based on the influencing factors and prediction results; and feed back the decision support information to the real-time data source to form a closed-loop control mechanism. Among them, the processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component can also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above method.

[0100] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0101] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0102] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0103] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0104] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0105] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 A big data analysis and processing method based on deep learning in the illustrated embodiment.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0107] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A big data analysis and processing method based on deep learning, characterized in that: include: Using a real-time data collection system, real-time data streams containing structured data and unstructured data from multiple heterogeneous data sources are collected and processed to obtain a data stream set; According to the distributed computing framework, real-time pattern recognition processing is performed on the data stream set to obtain behavior pattern information; Based on the behavior pattern information, a dynamic data graph capable of reflecting the correlation between different data nodes is constructed; Using a graph neural network model, the relationship between nodes and edges in the dynamic data graph is analyzed and processed to obtain the influence factors between nodes and predict future trends; generating decision support information based on the influencing factors and prediction results; The decision support information is fed back to the real-time data source to form a closed-loop control mechanism.

2. The method according to claim 1, characterized in that: The graph neural network model is used to analyze and process the relationship between nodes and edges in the dynamic data graph, obtain the influence factors between nodes and predict future trends, including: The dynamic data graph is used as input data, and a weight matrix of nodes and edges in the dynamic data graph is defined to obtain a graph neural network model; In each layer of the graph neural network model, nodes in the dynamic data graph and information of adjacent nodes to the nodes are exchanged to update feature representations of each node to obtain updated node features; According to the updated node features, a pooling operation is applied to aggregate the updated node features to obtain key information with reduced dimensions; Based on the key information of reducing the dimension, in the last layer of the graph neural network model, a readout function is used to summarize the final feature representations of all nodes to obtain a global representation of the dynamic data graph; Utilizing the global representation, through a regression algorithm, the influence factors between nodes in the dynamic data graph are analyzed and processed to obtain influence factors that quantify the degree of interaction between nodes; Based on the influencing factors, a graph neural network model extended by a recurrent neural network is used to predict the changing trend of the dynamic data graph to obtain a prediction result, which includes future node states and their mutual influence.

3. The method according to claim 2, characterized in that The global representation is used to analyze and process the influence factors between nodes in the dynamic data graph through a regression algorithm to obtain influence factors that quantify the degree of interaction between nodes, including: Using the global representation generated by the graph neural network model, feature extraction is performed on the relationship between nodes in the dynamic data graph to obtain a global feature vector containing information about the interaction between nodes; Applying a regression algorithm to train the global feature vector to obtain a regression model that quantifies the degree of interaction between nodes; According to the trained regression model, the newly acquired global feature vector is predicted and processed to obtain the quantitative value or category label of the degree of interaction between nodes, which is used as the influencing factor.

4. The method according to claim 3, characterized in that Based on the influencing factors, the graph neural network model extended by the recurrent neural network is used to predict the change trend of the historical dynamic data graph to obtain the state of future nodes and their mutual influence, including: Based on the influencing factors, feature extraction processing is performed on the node states in the dynamic data graph and the influences between them to obtain time series features; Using a graph neural network model extended by a recurrent neural network, the time series features are dynamically simulated and processed to obtain a dynamic evolution model; According to the dynamic evolution model, the dynamic data map at future time points is predicted and processed to obtain the states of future nodes and their mutual influences; Evaluate and process the states of the future nodes and their mutual influence to obtain a credibility index of the prediction result, and adjust the parameters of the graph neural network model extended by the recurrent neural network as needed; Output the prediction results.

5. The method according to claim 2, characterized in that: In each layer of the graph neural network model, nodes in the dynamic data graph and information of adjacent nodes of the nodes are exchanged to update feature representation of each node to obtain updated node features, including: Calculate the nodes in the dynamic data graph and with nodes Neighboring nodes The attention weight between : ； in, Is a node and nodes The attention weights between is the attention coefficient vector, Represents the concatenation operation of node feature vectors, that is, connecting two node feature vectors together to form a new vector. Is an activation function used to increase the nonlinearity in the graph neural network model. Represents the weight matrix, which is used to linearly transform node features. Applied to The node feature vector of the layer, and Respectively represent Node in layer and nodes The node feature vector of Representation Node The neighbor set of all nodes The set of adjacent nodes. Indicates Node in layer The characteristic vector of Indicates Node in layer The eigenvector of For nodes in the dynamic data graph and with nodes Neighboring nodes The information is exchanged to obtain the updated node The node feature vector : ； in, Is a node exist The updated node feature vector of the layer, and They are The weight matrix and bias vector of the layer, is the activation function, Is a node exist The node feature vector of the layer, Is a node exist The node feature vector of the layer.

6. The method according to claim 1, characterized in that The step of constructing a dynamic data graph that can reflect the correlation between different data nodes based on the behavior pattern information includes: Performing entity object extraction processing on the behavior pattern information to obtain entity objects as nodes, and assigning a unique identifier to each node; Identify the relationship between the entity objects to obtain edges connecting different nodes to reflect the correlation between the entities; Record and process the time series changes of the relationship between the nodes to obtain time series information reflecting the time dimension of the dynamic data graph; Analyze and process the importance and group characteristics of the nodes to obtain feature information that enhances the performance of the association between nodes in the dynamic data graph; A dynamic data graph is constructed based on the entity objects of the nodes, the edges connecting different nodes, the time series information, and the characteristic information representing the correlation between the nodes.

7. The method according to claim 1, characterized in that The step of performing instant pattern recognition processing on the data stream set according to the distributed computing framework to obtain behavior pattern information includes: Cleaning and standardizing the data in the data stream set to obtain pre-processed data with noise eliminated and unified data format; Using a streaming processing engine, the pre-processed data is analyzed and processed in real time to obtain a preliminary feature representation; Performing pattern mining processing on the preliminary feature representation to obtain pattern features; Based on the pattern features, high-level abstract processing is performed on the pattern features to obtain behavior pattern information.

8. A big data analysis and processing system based on deep learning, characterized in that: include: A collection module is used to collect and process real-time data streams containing structured data and unstructured data from multiple heterogeneous data sources using a real-time data collection system to obtain a data stream set; An identification module, used for performing instant pattern recognition processing on the data stream set according to a distributed computing framework to obtain behavior pattern information; A construction module, used to construct a dynamic data graph that can reflect the correlation between different data nodes based on the behavior pattern information; An analysis module is used to analyze and process the relationship between nodes and edges in the dynamic data graph using a graph neural network model, obtain the influence factors between nodes and predict future trends; A generation module, used to generate decision support information according to the influencing factors and prediction results; The feedback module is used to feed back the decision support information to the real-time data source to form a closed-loop control mechanism.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a big data analysis and processing method based on deep learning as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, a big data analysis and processing method based on deep learning as described in any one of claims 1 to 7 is implemented.

Images