A method and system for information collection and analysis based on big data

Abstract

The present application provides a method and system for information collection and analysis based on big data. Among them, the relevant data of multiple heterogeneous data sources are automatically identified and captured; the relevant data are streamed in real time, and an information graph is constructed based on the time series characteristics and event correlation of the relevant data; the importance of the nodes in the information graph is evaluated using graph theory algorithms and machine learning techniques to obtain node importance scores, and the key information nodes and potential information islands in the information graph are determined based on the node importance scores; based on the node importance scores of key information nodes and potential information islands, the current information retrieval strategy is optimized to obtain an optimized information retrieval strategy; the selection and priority sorting of multiple heterogeneous data sources are adjusted according to the optimized information retrieval strategy, and the feedback is fed back to the information collection engine. The technical solution provided by the present application improves the efficiency, accuracy and intelligence level of information collection and processing.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of big data processing technology, and in particular, to a method and system for collecting and analyzing information based on big data. Background Art

[0002] In today's highly information-based society, both enterprises and individuals are faced with the challenge of massive amounts of information. Especially in the fields of financial analysis, market research, security monitoring, etc., users need to quickly and accurately obtain valuable information from widely distributed and diverse data sources.

[0003] There are a variety of information collection and analysis tools on the market. They usually use web crawler technology to automatically crawl the required data from the Internet and combine it with big data platforms such as Hadoop or Spark for batch processing.

[0004] Although existing solutions have met the needs of information collection and preliminary processing to a certain extent, there are still the following shortcomings: most traditional methods are difficult to adjust their workflows according to the specific needs of users, and are particularly rigid when faced with a rapidly changing data environment, resulting in low efficiency, low accuracy and low intelligence in information collection and processing. Summary of the invention

[0005] The embodiments of the present application provide a method and system for information collection and analysis based on big data, so as to solve the problems of low efficiency, low precision and low intelligence level of information collection and processing in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a method for collecting and analyzing information based on big data, comprising:

[0007] Start the information collection engine to automatically identify and capture relevant data from multiple heterogeneous data sources based on preset industry standards and user customization requirements;

[0008] A distributed computing framework is used to perform real-time streaming processing on the relevant data, and an information graph is constructed according to the time series characteristics and event correlation of the relevant data, wherein the information graph includes a plurality of nodes, and the information graph is used to dynamically reflect the interaction mode between the plurality of nodes and the trend of change over time;

[0009] Using graph theory algorithms and machine learning techniques to evaluate the importance of nodes in the information graph, obtain node importance scores, and determine key information nodes and potential information islands in the information graph based on the node importance scores, wherein the node importance scores include: node attribute information and the strength of relationships between nodes;

[0010] Based on the node importance scores of the key information nodes and the potential information islands, optimizing the current information retrieval strategy to obtain an optimized information retrieval strategy;

[0011] The selection and priority sorting of the multi-dimensional heterogeneous data sources are adjusted according to the optimized information retrieval strategy and fed back to the information collection engine.

[0012] In a second aspect, an embodiment of the present application provides an information collection and analysis system based on big data, including:

[0013] The identification and crawling module is used to start the information collection engine and automatically identify and crawl relevant data from multiple heterogeneous data sources based on preset industry standards and user customization requirements;

[0014] A construction module, used to perform real-time streaming processing on the relevant data using a distributed computing framework, and to construct an information graph according to the time series characteristics and event correlation of the relevant data, wherein the information graph includes a plurality of nodes, and the information graph can dynamically reflect the interaction mode between the plurality of nodes and the trend of change over time;

[0015] An evaluation module, for evaluating the importance of nodes in the information graph using graph theory algorithms and machine learning techniques to obtain node importance scores, and determining key information nodes and potential information islands in the information graph based on the node importance scores;

[0016] An optimization module, used to optimize the information retrieval strategy based on the node importance scores of the key information nodes and the potential information islands, so as to enhance the pertinence of subsequent data crawling;

[0017] A module is formed to adjust the selection and priority sorting of the multi-heterogeneous data sources according to the optimized information retrieval strategy, and feed back to the information collection engine.

[0018] In a third aspect, an embodiment of the present application provides a computing device, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a method for collecting and analyzing information based on big data as described in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program, which, when executed by a computer, implements an information collection and analysis method based on big data as described in the first aspect.

[0020] In an embodiment of the present application, relevant data from multiple heterogeneous data sources are automatically identified and captured; relevant data are streamed in real time, and an information graph is constructed based on the time series characteristics and event correlation of the relevant data; graph theory algorithms and machine learning techniques are used to evaluate the importance of nodes in the information graph to obtain node importance scores, and based on the node importance scores, key information nodes and potential information islands in the information graph are determined; based on the node importance scores of key information nodes and potential information islands, the current information retrieval strategy is optimized to obtain an optimized information retrieval strategy; the selection and priority sorting of multiple heterogeneous data sources are adjusted according to the optimized information retrieval strategy, and the feedback is fed back to the information collection engine.

[0021] This application processes real-time data streams through a distributed computing framework, supports multiple types of data sources (such as text, pictures, audio and video), and conducts in-depth analysis of time series data, combining event detection algorithms to accurately capture key events. Graph theory algorithms and machine learning techniques are used to build information graphs, evaluate the importance of nodes, and identify key information nodes and information islands. Based on these evaluation results, the system can dynamically adjust data source selection and priority, optimize information retrieval strategies, and improve data processing efficiency and resource utilization. In addition, this method also promotes cross-domain information flow and sharing, supports data-driven decision-making, and adapts to growing data needs through automation and scalability.

[0022] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A flowchart of a method for collecting and analyzing information based on big data provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the structure of an information collection and analysis system based on big data provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0028] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to different types.

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0030] Figure 1 A flowchart of a method for collecting and analyzing information based on big data is provided for an embodiment of the present application, such as Figure 1 As shown, the method includes:

[0031] 101. Start the information collection engine to automatically identify and capture relevant data from multiple heterogeneous data sources based on preset industry standards and user customization requirements;

[0032] In this step, the information collection engine is an automated tool or system that can collect data from different sources according to pre-set rules or user-specified requirements. These data may come from different formats and structures (i.e., multiple heterogeneous data sources), such as web pages, databases, API interfaces, etc.

[0033] In the embodiment of the present application, the data type and scope to be collected are first defined, and then the information collection engine is configured to automatically search and extract these data. This may involve a variety of technical means such as web crawlers, API calls, database queries, etc. to ensure that a wide range of data sources can be covered.

[0034] 102. Use a distributed computing framework to perform real-time streaming processing on the relevant data, and construct an information graph based on the time series characteristics and event correlation of the relevant data, wherein the information graph includes multiple nodes, and the information graph is used to dynamically reflect the interaction mode between the multiple nodes and the trend of change over time;

[0035] In this step, by using distributed computing frameworks (such as Apache Spark, Flink, etc.), large-scale data sets can be processed efficiently and real-time data analysis can be supported. Information graphs are a way to visualize data points (nodes) and their connections, which helps to understand complex data relationships.

[0036] In the embodiment of the present application, the data collected in step 101 are analyzed according to the time sequence or the relationship between events to form a series of continuous data streams. Then, a model is established using graph theory principles, which can reflect the connection between each data node and the change trend over time.

[0037] Optionally, in step 102, "the use of a distributed computing framework to perform real-time streaming processing on the relevant data, and constructing an information graph based on the time series characteristics of the relevant data and event correlation" includes: using a distributed computing framework set in a cluster environment to receive a multivariate heterogeneous data stream from an information collection engine, wherein the multivariate heterogeneous data stream includes text, images, video and audio; performing preprocessing operations on the received multivariate heterogeneous data stream through a real-time stream processing component in the distributed computing framework; applying a time series analysis algorithm to parse the timestamp information in the preprocessed multivariate heterogeneous data stream to identify the time series characteristics; using an event detection algorithm to extract target event information from the preprocessed multivariate heterogeneous data stream, wherein the target event includes: a data packet with preset keywords and preset types; based on the time series characteristics and target event information, using graph database technology to construct the information graph, wherein each node in the information graph represents an entity, and the edges in the information graph represent the connection between entities, and the weight of the edge is used to characterize the strength or importance of the connection between entities.

[0038] Distributed computing framework: An architectural design that allows multiple computers to work together to complete large-scale data processing tasks. Common distributed computing frameworks include Apache Spark, Apache Flink, etc.

[0039] Real-time stream processing: A technology that processes continuous data streams. Data is processed immediately after it is generated, which is suitable for application scenarios that require instant response.

[0040] Multi-heterogeneous data stream: refers to a complex data set composed of multiple types (such as text, images, video, audio, etc.).

[0041] Preprocessing operations: Perform preliminary processing such as cleaning and formatting on the raw data for subsequent analysis.

[0042] Time series characteristics: refers to the patterns or trends that appear when data is arranged in chronological order.

[0043] Event detection algorithms: Algorithms used to automatically identify the occurrence of specific events, usually based on pattern matching, rule engines, or more complex machine learning methods.

[0044] Graph database technology: technology for storing and querying graph-structured data, which is very suitable for processing data sets with complex relationships between nodes.

[0045] Information graph: A graphical knowledge representation method used to represent entities (nodes) and their relationships (edges).

[0046] The distributed computing framework receives multiple types of data streams provided by the information collection engine. Secondly, the real-time stream processing component is used to perform preliminary cleaning and transformation on these data streams to make them suitable for further analysis. Then, the time series analysis technology is used to parse the timestamp information in the preprocessed data to find trends that change over time. Further, the event detection algorithm is used to identify specific events of interest from data from different sources, such as the detection of certain keywords or specific types of data packets. Finally, based on the discovered time series features and event information, the graph database technology is used to create an information graph. Each node in the graph represents an entity, and the edge represents the relationship between these entities. The weight of the edge reflects the importance or strength of the relationship.

[0047] In the embodiment of the present application, it is assumed that the power company wants to comprehensively monitor the status of the power system in its coverage area and promptly detect potential faults or abnormal conditions. First, data is collected from messages about power outage reports on social media platforms. Photos or videos of transmission lines taken by drone inspection systems. Audio call records of on-site staff communicating with the control center. Then the distributed computing framework is deployed in the company's data center to receive various real-time data streams through API interfaces or message queues. Further filter junk information and standardize expression. Compress to reduce the transmission burden, and apply basic visual processing technology to remove background interference. Transcribe into text for subsequent processing. Analyze the specific time when various data items occur, such as the time when social media messages are released, the specific date of drone inspections, etc. Secondly, find words related to power supply interruptions. Use AI technology to automatically identify problems such as line damage and tree obstacles. Analyze the content of the call and capture discussions about equipment failures. Finally, based on the above analysis results, create an information map reflecting the status of infrastructure such as substations and transmission lines. For example, when social media in a certain area frequently mentions power outages, the corresponding nodes in the area will be marked, and the connection weights between other nodes directly related to it (such as upstream power supply stations) will be adjusted accordingly to reflect the degree of this impact.

[0048] 103. Use graph theory algorithms and machine learning techniques to evaluate the importance of nodes in the information graph to obtain node importance scores, and determine key information nodes and potential information islands in the information graph based on the node importance scores, wherein the node importance scores include: node attribute information and the strength of relationships between nodes;

[0049] In this step, by quantitatively analyzing each node in the information graph, its relative importance in the entire network can be evaluated. The "importance" mentioned here may refer to the role played by the node in disseminating information, controlling resources, etc.

[0050] In the embodiment of the present application, graph theory algorithms (such as PageRank algorithm) and machine learning methods are used to measure the importance of each node, while considering the characteristics of the node itself (such as content quality, update frequency) and the closeness of connection with other nodes. Finally, it is determined which nodes are key information nodes and which may be isolated information islands.

[0051] Optionally, in step 103, "the use of graph theory algorithms and machine learning techniques to evaluate the importance of nodes in the information graph to obtain node importance scores, and determine key information nodes and potential information islands in the information graph based on the node importance scores" includes: using the distributed computing framework to extract attribute information of each node in the information graph, wherein the attribute information includes: basic attributes of the node, the number of connections between nodes, and the type of connections between nodes, and the distributed computing framework is loaded with graph theory algorithms and machine learning models; based on the attribute information, the strength of the relationship between nodes is calculated; using graph theory algorithms to evaluate the centrality of nodes in the information graph, and based on the centrality and the attribute information, the node importance score of each node in the information graph is calculated; the attribute information, the strength of the relationship between nodes, and the node importance score are input into the machine learning model to determine the key information nodes; analyzing the connection status of each node in the information graph, and determining the nodes with a connection number less than a preset threshold as the potential information islands.

[0052] Information graph: is a data structure represented in a graphical form, which contains nodes (representing entities) and edges (representing the relationships between entities).

[0053] Basic attributes of a node: refers to the information carried by the node itself, such as name, type, etc.

[0054] The number of connections between nodes: refers to the number of direct connections between a node and other nodes.

[0055] Type of connection between nodes: refers to the nature of the connection between different nodes, which can be different types of relationships such as collaboration and dependency.

[0056] Distributed computing framework: A system that can process data in parallel on multiple computers, such as Apache Hadoop or Spark.

[0057] Graph theory algorithm: a series of mathematical methods used to analyze graph structures, common ones include PageRank, Betweenness Centrality, etc.

[0058] Machine learning model: is an algorithm that learns patterns and regularities from data through training, which may involve supervised learning or unsupervised learning methods.

[0059] Node importance score: A quantitative indicator given based on the position of the node in the network and its characteristics.

[0060] Key information nodes: refers to those nodes that are very important to the entire network. They may have characteristics such as high degree and betweenness.

[0061] Potential information islands: refers to nodes that have few connections with other nodes and are relatively isolated.

[0062] First, the basic properties, number of connections, and types of all nodes are extracted from the information graph through a distributed computing framework. Based on the connections between nodes, a specific method is used to calculate the strength of the relationship between each pair of nodes. Secondly, a graph theory algorithm is used to measure the relative importance of each node in the entire graph. Each node is scored based on the node attributes and its centrality results. All the relevant information obtained above is then input into a pre-trained machine learning model to predict which nodes are key information nodes. Finally, the connection status of each node is checked, and those nodes that are below a certain set standard are marked as potential information islands.

[0063] In the embodiment of the present application, it is assumed that within the information system of an electric power company, an information map containing power plants, substations, transmission lines, etc. as nodes can be constructed. By applying the above process to the map, facilities that are critical to the stable operation of the power grid (i.e., key information nodes) can be identified, such as core substations located on the power transmission path. Areas that are weakly connected to other parts due to geographical location or other factors (potential information islands) are found, which may require additional attention or improvement measures to enhance their connectivity and reliability.

[0064] 104. Optimizing the current information retrieval strategy based on the node importance scores of the key information nodes and the potential information islands to obtain an optimized information retrieval strategy;

[0065] In this step, based on the knowledge of key information nodes and potential information islands obtained in the previous step, the existing information retrieval methods can be improved to make them more accurate and effective.

[0066] In an embodiment of the present application, the original information search logic is adjusted according to the node importance score of the node, such as increasing the attention to certain high-value areas and reducing the exploration of inefficient areas, thereby improving the overall information discovery efficiency.

[0067] Optionally, the "optimizing the current information retrieval strategy based on the node importance scores of the key information nodes and the potential information islands to obtain an optimized information retrieval strategy" in step 104 includes: using the node importance scores to adjust the current information retrieval strategy to obtain a preliminary optimization scheme, wherein the preliminary optimization scheme is used to increase the attention of key information nodes and reduce the search frequency of the potential information islands; using historical node importance scores and a preset learning model to predict the information graph to obtain prediction results, wherein the prediction results are used to characterize the nodes in the information graph that are transformed into key information nodes within a preset time period; based on the preliminary optimization scheme and the prediction results, determining the optimized information retrieval strategy.

[0068] Key information nodes: nodes with high importance in the information graph. They may be hubs in the network or nodes that play a decisive role in the overall function.

[0069] Potential information islands: nodes that are relatively isolated in the information graph and have few connections with other nodes.

[0070] Node importance scoring: Quantitative evaluation of the importance of nodes in information graphs through graph theory algorithms and machine learning models.

[0071] Information retrieval strategy: defines the methodology of how to effectively search and access information, including but not limited to the selection of query keywords, definition of search scope, etc.

[0072] Historical node importance score: the evaluation record of node importance over a period of time in the past.

[0073] Pre-trained learning model: A pre-trained machine learning model used to make predictions about future situations.

[0074] Prediction results: An estimate of future conditions based on existing data and models, especially whether a node is likely to become a key information node.

[0075] First, the existing information retrieval strategy is adjusted using the node importance scores of key information nodes and potential information islands that have been obtained. The purpose is to ensure that more attention is paid to the information of key information nodes during the retrieval process, while reducing the attention to potential information islands, thereby optimizing resource allocation. Next, the historical importance assessment data is combined with the preset machine learning model to predict which nodes may become new key information nodes in the future. This step aims to prepare in advance to adapt to changes in the graph structure. Finally, a more efficient information retrieval strategy is finally determined by comprehensively considering the preliminary optimization plan and the prediction results of future trends. This strategy not only reflects the current information distribution characteristics, but also takes into account potential changes in the future.

[0076] In the embodiment of the present application, it is assumed that the power company needs to regularly update its information retrieval system to ensure that key operation and maintenance information can be found quickly and accurately to support decision-making and emergency response. First, the importance of all sites and service points in the power grid (as nodes in the information map) is re-evaluated. Adjust the retrieval strategy according to the latest evaluation results, such as increasing attention to large hub substation information, while reducing the frequency of routine inspections of remote and small sites that rarely change. Then, using the change data of the importance of each node in the past few years, combined with seasonal load changes, new projects and other factors as input, the machine learning model predicts new key nodes that may appear in the next year. For example, a new industrial park is about to be put into use and is expected to become an important growth point for electricity demand. Finally, combining the above two points, a new information retrieval strategy is formed. The new strategy not only improves the support for known key nodes, but also pre-strengthens the information collection capabilities of nodes that may become important in the future in the forecast. Specific measures may include adjusting the frequency of data collection, priority sorting mechanisms, etc., to ensure that the most critical information can be quickly obtained in any case.

[0077] 105. Adjust the selection and priority sorting of the multiple heterogeneous data sources according to the optimized information retrieval strategy, and feed back to the information collection engine.

[0078] In this step, the optimized information retrieval strategy not only affects how to process the acquired data, but also guides the direction of future data collection.

[0079] In the embodiment of the present application, all the above analysis results are applied to the practical operation level, that is, changing the selection preference and access order of the information collection engine for different data sources to ensure that the most valuable information resources can be located faster and more accurately. This may mean paying more attention to data sources identified as key information nodes and less attention to information islands.

[0080] Optionally, the step 105 of "adjusting the selection and priority sorting of the multi-dimensional heterogeneous data sources according to the optimized information retrieval strategy, and feeding back to the information collection engine to which it belongs" includes: using the optimized information retrieval strategy and the node importance score to evaluate the multi-dimensional heterogeneous data sources to obtain an evaluation result, wherein the evaluation result includes the priority of each data source; based on the evaluation result, determining a target data source among the multi-dimensional heterogeneous data sources, wherein the target data source is a data source having a priority greater than a preset priority; prioritizing the target data source based on the priority to obtain a sorting result, and sending the sorting result as feedback to the information collection engine to which it belongs.

[0081] Optimized information retrieval strategy: An information retrieval method adjusted based on the node importance scores of key information nodes and potential information islands.

[0082] Multi-source heterogeneous data sources: refers to data sets from different channels and in different formats, such as text, images, videos, sensor data, etc.

[0083] Node importance scoring: Quantitative evaluation of the importance of nodes in information graphs through graph theory algorithms and machine learning models.

[0084] Priority: The order in which data sources are processed or retrieved. A high priority means that they are processed earlier or more frequently.

[0085] Information Collection Engine: A software system responsible for collecting information from multiple data sources and capable of adjusting data collection strategies based on received instructions.

[0086] First, all available multivariate heterogeneous data sources are evaluated using the optimized information retrieval strategy and node importance scores to determine the relative importance or priority of each data source. Based on the evaluation results, data sources with priorities higher than a preset threshold are selected as target data sources. These data sources are considered more critical to the current information retrieval purpose. The selected target data sources are then sorted according to their priorities to ensure that the most relevant data sources are processed first. Finally, the sorted results are fed back to the information collection engine so that the engine can adjust the data collection strategy based on the latest priority list and give priority to more important data sources.

[0087] In the embodiment of the present application, it is assumed that the power company hopes to optimize its data collection and analysis process to more effectively monitor and manage the operating status of the power system, especially to respond quickly in the face of emergencies. First, the optimized information retrieval strategy is used, combined with the previous evaluation results of key information nodes (such as major substations, key transmission lines) and potential information islands (such as distribution units in remote areas). Evaluate the importance of all available data sources, which may include real-time data from the SCADA system, user reports on social media, drone inspection images, etc. Based on the evaluation, determine which data sources are critical to the current monitoring target. For example, during the high temperature in summer, special attention may be paid to data sources related to air conditioning loads. Set a priority threshold, such as selecting only the top 20% of the highest priority data sources as target data sources. Then sort the selected target data sources to ensure that those data sources that directly affect the stability of power supply, such as alarm information in the SCADA system, get the highest priority. Sorting should also consider the timeliness of the data source, and data with strong immediacy should be given a higher processing priority. Finally, the sorted list of target data sources is fed back to the company's information collection engine. The information collection engine adjusts its data capture plan accordingly, giving priority to collecting and processing high-priority data source information. For example, an engine might increase the frequency of polling for SCADA alarm information while reducing the intensity of monitoring for non-critical social media content.

[0088] This application takes into account that in the era of big data, especially for large enterprises such as power, it is very important to effectively manage and utilize massive heterogeneous data sources. To achieve this, an intelligent information collection and analysis process is needed, which can automatically evaluate the value of different data sources and adjust the data collection strategy accordingly. By introducing a scientific and reasonable evaluation system, it can ensure that key information can be captured and processed in a timely manner, while reducing unnecessary waste of resources.

[0089] The options are as follows:

[0090] Optionally, in step 105, “adjusting the selection and priority sorting of the multi-dimensional heterogeneous data sources according to the optimized information retrieval strategy and feeding back to the information collection engine” includes:

[0091] Data source priority The calculation formula is:

[0092] ;

[0093] in, Represents the data source Provides a quality score for the data. Represents the data source The data update frequency score, Represents the data source The relevance score with key information nodes, Represents the data source Historical performance scores, are weighted coefficients, which respectively represent the weights of data quality, update frequency, relevance, and historical performance scores in the data source priority, and satisfy ;

[0094] Data Source Quality Score Calculation formula:

[0095] ;

[0096] in, is the number of key information nodes, It is The importance score of each key information node, Is the data source For Data contribution value of key information nodes;

[0097] Data Source Update frequency score Calculation formula:

[0098] ;

[0099] in, Is the data source The average update interval of and are the shortest and longest average update intervals among all data sources, respectively;

[0100] Data Source Relevance score to key information nodes Calculation formula:

[0101] ;

[0102] in, Is the data source With The similarity score of a key information node can be determined by calculating the semantic similarity between the data provided by the data source and the key information node. It is Importance score of key information nodes;

[0103] Data Source Historical performance score Calculation formula:

[0104] ;

[0105] in, is the number of historical time periods, It is the time period The weight of Is the data source In time period The performance score within a period can be calculated based on the data quality and update frequency of the data source during the period.

[0106] This method aims to evaluate the priority of each data source through a comprehensive priority calculation formula to ensure that the information retrieval system can select the best data source. In this way, the accuracy and reliability of information retrieval can be improved and the data fusion process can be optimized.

[0107] in, : Weighting coefficient, usually determined through expert evaluation or experimental adjustment. :The importance score of key information nodes is usually determined by domain experts based on actual needs. :The data contribution value of the data source to the key information node can be determined through data quality assessment tools or manual assessment. : The average update time interval, the shortest and the longest average update time interval of the data source can be obtained from the log or monitoring data of the data source. : The similarity score between the data source and the key information node can be calculated through natural language processing technology (such as word vector, TF-IDF, etc.). : The weight of a time period is usually determined based on the importance of the time period. For example, the weight of a recent time period can be higher. :The performance score of the data source within a time period can be calculated through data quality assessment and update frequency assessment tools.

[0108] Specifically, the node Importance Rating It can be evaluated comprehensively through a variety of factors. Importance Rating It can be calculated by the following four main factors: Centrality score , density score , machine learning model prediction score and time sensitivity score Each of these factors reflects the importance of different aspects of the node in the network. The following are the specific expressions and calculation methods of these factors:

[0109] ;

[0110] in, Is a node The centrality score of . Is a node The density score of . Is the machine learning model for the node The prediction score of . Is a node Time sensitivity score. are weighted coefficients, representing the weights of centrality, density, machine learning prediction, and time sensitivity in node importance scoring, and satisfying .

[0111] Data Source For Data contribution value of key information nodes It can be evaluated based on a variety of factors, including the quality of the data provided by the data source, the relevance of the data, and the importance of the data source in a specific context. It is mainly based on the relevance and quality of the data provided by the data source and the key information nodes. The following is a possible expression and calculation method:

[0112] ;

[0113] in, is the number of key information nodes. It is The importance score of each key information node. Is the data source For The data contribution value of each key information node.

[0114] Data Source In time period Performance score within It can be evaluated through multiple dimensions, including data quality, update frequency, data completeness and consistency, etc. Is the data source During a specific period of time The following is a possible expression and calculation method:

[0115] ;

[0116] in, is the number of historical time periods. It is the time period The weight of . Is the data source In time period Performance score within.

[0117] In the embodiment of this application, it is assumed that the power company wants to improve its data-driven operation and maintenance efficiency, especially for real-time monitoring and fault prediction of the power grid. First, the SCADA system provides real-time data on the operating status of the power grid. Social media provides user feedback and reports on power services. UAVs inspect transmission lines and substations for visual inspections. Meteorological data provides information on weather conditions that affect the operation of the power grid.

[0118] For SCADA systems, the data they provide is highly accurate and covers a wide range, so each key information node The score is high; however, the data from social media may be noisy. The score is low. SCADA systems may update once a minute, while social media data updates are more random. By calculating the average update interval of each data source, it can be determined and . Use NLP tools to analyze the correlation between SCADA data and grid operation status, as well as the correlation between reports of power outages in social media posts and actual power outage events. Review the performance of various data sources at critical moments over the past few months, such as the data integrity and timeliness of SCADA systems under extreme weather conditions, and the accuracy of social media reports.

[0119] First, set the appropriate weighting coefficient ,For example ,make sure , and then calculate the quality score of each data source according to the above method , Update frequency score , relevance score and historical performance scores Then substitute the scores of each item into the formula Get the priority of each data source Finally, the data sources are sorted according to the calculated priorities, and the results are fed back to the information collection engine to adjust the subsequent data collection strategy.

[0120] This application takes into account that in complex networks, node importance assessment is crucial for understanding network structure, optimizing resource allocation, and improving system performance. Especially in the field of energy and power, accurately assessing the importance of each node in the power grid (such as substations, transmission lines, etc.) can help improve the safety and reliability of the power grid. To this end, a comprehensive node importance scoring system is designed, which combines the concepts of centrality and density in graph theory, the predictive ability of machine learning models, and the time sensitivity of nodes to form a multi-dimensional evaluation framework.

[0121] The options are as follows:

[0122] Optionally, the node importance score The calculation formula is:

[0123] ;

[0124] in, Representation Node The centrality score of Representation Node The density score of Represents a machine learning model for a node The prediction score of Representation Node Time sensitivity score, are weighted coefficients, representing the weights of centrality, density, machine learning prediction, and time sensitivity in node importance scoring, and satisfying ;

[0125] node The centrality score of Calculation formula:

[0126] ;

[0127] in, is the number of all nodes in the information graph, Is a node and nodes The shortest path length between

[0128] node Density score Calculation formula:

[0129] ;

[0130] in, Is a node The number of directly connected edges, is the maximum number of edges of all nodes in the information graph;

[0131] Machine learning models for nodes Prediction score Calculation formula:

[0132] ;

[0133] in, Is a node The characteristic vector of It is a pre-trained machine learning model;

[0134] node Time sensitivity score Calculation formula:

[0135] ;

[0136] in, is the length of the time window, is the weight of each time point in the time window, Is a node At the point in time activity score.

[0137] The overall design aims to provide a comprehensive and objective method to evaluate the importance of nodes in the information graph. By integrating information from different aspects, the scoring system can more accurately reflect the status of nodes in the network and their potential value. This method not only takes into account the structural properties of the nodes themselves, but also incorporates dynamic behavior factors, making the evaluation results closer to the actual situation.

[0138] Centrality ): Centrality reflects the intermediary position of a node in the network and is one of the commonly used indicators to measure the influence of a node. It is calculated by using the sum of the reciprocals of the shortest path lengths to reflect the accessibility of the node in the entire network.

[0139] density( ): Density reflects the ratio of the number of direct connections of a node to the maximum possible number of connections, and it is an indicator of the local connectivity of the node.

[0140] Machine Learning Prediction ): Machine learning models are able to learn patterns from historical data and make predictions about the importance of nodes in the future. This helps identify nodes that are not currently important but may become important in the future.

[0141] Time Sensitivity ( ): Considering that the importance of nodes in the network may change over time, the time sensitivity score reflects the activity of the node at different time points.

[0142] Centrality ) is to use graph theory algorithms (such as Dijkstra algorithm) to calculate the shortest path length from a node to all other nodes , then calculate the sum of its reciprocals and divide by the total number of nodes .

[0143] density( ) is the number of edges directly connected to the statistical node , and the maximum number of edges in the entire graph Make a comparison.

[0144] Machine Learning Prediction ) is used to construct the feature vector , including the basic properties of the node, historical behavior, etc., and then input into the pre-trained machine learning model to obtain the prediction score .

[0145] Time sensitivity ( ) refers to recording the activity scores of node i at different time points , and according to the weights within the time window Weighted sum.

[0146] In the embodiment of the present application, it is assumed that the power company hopes to optimize the operation and maintenance strategy and improve the reliability and safety of the power system by accurately evaluating the importance of each node in the power grid.

[0147] First, set the weighting coefficient according to the actual situation ,For example , making sure the sum equals 1.

[0148] Then use the Dijkstra algorithm to calculate the shortest path lengths between all substations, transmission lines and other nodes in the power grid. Calculate the centrality score of each node, which is the sum of the reciprocals of the shortest path lengths divided by the total number of nodes. .

[0149] Secondly, count the number of edges directly connected to each node . Find the maximum number of edges for all nodes in the power grid , and then calculate the density score of each node .

[0150] Further collect historical data of each node, including load, failure frequency, geographic location, etc., to form a feature vector Use historical data to train a machine learning model, such as a random forest or neural network, to predict the importance scores of nodes .

[0151] Then monitor and record the activity of each node at different time points, such as the load during peak hours. Set the weight within the time window according to the actual needs of grid operation. , and calculate the time sensitivity score of each node .

[0152] Finally, substitute the above scores into the formula In the process of calculating the importance of each node, the comprehensive importance score of each node is obtained. The operation and maintenance strategy is adjusted according to the node importance score, such as increasing the inspection frequency of high-scoring nodes or optimizing the configuration of their surrounding facilities. In this way, power resources can be allocated more specifically and the overall operation efficiency of the power grid can be improved.

[0153] node At the point in time Activity score It can be defined and calculated in many ways, depending on the application scenario and the type of data available. In the energy and power sector, the activity of a node can be determined by many factors, such as load level, failure frequency, maintenance activities, etc. The following is a possible expression for calculating the activity of a node At the point in time Activity score :

[0154] ;

[0155] in, Is a node At the point in time The load score. Is a node At the point in time The failure frequency score. Is a node At the point in time Maintenance activity score. are weighted coefficients, representing the weights of load, failure frequency, and maintenance activity in the activity score, respectively, and satisfying .

[0156] Figure 2 A schematic diagram of a big data-based information collection and analysis system is provided for an embodiment of the present application. Figure 2 As shown, the system includes:

[0157] The identification and capture module 21 is used to start the information collection engine, automatically identify and capture relevant data from multiple heterogeneous data sources according to preset industry standards and user customization requirements;

[0158] A construction module 22 is used to perform real-time streaming processing on the relevant data using a distributed computing framework, and to construct an information graph according to the time series characteristics and event correlation of the relevant data, wherein the information graph includes a plurality of nodes, and the information graph can dynamically reflect the interaction mode between the plurality of nodes and the trend of change over time;

[0159] An evaluation module 23 is used to evaluate the importance of nodes in the information graph using graph theory algorithms and machine learning techniques to obtain node importance scores, and determine key information nodes and potential information islands in the information graph based on the node importance scores;

[0160] An optimization module 24, configured to optimize the information retrieval strategy based on the node importance scores of the key information nodes and the potential information islands, so as to enhance the pertinence of subsequent data capture;

[0161] A module 25 is formed, which is used to adjust the selection and priority sorting of the multi-heterogeneous data sources according to the optimized information retrieval strategy, and feed back to the information collection engine.

[0162] Figure 2 The information collection and analysis system based on big data can be executed Figure 1 The implementation principle and technical effect of the information collection and analysis method based on big data described in the illustrated embodiment will not be described in detail. The specific way in which each module and unit performs operations in the information collection and analysis system based on big data in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0163] In one possible design, Figure 2 The information collection and analysis system based on big data in the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0164] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0165] The processing component 32 is used to: automatically identify and capture relevant data from multiple heterogeneous data sources; perform real-time streaming processing on the relevant data, and construct an information graph based on the time series characteristics and event correlation of the relevant data; use graph theory algorithms and machine learning techniques to evaluate the importance of nodes in the information graph to obtain node importance scores, and determine key information nodes and potential information islands in the information graph based on the node importance scores; optimize the current information retrieval strategy based on the node importance scores of key information nodes and potential information islands to obtain an optimized information retrieval strategy; adjust the selection and priority sorting of multiple heterogeneous data sources according to the optimized information retrieval strategy, and feed back to the information collection engine.

[0166] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0167] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0168] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0169] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0170] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0171] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0172] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The illustrated embodiment is a method for collecting and analyzing information based on big data.

[0173] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0174] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0175] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for information collection and analysis based on big data, characterized in that: include: Start the information collection engine to automatically identify and capture relevant data from multiple heterogeneous data sources based on preset industry standards and user customization requirements; A distributed computing framework is used to perform real-time streaming processing on the relevant data, and an information graph is constructed according to the time series characteristics and event correlation of the relevant data, wherein the information graph includes a plurality of nodes, and the information graph is used to dynamically reflect the interaction mode between the plurality of nodes and the trend of change over time; Using graph theory algorithms and machine learning techniques to evaluate the importance of nodes in the information graph, obtain node importance scores, and determine key information nodes and potential information islands in the information graph based on the node importance scores, wherein the node importance scores include: node attribute information and the strength of relationships between nodes; Based on the node importance scores of the key information nodes and the potential information islands, optimizing the current information retrieval strategy to obtain an optimized information retrieval strategy; The selection and priority sorting of the multi-dimensional heterogeneous data sources are adjusted according to the optimized information retrieval strategy and fed back to the information collection engine.

2. The method according to claim 1, characterized in that The method of using a distributed computing framework to perform real-time streaming processing on the relevant data and constructing an information graph based on the time series characteristics and event correlation of the relevant data includes: Receiving a multi-dimensional heterogeneous data stream from an information collection engine using a distributed computing framework arranged in a cluster environment, wherein the multi-dimensional heterogeneous data stream includes text, images, video and audio; Performing preprocessing operations on received multi-heterogeneous data streams through real-time stream processing components in the distributed computing framework; Apply time series analysis algorithms to parse the timestamp information in the preprocessed multivariate heterogeneous data stream and identify time series characteristics; Utilizing an event detection algorithm, target event information is extracted from the preprocessed multi-heterogeneous data stream, wherein the target event includes: a data packet containing a preset keyword and a preset type; Based on the time series characteristics and target event information, the information graph is constructed using graph database technology, wherein each node in the information graph represents an entity, and the edges in the information graph represent the connection between entities. The weight of the edge is used to characterize the strength or importance of the connection between entities.

3. The method according to claim 1, characterized in that The use of graph theory algorithms and machine learning techniques to evaluate the importance of nodes in the information graph to obtain node importance scores, and determining key information nodes and potential information islands in the information graph based on the node importance scores, includes: Utilizing the distributed computing framework, extracting attribute information of each node in the information graph, wherein the attribute information includes: basic attributes of the node, the number of connections between nodes, and the type of connections between nodes, the distributed computing framework is loaded with graph theory algorithms and machine learning models; Based on the attribute information, calculating the relationship strength between nodes; Using a graph theory algorithm to evaluate the centrality of a node in the information graph, and based on the centrality and the attribute information, calculating a node importance score for each node in the information graph; Inputting the attribute information, the strength of the relationship between the nodes and the node importance score into the machine learning model to determine the key information node; The connection status of each node in the information graph is analyzed, and the nodes whose number of connections is less than a preset threshold are determined as the potential information islands.

4. The method according to claim 1, characterized in that: Based on the node importance scores of the key information nodes and the potential information islands, the current information retrieval strategy is optimized to obtain an optimized information retrieval strategy, including: Using the node importance score, the current information retrieval strategy is adjusted to obtain a preliminary optimization scheme, wherein the preliminary optimization scheme is used to increase the attention of key information nodes and reduce the frequency of retrieval of potential information islands; The information graph is predicted using the historical node importance score and a preset learning model to obtain a prediction result, wherein the prediction result is used to characterize the nodes in the information graph that are transformed into key information nodes within a preset time period; Based on the preliminary optimization scheme and the prediction results, the optimized information retrieval strategy is determined.

5. The method according to claim 1, characterized in that The step of adjusting the selection and priority sorting of the multi-dimensional heterogeneous data sources according to the optimized information retrieval strategy and feeding back to the corresponding information collection engine includes: Using the optimized information retrieval strategy and the node importance score, the multivariate heterogeneous data source is evaluated to obtain an evaluation result, wherein the evaluation result includes the priority of each data source; Based on the evaluation result, determining a target data source among the multiple heterogeneous data sources, wherein the target data source is a data source having a priority greater than a preset priority; The target data sources are prioritized based on the priorities to obtain a ranking result, and the ranking result is sent as feedback to the corresponding information collection engine.

6. The method according to claim 3, characterized in that The node importance score The calculation formula is: ； in, Representation Node The centrality score of Representation Node The density score of Represents a machine learning model for a node The prediction score of Representation Node Time sensitivity score, are weighted coefficients, representing the weights of centrality, density, machine learning prediction, and time sensitivity in node importance scoring, and satisfying ; node The centrality score of Calculation formula: ； in, is the number of all nodes in the information graph, Is a node and nodes The shortest path length between node Density score Calculation formula: ； in, Is a node The number of directly connected edges, is the maximum number of edges of all nodes in the information graph; Machine learning models for nodes The predicted score Calculation formula: ； in, Is a node The characteristic vector of It is a pre-trained machine learning model; node Time sensitivity score Calculation formula: ； in, is the length of the time window, is the weight of each time point in the time window, Is a node At the point in time activity score.

7. The method according to claim 5, characterized in that The adjusting the selection and priority sorting of the multi-heterogeneous data sources according to the optimized information retrieval strategy and feeding back to the information collection engine includes: Data source priority The calculation formula is: ； in, Represents the data source Provides a quality score for the data. Represents the data source The data update frequency score, Represents the data source The relevance score with key information nodes, Represents the data source Historical performance scores, are weighted coefficients, which respectively represent the weights of data quality, update frequency, relevance, and historical performance scores in the data source priority, and satisfy ; Data Source Quality Score Calculation formula: ； in, is the number of key information nodes, It is The importance score of each key information node, Is the data source For Data contribution value of key information nodes; Data Source Update frequency score Calculation formula: ； in, Is the data source The average update interval of and are the shortest and longest average update intervals among all data sources, respectively; Data Source Relevance score to key information nodes Calculation formula: ； in, Is the data source With The similarity score of a key information node can be determined by calculating the semantic similarity between the data provided by the data source and the key information node. It is Importance score of key information nodes; Data Source Historical performance score Calculation formula: ； in, is the number of historical time periods, It is the time period The weight of Is the data source In time period The performance score within a period can be calculated based on the data quality and update frequency of the data source during the period.

8. An information collection and analysis system based on big data, characterized in that: include: The identification and crawling module is used to start the information collection engine and automatically identify and crawl relevant data from multiple heterogeneous data sources based on preset industry standards and user customization requirements; A construction module, used to perform real-time streaming processing on the relevant data using a distributed computing framework, and to construct an information graph according to the time series characteristics and event correlation of the relevant data, wherein the information graph includes a plurality of nodes, and the information graph can dynamically reflect the interaction mode between the plurality of nodes and the trend of change over time; An evaluation module, for evaluating the importance of nodes in the information graph using graph theory algorithms and machine learning techniques to obtain node importance scores, and determining key information nodes and potential information islands in the information graph based on the node importance scores; An optimization module, used to optimize the information retrieval strategy based on the node importance scores of the key information nodes and the potential information islands, so as to enhance the pertinence of subsequent data crawling; A module is formed to adjust the selection and priority sorting of the multi-heterogeneous data sources according to the optimized information retrieval strategy, and feed back to the information collection engine.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement an information collection and analysis method based on big data as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, an information collection and analysis method based on big data as described in any one of claims 1 to 7 is implemented.

Images