Data acquisition and analysis system and method based on web crawler technology

Abstract

The invention relates to the field of data collection and analysis, and particularly discloses a data collection and analysis system and method based on a web crawler technology, and the method comprises the steps: firstly obtaining the historical record data of power equipment collected by a web crawler and the network data of power supply equipment at a plurality of preset time points, and then carrying out the deep learning technology, and finally, generating a power grid equipment health state report through a generator, so as to identify potential problems and rules in equipment operation, improve the accuracy of fault prediction, help to implement preventive maintenance measures, and reduce downtime and maintenance cost.

Description

Technical Field

[0001] The present application relates to the field of data collection and analysis, and more specifically, to a data collection and analysis system and method based on web crawler technology. Background Art

[0002] Electrified railways refer to railways that can power electric trains. Electric locomotives on electrified railways do not have their own energy and need to be continuously supplied with electricity by the power grid along the railway to drive the vehicles. In order to ensure the stable operation of electrified railways and their electric locomotives, it is necessary to carry out certain power planning for the power grid along the electrified railways.

[0003] Before conducting power planning, a detailed diagnosis and analysis of the current status of the power grid must be conducted. Although the daily production and management of power grid companies have been informatized, power planning has never kept up with the pace of information technology. This is mainly because the various data required are scattered in different information systems. For example, the main network equipment ledger is stored in the PMS system, the distribution network equipment ledger is in the distribution network PMS system, and the production data such as equipment load rate is in IDP600, and the marketing data such as power consumption and users are in the power consumption collection system. The intercommunication and interconnection of data between different systems is very difficult.

[0004] The current power planning and collection method still relies on manual data collection from various systems and the use of office software such as Excel to process the data. The entire process is completely dependent on manual work and lacks effective information technology support, so it is inefficient. At the same time, power grid diagnosis under existing technologies also faces similar challenges. Due to its cumbersome workload, it can generally only be performed once a year and cannot achieve continuous dynamic monitoring.

[0005] Therefore, a data collection and analysis system and method based on web crawler technology is desired. Summary of the invention

[0006] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides a data collection and analysis system and method based on web crawler technology, which first obtains the historical record data of power equipment collected by the web crawler and the network data of the power supply equipment at multiple predetermined time points, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally generates a power grid equipment health status report through a generator to identify potential problems and rules in equipment operation, improve the accuracy of fault prediction, help implement preventive maintenance measures, and reduce downtime and maintenance costs.

[0007] According to one aspect of the present application, a data collection and analysis system based on web crawler technology is provided, which includes:

[0008] The power equipment data acquisition module is used to acquire the power equipment historical record data collected by the network crawler and the network data of the power supply equipment at multiple predetermined time points;

[0009] A power equipment data extraction module, used to extract power equipment historical record associated feature vectors and power supply equipment network data associated feature vectors from the power equipment historical record data collected by the network crawler and the network data of the power supply equipment at multiple predetermined time points;

[0010] The power grid equipment health status report generating module is used to generate a power grid equipment health status report based on the power equipment history record associated feature vector and the power supply equipment network data associated feature vector.

[0011] According to another aspect of the present application, a data collection and analysis method based on a web crawler technology is provided, which includes:

[0012] Obtaining historical record data of power equipment and network data of power supply equipment at multiple predetermined time points collected by a web crawler;

[0013] Extracting a power equipment historical record associated feature vector and a power supply equipment network data associated feature vector from the power equipment historical record data collected by the network crawler and the network data of the power supply equipment at multiple predetermined time points;

[0014] A power grid equipment health status report is generated based on the power equipment history record associated feature vector and the power supply equipment network data associated feature vector.

[0015] Compared with the prior art, the present application provides a data collection and analysis system and method based on web crawler technology, which first obtains the historical record data of power equipment collected by the web crawler and the network data of the power supply equipment at multiple predetermined time points, and then uses deep learning technology to perform feature extraction and correlation analysis on the two. Finally, a generator is used to generate a power grid equipment health status report to identify potential problems and patterns in equipment operation, improve the accuracy of fault prediction, help implement preventive maintenance measures, and reduce downtime and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1It is a block diagram of a data collection and analysis system based on web crawler technology according to an embodiment of the present application.

[0018] Figure 2 It is a block diagram of an electric power equipment data extraction module in a data acquisition and analysis system based on web crawler technology according to an embodiment of the present application.

[0019] Figure 3 It is a block diagram of a power supply equipment network data feature extraction unit in a data collection and analysis system based on network crawler technology according to an embodiment of the present application.

[0020] Figure 4 This is a block diagram of a power grid equipment health status report generation module in a data collection and analysis system based on web crawler technology according to an embodiment of the present application.

[0021] Figure 5 The present invention is a flowchart of a data collection and analysis method based on web crawler technology according to an embodiment of the present application.

[0022] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0024] Figure 1 Schematic diagram of a data collection and analysis system based on web crawler technology in an embodiment of the present application. Figure 1 As shown, according to the data collection and analysis system 100 based on the web crawler technology of the embodiment of the present application, it includes: an electric power equipment data acquisition module 110, which is used to obtain the electric power equipment historical record data collected by the web crawler and the network data of the power supply equipment at multiple predetermined time points; an electric power equipment data extraction module 120, which is used to extract the electric power equipment historical record associated feature vector and the power supply equipment network data associated feature vector from the electric power equipment historical record data collected by the web crawler and the network data of the power supply equipment at multiple predetermined time points; a power grid equipment health status report generation module 130, which is used to generate a power grid equipment health status report based on the electric power equipment historical record associated feature vector and the power supply equipment network data associated feature vector.

[0025] In the above-mentioned data collection and analysis system 100 based on web crawler technology, the power equipment data acquisition module 110 is used to obtain the power equipment historical record data collected by the web crawler and the network data of the power supply equipment at multiple predetermined time points. It should be understood that before conducting power planning, a comprehensive diagnosis and analysis of the current status of the power grid must be carried out. Although the daily production and management of the power grid company have been informatized, power planning has not kept up with the pace of information technology. The main problem is that the required data is scattered in different information systems: the main network equipment information is stored in the PMS system, the distribution network equipment information is in the distribution network PMS system, and the production data such as the equipment load rate is stored in the IDP600, and the power and user data are in the power collection system. Data intercommunication between different systems is difficult. At present, power planning still relies on manual data collection from multiple systems and processing with the help of software such as Excel. This process is completely dependent on manpower and lacks effective information technology support, resulting in low efficiency. At the same time, power grid diagnosis under the existing technology also faces similar challenges. It can usually only be carried out once a year, and continuous dynamic monitoring cannot be achieved. Therefore, in the technical solution of the present application, by obtaining the historical record data of power equipment collected by a web crawler and the network data of the power supply equipment at multiple predetermined time points, and combining it with deep learning technology, a power grid equipment health status report is generated. By discovering potential hidden dangers and patterns in equipment operation, the accuracy of fault prediction is enhanced, and the implementation of preventive maintenance measures is supported, thereby reducing downtime and maintenance expenses.

[0026] Specifically, the historical record data of power equipment can reflect the equipment's past operating status and performance, including important information such as fault occurrence and maintenance records. These data play a key role in predicting possible problems of equipment in the future and optimizing equipment operation modes. Obtaining network data of power supply equipment at different time points can reveal the state changes of equipment and network load conditions in real-time operation. By analyzing network data, load fluctuations of equipment, opportunities for energy efficiency improvement, and changes in network topology can be discovered in a timely manner, thereby improving the efficiency and stability of the power supply system. Combining these two types of data, the system can use advanced data mining and machine learning technologies to establish models and prediction algorithms for power equipment. These models not only help to achieve real-time monitoring and intelligent early warning of equipment operation, but also provide data-driven decision support for equipment operation and maintenance. By accurately grasping the operating status and network data of the equipment, the reliability, safety, and economy of power equipment can be maximized, while optimizing the utilization of energy resources and promoting the power industry to move towards intelligence and sustainable development.

[0027] In the above-mentioned data collection and analysis system 100 based on web crawler technology, the power equipment data extraction module 120 is used to extract power equipment historical record associated feature vectors and power supply equipment network data associated feature vectors from the power equipment historical record data collected by the web crawler and the network data of the power supply equipment at multiple predetermined time points. It should be understood that by comprehensively analyzing the power equipment historical records and network data, the system can use data mining and machine learning technology to build prediction models and intelligent algorithms. These models can not only realize real-time monitoring and prediction of equipment operation status, but also improve the reliability and efficiency of equipment, reduce energy waste and system operation risks. By extracting and analyzing associated feature vectors, the system can provide a scientific basis for the long-term operation and maintenance strategies of power equipment, and promote the development of the power industry towards intelligence and sustainable development.

[0028] Figure 2 FIG. 1 is a block diagram of a power equipment data extraction module in a data collection and analysis system based on a web crawler technology according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the power equipment data extraction module 120 includes: a power equipment historical record feature extraction unit 121, used to perform feature extraction on the power equipment historical record data collected by the web crawler to obtain a power equipment historical record associated feature vector; a power supply equipment network data feature extraction unit 122, used to perform feature extraction on the network data of the power supply equipment at multiple predetermined time points to obtain a power supply equipment network data associated feature vector.

[0029] It should be understood that the historical record data of power equipment usually contains detailed information such as the operating status, fault events, maintenance records, etc. of the equipment in different time periods. These data are large and complex, and direct use may lead to information redundancy or excessive noise, which is not conducive to subsequent analysis and application. Therefore, the process of feature extraction is to extract the most representative and informative features from the original data by applying methods such as statistics, signal processing or machine learning. The feature vector associated with the historical record of power equipment can include various types of features, such as time series features, frequency distribution features, abnormal event features, etc. These features can help the system understand the operating mode and trend of the equipment, so as to predict the future behavior of the equipment and warn of faults. For example, by extracting the operating time distribution characteristics of the equipment, the workload of the equipment and the frequently occurring fault modes can be analyzed; by extracting the abnormal event features, possible problems of the equipment can be quickly responded to and diagnosed.

[0030] Furthermore, the network data of power supply equipment at multiple predetermined time points are extracted for features, mainly to deeply understand the operating status, network load and energy usage of the power system, so as to optimize equipment management and energy efficiency management. The network data of power supply equipment at different time points reflects the real-time status and operating characteristics of the power system. These data include information such as load changes of equipment, current and voltage fluctuations, and changes in the topology of the power grid, which are of great significance to the stability and reliability of the system. Through feature extraction, representative features can be extracted from these complex time series data, such as the peak and valley distribution of power demand, the energy efficiency performance of equipment during operation, and the communication frequency between equipment. The extraction of associated feature vectors of power supply equipment network data helps to establish models and prediction algorithms for power systems. By analyzing feature vectors, potential problems and trends in equipment operation can be identified, such as frequent load overloads, uncoordinated problems between equipment, or data transmission delays between grid nodes. These feature vectors not only provide a basis for equipment failure prediction, but also support decision makers in formulating optimized grid operation strategies and energy allocation plans.

[0031] In a specific embodiment of the present application, the power equipment history record feature extraction unit 121 is specifically used to: perform power equipment history record text preprocessing on the power equipment history record data collected by the web crawler to obtain a power equipment history record word vector sequence; pass the power equipment history record word vector sequence through a converter-based power equipment history record semantic encoder to obtain a power equipment history record associated feature vector.

[0032] It should be understood that the power equipment history record data usually records detailed information such as the equipment's operating status, event occurrence, and maintenance records in text form. These text data may contain a large amount of natural language expressions and technical terms, but directly using these text data for analysis has the problems of large data volume, information redundancy, and unstructured. Therefore, the first step of preprocessing is to clean and standardize the text data, remove useless information such as punctuation and stop words, and perform stemming or word form normalization to reduce the complexity and diversity of the vocabulary. One of the goals of power equipment history record text preprocessing is to convert text data into a sequence of word vectors. Word vector is a mathematical expression that maps each word to a vector, allowing computers to understand and process the semantic relationship and contextual information between words. Through technologies such as Bag-of-Words or Word Embeddings (such as Word2Vec, GloVe), each text record can be converted into a sequence of word vectors. Such a conversion allows text data to be used as input for machine learning models to establish applications such as text classification, sentiment analysis, and keyword extraction. Specifically, the power equipment historical record data collected by the web crawler is segmented to obtain a word sequence; the embedding layer of the sequence encoder of the Clip model is used to map each word in the word sequence into a word embedding vector to obtain a power equipment historical record word vector sequence.

[0033] Furthermore, the power equipment history record word vector sequence is processed by a converter-based encoder to obtain the power equipment history record associated feature vector, in order to further improve the data representation ability and semantic understanding, so as to more accurately describe and analyze the operating status and behavioral characteristics of the equipment. The power equipment history record word vector sequence is a way of converting text data into vector representation, mapping each text record into a mathematical vector. Although this representation method can retain certain semantic information, it may still have insufficient context understanding or information loss. Therefore, a converter-based encoder (such as the Transformer model) can be used to further extract and encode context information and semantic associations in text data, thereby generating richer and more efficient feature representations. Among them, the converter-based encoder has the ability to handle long-distance dependencies and capture global semantic information, which makes it particularly effective in processing power equipment history record data. Through the Transformer model, the complex relationship between each word vector and its surrounding words can be considered at the same time, thereby establishing a more accurate text representation. This model structure includes a self-attention mechanism and a multi-head attention mechanism, which can effectively capture the key words and semantic associations in the text sequence, and improve the expression ability of the feature vector and the richness of the data. The converter-based encoder can also improve processing efficiency and model generalization capabilities through pre-trained models (such as BERT, GPT, etc.). The pre-trained model learns rich semantic representations and language patterns through self-supervised learning of large-scale text data, and can complete high-quality feature extraction and semantic understanding tasks under limited labeled data. This method not only improves the quality of feature vectors, but also can cope with the complex and diverse features in the power equipment history record data. Specifically, the converter-based Bert model of the converter-based power equipment history record semantic encoder is used to perform global context-based semantic encoding on the power equipment history record word vector sequence to obtain multiple feature vectors; and, the multiple feature vectors are cascaded to obtain a power equipment history record associated feature vector.

[0034] Figure 3 FIG. 1 is a block diagram of a power supply equipment network data feature extraction unit in a data collection and analysis system based on network crawler technology according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the power supply equipment network data feature extraction unit 122 is specifically used for: a power supply equipment network data semantic coding subunit 1221, used to semantically encode the network data of the power supply equipment at multiple predetermined time points to obtain multiple power supply equipment network data time series semantic feature vectors; a power supply equipment network data feature coding subunit 1222, used to feature encode the multiple power supply equipment network data time series semantic feature vectors to obtain a power supply equipment network data association feature vector.

[0035] It should be understood that the network data of power supply equipment at different time points reflects the real-time operating status of the equipment, load changes, grid topology adjustments and other information. The time series nature of these data means that it is necessary not only to consider the data at a single time point, but also to analyze the change trend and evolution law of the data in the time dimension. Through semantic encoding, these time series data can be converted into feature vector sequences with semantic associations, so that the data can better express the evolution process and change trend of the equipment status. The generation process of time series semantic feature vectors utilizes modern natural language processing technologies, such as Transformer models or sequence models based on recurrent neural networks (RNNs). These models can capture the semantic dependencies and contextual information between different time points in sequence data, and effectively express the dynamic changes of time series data. For example, through the self-attention mechanism of the Transformer model, the complex relationship between the data at each time point and the previous and next time points can be taken into account, thereby generating a feature vector with high semantics.

[0036] Furthermore, feature encoding is performed on the time series semantic feature vectors of network data of multiple power supply devices, aiming to comprehensively consider the mutual influence and correlation between multiple devices, thereby improving the accuracy and efficiency of power system management and optimization decisions. The semantic feature vectors generated by multiple power supply devices in the time series reflect the operating status and performance characteristics of each device at different time points. These vectors contain data such as load, power consumption, temperature change, etc. of the equipment. Through semantic encoding technology (such as Transformer model or LSTM model), these complex time series data can be converted into mathematical feature representations with semantic information. The feature encoding process utilizes the ability of deep learning models to understand sequence data, so that the historical operation data of each device is more richly and accurately expressed. Feature encoding comprehensively processes the time series semantic feature vectors of multiple devices to generate associated feature vectors of power supply device network data. These associated feature vectors not only consider the operation of a single device, but also the mutual influence and synergy between devices. For example, through attention mechanism or sequence alignment technology, data interaction and coordinated changes between devices can be captured, thereby revealing the association pattern and influence link between devices in the power system.

[0037] In a specific embodiment of the present application, the power supply equipment network data semantic encoding subunit uses a city security basic text semantic context encoder to semantically encode the network data of the power supply equipment at multiple predetermined time points.

[0038] It should be understood that the basic text semantic context encoders for urban security (such as BERT, RoBERTa, etc.) have demonstrated strong semantic understanding capabilities in processing large-scale text data through pre-training models. These models have learned the relationship between rich semantic information and text context through unsupervised learning of large-scale corpora, and can convert natural language text into deep semantic representations. The basic text semantic context encoder for urban security is suitable for semantic encoding of power supply equipment network data at multiple predetermined time points because it can effectively capture long-term dependencies and global semantic information in text data. When processing historical record data of power equipment, this model can take into account the complex relationship between the data at each time point and the time points before and after it, thereby improving the semantic expression ability and accuracy of the data. The basic text semantic context encoder for urban security is also flexible and adjustable, and can be fine-tuned and optimized according to the characteristics of specific tasks and data. Specifically, the network data of the power supply equipment at multiple predetermined time points are segmented to obtain a network data word sequence; the embedding layer of the city security basic text semantic context encoder is used to map each network data word in the network data word sequence into a network data word embedding vector to obtain a sequence of network data word embedding vectors; the converter-based Bert model of the city security basic text semantic context encoder is used to perform global context semantic encoding on the sequence of network data word embedding vectors to obtain multiple power supply equipment network data time series semantic feature vectors.

[0039] In a specific embodiment of the present application, the power supply equipment network data feature encoding subunit 1222 includes: splicing the multiple power supply equipment network data time series semantic feature vectors to obtain a power supply equipment network data feature matrix; encoding the power supply equipment network data feature matrix through a power supply equipment network data multi-layer perceptron encoding neural network to obtain a power supply equipment network data association feature vector.

[0040] It should be understood that the time series semantic feature vectors of multiple power supply equipment network data reflect the operating status and characteristics of each device at different time points. After semantic encoding through deep learning models (such as Transformer, LSTM, etc.), these feature vectors already contain rich equipment operation information and time series change trends. Splicing these feature vectors can integrate the feature information of different devices into an overall data structure, so that the characteristics of each device are uniformly expressed in the spatial dimension. The generation of the power supply equipment network data feature matrix can effectively capture the correlation and synergy between devices. By splicing the time series semantic feature vectors of multiple devices, a data relationship model between devices can be established to reveal the possible interaction and linkage effects between devices. This comprehensive analysis helps to identify abnormal behaviors in the equipment group, optimize the integrated operation strategy of the equipment, and provide a basis for the overall evaluation of system performance.

[0041] Furthermore, the power supply equipment network data feature matrix is ​​encoded by the power supply equipment network data multi-layer perceptron (MLP) neural network in order to further extract and characterize the complex data associations and interaction patterns between devices, thereby supporting higher-level system analysis and decision-making. The power supply equipment network data feature matrix is ​​obtained by splicing the time-series semantic feature vectors of multiple devices, which reflects the comprehensive characteristics of each device at different time points. These features include information such as the operating status, energy consumption, and load changes of the equipment, which have been preliminarily semantically encoded and integrated. However, the information in the feature matrix still needs further deep processing and abstraction to reveal the potential association patterns between devices. The power supply equipment network data multi-layer perceptron encoding neural network (MLP) is a classic deep learning model with strong nonlinear mapping capabilities and flexibility. Through MLP, a higher level of feature abstraction and representation can be constructed on the basis of the feature matrix to further extract the complex relationships and data interaction patterns between devices. Through the combination of multiple layers of neurons and the role of activation functions, MLP can learn the nonlinear relationship between features, thereby improving the data expression ability and modeling accuracy. The generation of the associated feature vector of the power supply equipment network data is through the MLP coding neural network, which not only considers the characteristics of a single device, but also can capture the interaction and influence between devices. These associated feature vectors can reflect the common patterns and abnormal behaviors in the equipment group, which helps to identify potential problems and optimization space that may exist in the system. For example, through the training and optimization of the MLP model, the dynamic analysis and prediction of the operating status of the equipment group can be realized, providing decision support for equipment management and energy scheduling. By processing the power supply equipment network data feature matrix through the MLP coding neural network, the associated feature vector of the power supply equipment network data is obtained, which not only strengthens the abstraction and expression ability of the data, but also provides more in-depth and comprehensive data support for the intelligent management and optimization of the power system. This method not only helps to achieve efficient analysis and integrated modeling of equipment data, but also opens up new technical paths and application prospects for intelligent decision-making and data-driven development in the power industry. Specifically, each fully connected layer of the multi-layer perceptron coding neural network of the power supply equipment network data is fully connected in the forward pass of the layer to extract the high-dimensional implicit features of the eigenvalues ​​of each position in the power supply equipment network data feature matrix.

[0042] In the above-mentioned data acquisition and analysis system 100 based on web crawler technology, the power grid equipment health status report generation module 130 is used to generate a power grid equipment health status report based on the power equipment history record associated feature vector and the power supply equipment network data associated feature vector. It should be understood that the power equipment history record associated feature vector contains comprehensive information about the long-term operating status of the equipment. These feature vectors are extracted from the equipment historical data through deep learning or machine learning models, and can reflect the operating trends, periodic changes and possible abnormal behaviors of the equipment. The power supply equipment network data associated feature vector further considers the interaction and association effects between devices. These feature vectors reveal the collective behavior and potential systemic problems in the equipment group through the data interaction mode and influence link between multiple devices. By analyzing these associated feature vectors, common failure modes, synergies or load imbalances that may exist between devices can be identified, providing clues for system managers to discover and solve problems early.

[0043] Figure 4 FIG. 1 is a block diagram of a power grid equipment health status report generation module in a data collection and analysis system based on a web crawler technology according to an embodiment of the present application. Figure 4 As shown, in a specific embodiment of the present application, the power grid equipment health status report generation module 130 is specifically used for: a power grid equipment feature fusion unit 131, used to fuse the power equipment history record associated feature vector and the power supply equipment network data associated feature vector to obtain a power grid equipment health status feature vector; a power grid equipment feature optimization unit 132, used to perform subclass metric correlation prediction based on superposition paradigm feature adaptation on the power grid equipment health status feature vector to obtain an optimized power grid equipment health status feature vector; a health status report generation unit 133, used to pass the optimized power grid equipment health status feature vector through a generator to generate a power grid equipment health status report.

[0044] It should be understood that the fusion of these two types of feature vectors, namely the power equipment history record-related feature vectors and the power supply equipment network data-related feature vectors, can comprehensively evaluate the health status of power grid equipment in multiple dimensions. By integrating long-term historical data and the interaction between devices, the generated power grid equipment health status feature vector can more accurately describe the overall operation and potential problems of the equipment. The generation of this comprehensive feature vector not only helps to achieve in-depth analysis of the health status of a single device, but also reveals the overall trends and collective behaviors that may exist in the device group. For example, abnormal devices in the device cluster, potential equipment failure modes, or operating strategies that need to be optimized can be identified to provide system managers with timely decision support and action recommendations.

[0045] Furthermore, in the technical solution of the present application, considering that the historical record data of power equipment and the network data of power supply equipment may come from different data sources and equipment, these data may differ in quality, format, integrity, etc., resulting in inconsistent local distribution of feature vectors. Different equipment in the power grid may exhibit different operating characteristics due to factors such as model, age, maintenance status, etc. Such differences between equipment may lead to uneven distribution of feature vectors in local areas. The network data of power supply equipment changes over time, and data at different time points may exhibit different characteristics due to changes in factors such as power grid load and environmental conditions, resulting in differences in the local distribution of temporal semantic feature vectors. The converter-based power equipment historical record meaning encoder and multi-layer perceptron encoding neural network may not be able to fully capture all the feature information related to the health status of the equipment, or have different sensitivities to certain features, resulting in differences in the distribution of feature vectors in local areas. When fusing the power equipment historical record-related feature vectors and the power supply equipment network data-related feature vectors, the importance and weights of different features may not be properly processed, resulting in inconsistencies in the fused power grid equipment health status feature vectors in some local areas, which in turn results in the power grid equipment health status feature vectors being generated by the generator. There is a class harmony offset relative to the predetermined class label of the generator, affecting the generation iteration effect. Based on this, the subclass metric correlation prediction based on the superposition paradigm feature adaptation is performed on the power grid equipment health status feature vector.

[0046] Among them, the subclass metric correlation prediction based on superposition paradigm feature adaptation of the power grid equipment health state feature vector is performed to obtain the optimized power grid equipment health state feature vector, including: determining the weight learning parameter matrix of the generator; extracting each column vector in the weight learning parameter matrix to obtain a set of weight learning parameter column vectors; using the following optimization formula to respectively calculate the metric correlation prediction factor between each weight learning parameter column vector in the set of weight learning parameter column vectors and the power grid equipment health state feature vector to obtain a metric correlation prediction factor modulation vector, wherein the optimization formula is:

[0047]

[0048] Among them, V i represents each weight learning parameter column vector in the set of weight learning parameter column vectors, V c represents the health status feature vector of the power grid equipment, It means subtracting by position. is the superposition norm of the two norms of the eigenvectors, ||·|| F is the Frobenius norm of the eigenvector, M prepresents the weight learning parameter matrix, exp(·) represents the exponential operation with the natural constant e as the base, V p Representing the metric correlation prediction factor modulation vector; using the metric correlation prediction factor modulation vector as a weight vector, calculating the point-by-point multiplication between the metric correlation prediction factor modulation vector and the power grid equipment health status feature vector to obtain an optimized power grid equipment health status feature vector.

[0049] In the technical solution of the present application, it is considered that there are local distribution differences in the health status feature vector of the power grid equipment, which leads to the problem that when it is generated by the generator, there will be a class harmony offset relative to the predetermined class label of the generator, affecting the generation iteration effect. Based on this, in the technical solution of the present application, the subclass metric correlation prediction based on the superposition paradigm feature adaptation is performed on the health status feature vector of the power grid equipment, which uses the superposition paradigm of the two norms between each column vector in the weight learning parameter matrix of the generator and the health status feature vector of the power grid equipment as the subclass label related metric integral factor and the matrix norm of the weight learning parameter of the generator as the differential factor, and then uses the association paradigm between the subclass label related metric integral factor and the differential factor to perform bidirectional linear manifold modulation on the health status feature vector of the power grid equipment, thereby improving the local feature distribution consistency of the health status feature vector of the power grid equipment by adaptively stretching and compressing the health status feature vector of the power grid equipment, thereby reducing the harmony offset of the health status feature vector of the power grid equipment relative to the predetermined class label of the generator, thereby improving the accuracy of the generation result.

[0050] Furthermore, the optimized power grid equipment health status feature vector contains comprehensive equipment data features processed by deep learning or machine learning models. The optimized power grid equipment health status feature vector provides an in-depth analysis of the equipment health status by integrating the equipment history, operating status and the correlation effect between the equipment. However, the optimized power grid equipment health status feature vector itself is usually high-dimensional and abstract, and needs to be converted into a more specific and operational form through a generator. As a data processing and visualization tool, the generator can convert the abstract feature vector into an intuitive power grid equipment health status report. Among them, the generator can map the feature vector into a specific health status score, operating trend chart, equipment failure prediction and other forms based on a preset template or algorithm. Such a report not only provides the current status and performance evaluation of the equipment, but also shows the possible future development trend and optimization direction of the equipment. The generated power grid equipment health status report can help operation and maintenance personnel monitor the health status of the equipment in real time, detect potential failures or abnormal behaviors of the equipment as early as possible, and take preventive maintenance measures to improve the reliability and operation efficiency of the equipment.

[0051] In summary, the embodiment of the present application first obtains the historical record data of the power equipment collected by the web crawler and the network data of the power supply equipment at multiple predetermined time points, and then uses deep learning technology to perform feature extraction and correlation analysis on the two. Finally, a generator is used to generate a power grid equipment health status report to identify potential problems and patterns in equipment operation, improve the accuracy of fault prediction, help implement preventive maintenance measures, and reduce downtime and maintenance costs.

[0052] As described above, the data collection and analysis system 100 based on the web crawler technology according to the embodiment of the present application can be implemented in various terminal devices. In one example, the data collection and analysis system 100 based on the web crawler technology can be integrated into the terminal device as a software module and / or a hardware module. For example, the data collection and analysis system 100 based on the web crawler technology can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the data collection and analysis system 100 based on the web crawler technology can also be one of the many hardware modules of the terminal device.

[0053] Alternatively, in another example, the data collection and analysis system 100 based on web crawler technology and the terminal device may also be separate devices, and the data collection and analysis system 100 based on web crawler technology may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0054] Figure 5 Flowchart of a data collection and analysis method based on web crawler technology according to an embodiment of the present application. Figure 5 As shown, according to the data collection and analysis method based on web crawler technology in an embodiment of the present application, it includes: S110, acquiring the historical record data of power equipment collected by the web crawler and the network data of the power supply equipment at multiple predetermined time points; S120, extracting the power equipment historical record associated feature vector and the power supply equipment network data associated feature vector from the power equipment historical record data collected by the web crawler and the network data of the power supply equipment at multiple predetermined time points; S130, generating a power grid equipment health status report based on the power equipment historical record associated feature vector and the power supply equipment network data associated feature vector.

[0055] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned data collection and analysis method based on web crawler technology have been referred to above. Figures 1 to 4 The data collection and analysis system based on web crawler technology has been introduced in detail, and therefore, its repeated description will be omitted.

[0056] Below, reference Figure 6 To describe an electronic device according to an embodiment of the present application.

[0057] Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0058] The processor 11 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in the embodiment of the present application; the memory 12 can be implemented in the form of a read-only memory (Read Only Memory, ROM), a static storage device, a dynamic storage device or a random access memory (Random Access Memory, RAM), etc.

[0059] The memory 12 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 12, and the processor 11 calls and executes the data collection and analysis method based on the web crawler technology of the embodiment of this application;

[0060] Input / output interface 13, used to implement information input and output;

[0061] The communication interface 14 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0062] A bus 15 that transmits information between the various components of the device (e.g., the processor 11, the memory 12, the input / output interface 13, and the communication interface 14);

[0063] The processor 11 , the memory 12 , the input / output interface 13 and the communication interface 14 are connected to each other in communication within the device via a bus 15 .

[0064] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned data collection and analysis method based on web crawler technology is implemented.

[0065] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0066] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0067] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0068] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0069] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0070] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0071] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0072] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0073] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs. The preferred embodiment of the embodiment of the present application is described above with reference to the accompanying drawings, and the scope of rights of the embodiment of the present application is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of rights of the embodiments of the present application.

Claims

1. A data collection and analysis system based on web crawler technology, characterized in that: include: The power equipment data acquisition module is used to acquire the power equipment historical record data collected by the network crawler and the network data of the power supply equipment at multiple predetermined time points; A power equipment data extraction module, used to extract power equipment historical record associated feature vectors and power supply equipment network data associated feature vectors from the power equipment historical record data collected by the network crawler and the network data of the power supply equipment at multiple predetermined time points; The power grid equipment health status report generating module is used to generate a power grid equipment health status report based on the power equipment history record associated feature vector and the power supply equipment network data associated feature vector.

2. The data collection and analysis system based on web crawler technology according to claim 1 is characterized in that: The electric power equipment data extraction module comprises: An electric power equipment historical record feature extraction unit, used for extracting features from the electric power equipment historical record data collected by the web crawler to obtain an electric power equipment historical record associated feature vector; The power supply equipment network data feature extraction unit is used to extract features from the network data of the power supply equipment at multiple predetermined time points to obtain a power supply equipment network data associated feature vector.

3. The data collection and analysis system based on web crawler technology according to claim 2 is characterized in that: The power equipment historical record feature extraction unit comprises: Performing power equipment historical record text preprocessing on the power equipment historical record data collected by the web crawler to obtain a power equipment historical record word vector sequence; The electric equipment history record word vector sequence is passed through a converter-based electric equipment history record semantic encoder to obtain an electric equipment history record associated feature vector.

4. The data collection and analysis system based on web crawler technology according to claim 3 is characterized in that: The power supply equipment network data feature extraction unit is specifically used for: A power supply equipment network data semantic coding subunit, used for semantically coding the network data of the power supply equipment at a plurality of predetermined time points to obtain a plurality of power supply equipment network data time series semantic feature vectors; The power supply equipment network data feature encoding subunit is used to perform feature encoding on the multiple power supply equipment network data temporal semantic feature vectors to obtain a power supply equipment network data associated feature vector.

5. The data collection and analysis system based on web crawler technology according to claim 4 is characterized in that: The power supply equipment network data semantic encoding subunit uses a city security basic text semantic context encoder to semantically encode the network data of the power supply equipment at multiple predetermined time points.

6. The data collection and analysis system based on web crawler technology according to claim 5 is characterized in that: The power supply equipment network data feature encoding subunit is specifically used for: splicing the plurality of power supply equipment network data time series semantic feature vectors to obtain a power supply equipment network data feature matrix; The power supply equipment network data feature matrix is ​​encoded through a power supply equipment network data multi-layer perceptron encoding neural network to obtain a power supply equipment network data associated feature vector.

7. The data collection and analysis system based on web crawler technology according to claim 6 is characterized in that: The power grid equipment health status report generating module includes: A power grid equipment feature fusion unit, used to fuse the power equipment historical record associated feature vector and the power supply equipment network data associated feature vector to obtain a power grid equipment health status feature vector; A power grid equipment feature optimization unit, configured to perform subclass metric correlation prediction based on superposition paradigm feature adaptation on the power grid equipment health state feature vector to obtain an optimized power grid equipment health state feature vector; The health status report generating unit is used to pass the optimized power grid device health status feature vector through a generator to generate a power grid device health status report.

8. The data collection and analysis system based on web crawler technology according to claim 7 is characterized in that: The power grid equipment feature optimization unit is specifically used for: Determining a weight learning parameter matrix of the generator; Extracting each column vector in the weight learning parameter matrix to obtain a set of weight learning parameter column vectors; Respectively calculating the metric correlation prediction factor between each weight learning parameter column vector in the set of weight learning parameter column vectors and the power grid device health state feature vector to obtain a metric correlation prediction factor modulation vector; The metric correlation prediction factor modulation vector is used as a weight vector, and the position point multiplication between the metric correlation prediction factor modulation vector and the power grid device health state feature vector is calculated to obtain an optimized power grid device health state feature vector.

9. A data collection and analysis method based on web crawler technology, characterized in that: include: Obtaining historical record data of power equipment and network data of power supply equipment at multiple predetermined time points collected by a web crawler; Extracting a power equipment historical record associated feature vector and a power equipment network data associated feature vector from the power equipment historical record data collected by the network crawler and the network data of the power supply equipment at a plurality of predetermined time points; A power grid equipment health status report is generated based on the power equipment history record associated feature vector and the power supply equipment network data associated feature vector.

10. The data collection and analysis method based on web crawler technology according to claim 9 is characterized in that: Extracting a power equipment historical record associated feature vector and a power supply equipment network data associated feature vector from the power equipment historical record data collected by the network crawler and the network data of the power supply equipment at multiple predetermined time points includes: Performing feature extraction on the power equipment historical record data collected by the web crawler to obtain a power equipment historical record associated feature vector; Feature extraction is performed on the network data of the power supply device at multiple predetermined time points to obtain a power supply device network data associated feature vector.

Images