Power data mining method and system based on big data

Abstract

The invention relates to the technical field of data analysis, and particularly discloses an electric power data mining method and system based on big data, and the method comprises the steps: firstly carrying out the recursive classification and data clustering processing of electric power data, calculating the direction vector of each data block, and further introducing a semantic consistency analysis algorithm based on self-supervised learning, semantic consistency query response analysis is carried out on the direction vector of each data block and the direction vector set to mine the semantic consistency degree of each data block relative to the whole data set, potential interference data blocks are identified and removed based on the semantic consistency degree, and then frequency domain compression of effective data blocks is achieved. In this way, interference data blocks which are difficult to find by a traditional statistical analysis method can be effectively recognized, and a purer and high-quality data basis is provided for subsequent data mining.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and more specifically, to a power data mining method and system based on big data. BACKGROUND

[0002] Under the impetus of global energy transformation and digitalization, the power industry is accelerating towards intelligent and efficient direction. As a digital representation of the operating state of the power system, power data covers various types such as power load data, power generation data, power supply quality data, power consumer behavior data and weather data, and has crucial value for power system planning, scheduling, operation and market operation. In the early days, the processing and analysis methods of power data were relatively simple, mainly relying on manual experience and basic statistical methods. With the continuous expansion and increasing complexity of the power system, traditional methods are difficult to cope with the challenges brought by massive data, and cannot fully tap the potential value of data. In this context, power data mining technology emerges as the times require and becomes a research hotspot in the power field.

[0003] Currently, the most commonly used power data mining technology is the data mining method based on clustering analysis, which usually divides the power data into several data blocks and then clusters the data blocks to discover the rules and associations in the data. For example, the invention patent with publication number CN116304931A proposes a power data mining method based on big data, which eliminates interference data blocks and performs frequency domain compression on effective data blocks through recursive classification, data clustering, direction vector consistency analysis, frequency domain compression and other technical means, to realize more efficient data mining.

[0004] Specifically, the existing technology refines data classification layer by layer through recursive classification rules, extracts the direction vector of the data block by combining singular value decomposition and clustering algorithm, and judges whether the data block is an interference data block based on the average angle size between the direction vector of each data block and the direction vector of other data blocks. However, in the identification of interference data blocks, the existing technology only relies on the statistical characteristics between the direction vectors to judge the interference, which is easily affected by data noise and outliers, resulting in low recognition accuracy. For example, in the power load mutation scenario, some data blocks may deviate from the local clustering center but conform to the global trend of change, and the existing method may mistakenly identify them as interference data blocks, resulting in loss of valid information and affecting the accuracy of data mining.

[0005] Therefore, an optimized power data mining method and system based on big data are expected. SUMMARY

[0006] In order to solve the above technical problems, the present application is proposed. Embodiments of the present application provide a power data mining method and system based on big data, which first performs recursive classification and data clustering processing on power data, calculates the direction vector of each data block, and further introduces a semantic consistency analysis algorithm based on self-supervised learning, performs semantic consistency query response analysis on the direction vector of each data block and the direction vector set, to mine the semantic consistency degree of each data block relative to the whole data set, based on which potential interference data blocks are identified and removed, and then frequency domain compression of effective data blocks is realized. In this way, interference data blocks that are difficult to detect by traditional statistical analysis methods can be effectively identified, providing a more pure and high-quality data basis for subsequent data mining.

[0007] According to an aspect of the present application, a power data mining method based on big data is provided, which comprises: obtaining power data, performing recursive classification based on a tree structure on the power data to obtain a classification tree of the power data; performing data clustering processing on data values contained in all terminal nodes of the classification tree to obtain a plurality of data blocks with different clustering centers, and calculating the direction vector of each data block to obtain a set of direction vectors; removing interference data blocks based on the set of direction vectors; performing frequency domain compression on the data blocks based on the direction vectors of the remaining data blocks to obtain frequency domain compressed data blocks as the result of data mining; wherein removing interference data blocks based on the set of direction vectors comprises: performing single-set semantic consistency query response analysis based on self-attention guidance on each direction vector in the set of direction vectors to determine whether the data block corresponding to each direction vector is an interference data block, and removing the interference data block.

[0008] According to another aspect of the present application, a power data mining system based on big data is provided, which comprises: a power data classification module, configured to obtain power data, perform recursive classification based on a tree structure on the power data to obtain a classification tree of the power data; a data clustering module, configured to perform data clustering processing on data values contained in all terminal nodes of the classification tree to obtain a plurality of data blocks with different clustering centers, and calculate the direction vector of each data block to obtain a set of direction vectors; an interference data block removal module, configured to remove interference data blocks based on the set of direction vectors; a data block frequency domain compression module, configured to perform frequency domain compression on the data blocks based on the direction vectors of the remaining data blocks to obtain frequency domain compressed data blocks as the result of data mining; The interference data block elimination module is configured to: respectively perform self-attention-guided single-set semantic consistency query response analysis on each direction vector in the set of direction vectors to determine whether the data block corresponding to each direction vector is an interference data block, and eliminate the interference data block.

[0009] Compared with the prior art, the power data mining method and system based on big data provided by the application first perform recursive classification and data clustering processing on power data, calculate the direction vector of each data block, and further introduce a semantic consistency analysis algorithm based on self-supervised learning, perform semantic consistency query response analysis on the direction vector of each data block and the set of direction vectors, mine the semantic consistency degree of each data block relative to the whole data set, identify and eliminate potential interference data blocks based on this, and then realize frequency domain compression of effective data blocks. In this way, interference data blocks that are difficult to detect by traditional statistical analysis methods can be effectively identified, providing a purer and higher-quality data basis for subsequent data mining. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description of embodiments of the present application taken in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of embodiments of the present application and constitute a part of the specification, together with the description, to explain the present application and, together with the description, serve to explain the present application. The drawings do not limit the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 A flowchart of the power data mining method based on big data according to the embodiments of the present application.

[0012] Figure 2 A data flow diagram of the power data mining method based on big data according to the embodiments of the present application.

[0013] Figure 3 A flowchart of sub-step S3 of the power data mining method based on big data according to the embodiments of the present application.

[0014] Figure 4 A flowchart of sub-step S32 of the power data mining method based on big data according to the embodiments of the present application.

[0015] Figure 5 A flowchart of sub-step S321 of the power data mining method based on big data according to the embodiments of the present application.

[0016] Figure 6 A flowchart of sub-step S322 of the power data mining method based on big data according to the embodiments of the present application.

[0017] Figure 7 Flow chart for sub-step S33 of the big data based power data mining method according to the embodiments of the present application.

[0018] Figure 8 Block diagram of the big data based power data mining system according to the embodiments of the present application. DETAILED DESCRIPTION

[0019] As shown in the present application and claims, unless the context clearly indicates otherwise, the words “a”, “an”, “one”, and / or “the” do not necessarily refer to only one, but can include more than one. In general, the terms “including”, “includes”, “containing”, “contains” and / or “containing”, “contains” are intended to be open-ended terms that specifically indicate the presence of the stated steps and elements in the resulting system, method, and / or device, but do not preclude the presence or addition of one or more other steps, elements, or elements.

[0020] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, however, any number of different modules can be used and run on the user terminal and / or server. The modules are merely illustrative, and different aspects of the system and method can use different modules.

[0021] Flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in sequence. On the contrary, various steps can be processed in reverse order or simultaneously, as needed. Other operations can also be added to these processes, or one or more steps can be removed from these processes.

[0022] In the following, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and are not all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein.

[0023] It is worth noting that in the present application, all actions of obtaining data are carried out in accordance with the data protection regulations of the country where the device is located, and with the authorization given by the owner of the corresponding device.

[0024] As mentioned in the background art, patent CN116304931A proposes a big data based power data mining method, which eliminates interference data blocks and performs frequency domain compression on effective data blocks by using various technical means such as recursive classification of power data, data clustering, direction vector consistency analysis, and frequency domain compression, to achieve more efficient data mining.

[0025] Specifically, the prior art refines data layer by layer through recursive classification rules, and extracts the direction vectors of data blocks in combination with singular value decomposition and clustering algorithms, and the basis for judging whether the data block is interference data is the average angle between the direction vectors. However, this method which only relies on the statistical characteristics of the direction vector is easily affected by data noise and outliers, and has low recognition accuracy. For example, in the power load mutation scene, some data blocks may deviate from the local clustering center but conform to the global change trend, and the existing method may misjudge them as interference data blocks, resulting in loss of effective information and affecting the accuracy of data mining. In view of the above technical problems, the present application proposes a power data mining method based on big data, which first performs recursive classification and data clustering processing on the power data, calculates the direction vector of each data block, and further introduces a semantic consistency analysis algorithm based on self-supervised learning, which analyzes the semantic consistency of each data block with respect to the whole data set by performing semantic consistency query response analysis on the direction vector of each data block and the direction vector set, to mine the semantic consistency degree of each data block with respect to the whole data set, and based on this, identify and eliminate potential interference data blocks, and then realize frequency domain compression of effective data blocks. In this way, interference data blocks that are difficult to detect by traditional statistical analysis methods can be effectively identified, providing a more pure and high-quality data basis for subsequent data mining.

[0026] Figure 1 A flowchart of the power data mining method based on big data according to an embodiment of the present application. Figure 2 A data flow diagram of the power data mining method based on big data according to an embodiment of the present application. As shown in Figure 1 and Figure 2 The power data mining method based on big data includes the following steps: S1, obtaining power data, performing recursive classification based on tree structure on the power data to obtain a classification tree of the power data; S2, performing data clustering processing on the data values contained in all terminal nodes of the classification tree to obtain a plurality of data blocks with different clustering centers, and calculating the direction vector of each data block to obtain a set of direction vectors; S3, eliminating interference data blocks based on the set of direction vectors; S4, performing frequency domain compression on the data blocks based on the direction vectors of the remaining data blocks, obtaining frequency domain compressed data blocks as the result of data mining.

[0027] In the above power data mining method based on big data, the step S1, acquiring power data, classifies the power data based on a tree structure to obtain a classification tree of the power data. It should be understood that the power data has a multi-source heterogeneous characteristic (such as power load data, power generation data, power supply quality data, power consumption data, and various types), and the internal correlation is complex and the noise distribution is uneven. Therefore, by recursively classifying the power data to construct a hierarchical classification tree, the data organization method is refined layer by layer, the original data can be disassembled into smaller subsets, and structured input is provided for subsequent clustering and semantic analysis. Specifically, in the classification process of the power data, different dimensional rules such as time rules, numerical rules, and type rules are alternately used to adapt to the multi-modal characteristics of the power data and avoid the key information fragmentation problem caused by a single classification dimension. For example, for a regional power load data set, the type rule is used to separate the instantaneous load and average load data in the first classification; the time rule is used to divide the instantaneous load data by hour in the second classification; the numerical rule is used to divide the high, medium, and low load intervals for each hour of data in the third classification, until each terminal node contains only a single data value, thereby constructing a complete classification tree. In this way, the classification tree maps the original power data into a multi-level structure, which not only retains the global distribution characteristics of the data (such as time trend, numerical range), but also realizes local fine-grained division (such as load fluctuation details in a specific time period), providing clear context information for subsequent data clustering mining.

[0028] In the process of specific implementation, considering a regional power load data set, the type rule can be used to distinguish instantaneous load and average load data in the initial stage. This division helps to understand the basic characteristics and variation trends of different types of loads. The time rule is further applied to the instantaneous load data, and the data is subdivided according to different time periods (such as every hour). This step can capture the specific patterns of load fluctuations over time. For example, at different times of the day, due to factors such as business activities and residential electricity usage habits, power load will show obvious periodic changes. By classifying these data according to the time dimension, the variation law of the load can be better understood and predicted.

[0029] Subsequently, for each hour segment of data, a numerical rule is employed to divide it into high, medium, and low load intervals. This fine-grained division not only reveals the details of load fluctuations within a specific time period, but also helps identify key factors that may affect system performance. For example, high loads during peak hours can cause grid overload, while low loads during off-peak hours can result in resource waste. By carefully dividing the load intervals, strong support can be provided for power company dispatching decisions. In addition, other types of rules can be introduced according to specific application scenarios, such as geographic location rules or user behavior rules. Using geographic location rules can classify power consumption characteristics in different regions, while user behavior rules can help understand the behavior patterns of different consumer groups and their impact on the overall power grid.

[0030] Through a series of operations, the final goal of each terminal node containing only a single data value is achieved, and a complete classification tree structure is constructed. This classification tree is not just a simple data grouping tool, but through mapping the original power data into a multi-level structure, it not only preserves the global distribution characteristics of the data, such as time trends and numerical ranges, but also achieves local fine-grained division, providing clear contextual information for subsequent data mining. In actual operation, the problem of noise and outliers in power data must be considered. Due to the wide range of sources and diverse forms of power data, some interference factors are inevitable. Therefore, in addition to focusing on the basic classification logic of the data, algorithms need to be designed to detect and reduce the impact of these interference factors during recursive classification.

[0031] For example, a round of data cleaning step is added after each classification operation, using statistical methods or other advanced technical means to identify and eliminate data points that deviate significantly from normal values. This not only improves the accuracy of the classification results, but also lays a solid foundation for subsequent data mining work. At the same time, in order to adapt to the dynamic characteristics of power data, the classification tree needs to be constantly updated and optimized. With the continuous inflow of new data, the classification tree also needs to adjust its structure accordingly to reflect the latest power system operation status in a timely manner. This process involves periodically re-evaluating existing classification rules and adding new rules or modifying existing rules based on actual conditions to ensure that the classification tree can accurately capture the latest trends in power data.

[0032] In the process of constructing the classification tree, alternating the use of rules of different dimensions is the key to achieving fine-grained data organization. For example, in some cases, in addition to the time and numerical rules mentioned above, seasonal rules can be introduced to consider the impact of different seasons on power load. During the summer, air conditioning is used frequently, which will cause a significant increase in power load; while in winter, the use of heating equipment also has an important impact on power demand. By introducing seasonal rules, this periodic change can be reflected in the classification tree, making the classification more in line with the actual situation. In addition, special factors such as holiday effects can also be considered, because people's activity patterns during holidays are usually different from weekdays, which has a direct impact on power load.

[0033] In the above power data mining method based on big data, the step S2, the data clustering processing is performed on the data values contained in all terminal nodes of the classification tree to obtain a plurality of data blocks with different clustering centers, and the direction vector of each data block is calculated to obtain a set of direction vectors. That is, since the data after recursive classification is still relatively dispersed, in order to mine the correlation and potential distribution rule between the data, the data clustering processing needs to be further performed on the data values contained in all terminal nodes of the classification tree, so as to aggregate the data values with similar characteristics into the same data block, forming a plurality of data blocks with different clustering centers. In this way, based on the clustering center of each data block, the main feature direction of each data block can be calculated, thereby reducing the redundancy of data representation and improving the efficiency of data mining. In the specific implementation process, first, the data values of the terminal nodes of the classification tree are obtained as the input data matrix of the clustering algorithm. Then, the input data matrix is normalized to make the mean value of each row of data 0 and the variance 1, so as to eliminate the dimensional difference between different data dimensions. Then, the singular value decomposition is performed on the normalized data matrix, and the principal component vectors corresponding to the first k singular values are retained to filter high-frequency noise. Secondly, based on the result of singular value decomposition, the decomposition clustering algorithm is used to cluster the principal component vectors to obtain a plurality of clustering centers and corresponding data blocks. Further, by normalizing the clustering center vector of each data block as the direction vector of the data block, a set of direction vectors can be obtained. Here, the clustering center of the data block is a statistical representation of its data distribution, and the direction vector as the normalized direction information carrier of the clustering center can reflect the dominant trend and correlation characteristics of the data block, and further reveal the relative position relationship and motion trend between the data blocks.

[0034] In the above power data mining method based on big data, the step S3 is to eliminate the interference data block based on the set of direction vectors. In a specific example of the present application, the step S3 includes: respectively performing self-attention guided single-set semantic consistency query response analysis on each direction vector in the set of direction vectors to determine whether the data block corresponding to each direction vector is an interference data block, and eliminating the interference data block. It should be understood that the prior art relies on the angle between the direction vectors of the data blocks to determine the interference data, and cannot accurately distinguish between data deviation caused by noise or abnormal values and normal change trend of the data itself (such as periodic load fluctuation and equipment fault transient signal). In this regard, the present application proposes a self-attention guided single-set semantic consistency query response analysis method, which takes the set of direction vectors as a whole context reference, finds the matching degree of the direction vector of each data block with the whole context, captures the semantic consistency of the data block in the global data distribution, and thus improves the accuracy of interference identification. Wherein, Figure 3 FIG. 3 is a flowchart of the step S3 of the power data mining method based on big data according to an embodiment of the present application. As shown in FIG. 3, the step S3 includes the steps of: S31, extracting the direction vector of the first data block from the set of direction vectors as a query direction vector; S32, inputting the query direction vector and the set of direction vectors into a direction vector single semantic query search engine to obtain a query direction vector semantic alignment retrieval result embedding coding vector; S33, determining whether the first data block is an interference data block based on the query direction vector semantic alignment retrieval result embedding coding vector. Figure 3

[0035] Specifically, the step S31 extracts the direction vector of the first data block from the set of direction vectors as a query direction vector. It should be understood that in the interference data block identification process, the semantic consistency of each data block with the whole data set needs to be evaluated one by one. Therefore, the present application selects the direction vector of the data block in the set of direction vectors as the current query direction vector to be analyzed in index order, and analyzes the deviation degree of the data block in the global semantic by querying and interacting with other direction vectors in the set, so as to identify the interference data.

[0036] ​Specifically, the step S32 inputs the query direction vector and the set of direction vectors into a direction vector single-body semantic query search engine to obtain a query direction vector semantic alignment retrieval result embedding coding vector. Specifically, the present application considers that the traditional method relies on the static calculation of the angle between direction vectors, and cannot capture the contextual association between direction vectors. In this regard, the present application further introduces a direction vector single-body semantic query search engine to process the query direction vector and the set of direction vectors, generate a query direction vector semantic alignment retrieval result embedding coding vector by dynamically learning the semantic interaction mode of the query direction vector and the entire set of direction vectors, and comprehensively represent the semantic position of the data block in the global data distribution and its associated mode with the overall data set. Wherein, Figure 4 The flow chart of the sub-step S32 of the big data-based power data mining method according to the embodiment of the present application is shown in FIG. 32. Figure 4 As shown in FIG. 32, the step S32 includes steps of: S321, performing semantic query response coding on the query direction vector and each direction vector in the set of direction vectors respectively to obtain a set of query direction vector semantic alignment retrieval score coding vectors; and S322, based on the feature set of the set of query direction vector semantic alignment retrieval score coding vectors from the distribution characteristics, adaptively gating and aggregating the set of query direction vector semantic alignment retrieval score coding vectors to obtain the query direction vector semantic alignment retrieval result embedding coding vector.

[0037] Figure 5 The flow chart of the sub-step S321 of the big data-based power data mining method according to the embodiment of the present application is shown in FIG. 33. Figure 5 As shown in FIG. 33, the step S321 includes steps of: S3211, performing deconvolution coding driven feature optimization on the query direction vector to obtain a query direction semantic enhancement coding vector; and S3212, performing single-body semantic query coding on the query direction semantic enhancement coding vector and each direction vector in the set of direction vectors respectively to obtain the set of query direction vector semantic alignment retrieval score coding vectors.

[0038] In one specific example of the present application, the step S3211 is expressed by the formula as follows: Wherein, represents the query direction vector, represents the query direction semantic enhancement coding vector, represents the deconvolution weight matrix, represents the deconvolution coding processing, represents the calculation norm.

[0039] That is, the feature expressiveness of the query direction vector is enhanced through deconvolution coding, ensuring that the enhanced query direction semantic enhancement coding vector can reach consistency in feature scale with each direction vector in the direction vector set in subsequent processing, creating a foundation for the comparability of features from different sources in the same semantic space, so as to ensure the numerical stability of subsequent calculations.

[0040] In a specific example of the present application, step S3212 includes: concatenating and fusing the query direction semantic enhancement encoding vector and the direction vector and inputting the result into a tanh function-based neural network layer to obtain the query direction vector semantic alignment retrieval score encoding vector, which is expressed as: in, represents a set of direction vectors, 、 、 and Respectively represent the first, second, and third sets of direction vectors and direction vectors, is the number of direction vectors, represents the tanh function, represents the bias term, represents the weight matrix of the neural network layer, Indicates cascade operation, express and The query direction vector is semantically aligned with the retrieval score encoding vector.

[0041] That is, the feature information of the query direction semantic enhancement coding vector and the direction vector is integrated through cascade fusion, and the nonlinear mapping capability of the neural network layer is utilized to make the output query direction vector semantic alignment retrieval score coding vector not only a simple scalar score, but also contain richer semantic association information such as different dimension scores representing semantic similarities at different levels, so as to comprehensively characterize the degree of matching between the two in the multi-dimensional semantic space, and automatically focus on the semantic dimensions that are strongly correlated with interference discrimination in the hierarchical feature interaction, suppress the influence of local dimensional offset caused by data noise on the matching degree calculation, so that the generated query direction vector semantic alignment retrieval score coding vector can accurately characterize the complex semantic consistency state between the direction vectors, and improve the robustness of interference data block recognition.

[0042] Figure 6 FIG is a flowchart of sub-step S322 of the power data mining method based on big data according to an embodiment of the present application. Figure 6As shown, the step S322 includes steps of: S3221, determining the monomer semantic matching degree of each query direction vector semantic alignment search score encoding vector in the set of query direction vector semantic alignment search score encoding vectors based on the feature set of the set of query direction vector semantic alignment search score encoding vectors from the distribution characteristics, to obtain a set of query direction vector semantic consistency monomer query matching degrees; S3222, based on the set of query direction vector semantic consistency monomer query matching degrees, gating aggregation is performed on the set of query direction vector semantic alignment search score encoding vectors to obtain the query direction vector semantic alignment search result embedding encoding vector.

[0043] Here, when the monomer semantic query unit implements monomer semantic query score encoding by fusing the features of the query direction semantic reinforcement encoding vector and the corresponding direction vector , it is expected that the dynamic representation mechanism of the feature space to the semantic query encoding space will be optimized, and the intrinsic alignment characteristics of the feature space and the semantic query encoding space will be enhanced, to improve the representation fidelity of the monomer semantic query score. Based on this, in one preferred example of the present application, the step S3221 includes: first, based on the interaction between the query direction semantic reinforcement encoding vector and each direction vector in the set of direction vectors, symmetric constraint optimization is performed on each corresponding query direction vector semantic alignment search score encoding vector to obtain a set of optimized query direction vector semantic alignment search score encoding vectors.

[0044] Specifically, if the concatenated features of the query direction semantic reinforcement encoding vector and the corresponding direction vector are defined as , that is , then first construct the interaction potential field vector of the mapping space: wherein, represents the query direction vector semantic consistency monomer query feature concatenated vector between and , represents a linear mapping matrix, represents the interaction potential vector between and .

[0045] That is, the concatenated feature ontology is taken as a time-varying feature base to obtain its curvature form in the space mapping process.

[0046] Then, intrinsic alignment is performed between the feature space and the semantic query encoding space: wherein, denotes canonical fixed parameters, denotes a covariance matrix, denotes corresponding optimized query direction vector semantic alignment retrieval score encoding vectors.

[0047] the final canonical fixed parameters follow an inverse convolution enhanced isomorphic approach to maintain symmetry, i.e.: and is a covariance matrix, i.e. let

[0048] Thus, in the case of interaction potential vector as an intrinsic alignment generator, the covariant mapping trajectory guaranteed by the metric invariant constraint significantly improves the representation accuracy of the monomer semantic query score.

[0049] Then, the contextual semantic correlation degree of each optimized query direction vector semantic alignment retrieval score encoding vector in the set of the optimized query direction vector semantic alignment retrieval score encoding vectors relative to other optimized query direction vector semantic alignment retrieval score encoding vectors is calculated as the monomer semantic matching degree to obtain a set of the query direction vector semantic consistency monomer query matching degrees, which is expressed by the formula as: wherein, denotes the th optimized query direction vector semantic alignment retrieval score encoding vector in the set of the optimized query direction vector semantic alignment retrieval score encoding vectors, denotes the transpose of vector, denotes the exponential function operation with base e, denotes the normalized exponential function, denotes denotes corresponding query direction vector semantic consistency monomer query matching degrees.

[0050] That is, by analyzing the contextual semantic correlation degree of the optimized query direction vector semantic alignment retrieval score encoding vector and other vectors, the semantic matching degree is dynamically adjusted, so that the model can understand the relative position and importance of each optimized query direction vector semantic alignment retrieval score encoding vector in the global semantic space, realize the relative importance judgment similar to the contrastive thinking, and then more accurately fuse the semantic consistency information between the query direction vector and each reference direction vector in the subsequent gating aggregation step, improve the query direction vector overall semantic consistency analysis efficiency and accuracy.

[0051] ​In one specific example of the present application, the step S3222 comprises: first, inputting the set of query directional vector semantic consistency monomer query matching degrees into the relationship gate agent module to obtain the set of query directional vector semantic consistency monomer query self-attention weights, which is expressed by the formula as follows: wherein, denotes the gating threshold, denotes the mask operation, denotes the corresponding query directional vector semantic consistency monomer query self-attention weight.

[0052] That is, by introducing the relationship gate agent module, the set of query directional vector semantic consistency monomer query matching degrees is taken as the input signal, and the adjustable characteristics of the gating mechanism are utilized to dynamically screen and reshape the key matching degree features closely related to the interference discrimination, filter out the low signal-to-noise ratio matching signals caused by data noise or redundant correlation, so that the subsequent global matching information aggregation process can accurately focus on the semantic dimensions closely related to the interference data block discrimination, while weakening the weight interference irrelevant to the true semantic consistency, thereby improving the decoupling ability of the complex semantic patterns of the power data.

[0053] Then, based on the set of query directional vector semantic consistency monomer query self-attention weights, the set of optimized query directional vector semantic alignment retrieval score encoding vectors is weighted aggregated to obtain the query directional vector semantic alignment retrieval result embedding encoding vector, which is expressed by the formula as follows: wherein, denotes the query directional vector semantic alignment retrieval result embedding encoding vector.

[0054] That is, the set of query directional vector semantic consistency monomer query self-attention weights is utilized to differentially weight the optimized query directional vector semantic alignment retrieval score encoding vectors, actively select and filter in information integration, and realize comprehensive integration of multi-dimensional semantic information through weighted aggregation, so as to form the query directional vector semantic alignment retrieval result embedding encoding vector which comprehensively reflects the semantic association between the query data block and the overall set, thereby avoiding the effective information deletion or noise retention caused by isolated analysis, improving the accuracy and robustness of the interference data block identification, and providing more accurate comprehensive judgment basis for the identification of the interference data block.

[0055] Specifically, the step S33 determines whether the first data block is an interference data block based on the query directional vector semantic alignment retrieval result embedding encoding vector, wherein, Figure 7A flowchart of sub-step S33 of the big data-based power data mining method according to the embodiments of the present application is shown in FIG. 3. As shown in FIG. 3, the step S33 includes steps of: S331, performing semantic consistency explicit decoding on the query direction vector semantic alignment retrieval result embedding code vector to obtain a semantic consistency measurement coefficient; and S332, determining whether the first data block is an interference data block based on a comparison between the semantic consistency measurement coefficient and a preset threshold value. Figure 7

[0056] More specifically, the step S331 performs semantic consistency explicit decoding on the query direction vector semantic alignment retrieval result embedding code vector to obtain a semantic consistency measurement coefficient. It should be understood that, in order to quantitatively evaluate the semantic consistency of the query direction vector in the global data distribution, the present application designs a semantic consistency explicit decoder to perform decoding processing on the query direction vector semantic alignment retrieval result embedding code vector. Specifically, the semantic consistency explicit decoder is based on a multi-layer perception structure, and through layer-by-layer analysis of high-level semantic features in the query direction vector semantic alignment retrieval result embedding code vector, it is gradually mapped to a low-dimensional semantic consistency measurement space, and finally outputs a semantic consistency measurement coefficient between 0 and 1. The closer the semantic consistency measurement coefficient is to 1, the higher the semantic consistency of the query direction vector with the overall data set, i.e., the data block is more consistent with the normal change trend of the global data; on the contrary, if the coefficient is close to 0, it indicates that the semantic consistency of the data block with the overall data set is low, and there may be noise or outliers, and therefore it is considered as a potential interference data block. Through the explicit decoding process, a clear quantitative index can be provided for the judgment of the interference data block.

[0057] More specifically, the step S332 determines whether the first data block is an interference data block based on a comparison between the semantic consistency measurement coefficient and a preset threshold value, and removes the interference data block. Specifically, the preset threshold value is a limit value set according to a large number of experiments and experience, and a comparison between the semantic consistency measurement coefficient and the threshold value can determine whether the consistency of the data block with the overall data set is within an acceptable range, so as to judge whether it is an interference data block, so as to remove the interference data, improve the quality and reliability of the data, and avoid the negative impact of the interference data on the analysis and decision of the power system. For example, if the semantic consistency measurement coefficient is lower than the preset threshold value, it indicates that the semantic consistency of the first data block corresponding to the query direction vector with the direction vector set is poor, and the characteristics of the data block are significantly different from the overall data set, and therefore it is determined as an interference data block; on the contrary, it is considered that the data block is consistent with the overall data characteristics, and is not an interference data block.

[0058] ​In the above power data mining method based on big data, the step S4 is to perform frequency domain compression on the data block based on the direction vector of the remaining data block, and obtain a frequency domain compressed data block as the result of data mining. It can be understood that the frequency domain compression technique can reduce the size and complexity of the data while preserving the main features of the original signal, providing efficient data support for the stable operation and reasonable planning of the power system. Specifically, first, perform fast Fourier transform (FFT) on each data block to convert the time domain signal to a frequency domain representation. Then, divide the frequency domain representation into multiple subbands, and perform absolute value operation on each subband to obtain its amplitude spectrum. Next, divide the amplitude spectrum into multiple intervals, calculate the average value and standard deviation of the amplitude spectrum in each interval, and divide and merge the frequency domain intervals according to the pre-set tolerance. If the standard deviation of an interval is greater than the product of the tolerance and the average value, it means that the signal in this interval changes greatly and needs to be further subdivided to ensure that no key details are lost, then the interval is divided into two subintervals, and the average value of the new subintervals is calculated; otherwise, the interval remains unchanged. After completing all the necessary subdivision, the average value of each subinterval is taken as the compressed subband representation, and all the compressed subbands are combined to form the final frequency domain compressed data block as the result of data mining.

[0059] Specifically, fast Fourier transform (FFT) makes the signal changes that are difficult to understand intuitively in the time dimension present clear structural features in the frequency dimension. After FFT transformation, the frequency domain representation obtained not only contains all the information of the original signal, but also presents it in a more convenient form for analysis. This conversion provides a basis for subsequent amplitude spectrum calculation, making it possible to more accurately capture the key components in the signal. Next, the frequency domain representation is divided into multiple subbands, each subband representing the signal characteristics in different frequency ranges. Absolute value operation is performed on each subband to obtain its amplitude spectrum. The amplitude spectrum reflects the intensity distribution of the signal at each frequency point, and is crucial for identifying important frequency components in the signal. In this way, the main frequency components of the signal can be effectively separated from noise or secondary components, laying the foundation for further data compression.

[0060] Subsequently, calculating the mean and standard deviation of the amplitude spectrum within each bin allows for the assessment of the degree of variation in the signal across different frequency bins, in order to determine which bins require further subdivision, ensuring that critical details are not lost. The division and merging of frequency bins based on a pre-set tolerance is possible. The choice of tolerance is based on a trade-off between understanding the characteristics of the signal and the desired compression effect. If the standard deviation within a bin is greater than the product of the tolerance and the mean, it indicates that the signal within that bin varies greatly, meaning that important information may be present and needs to be preserved, thus the bin needs to be further subdivided to ensure that all critical details are considered. Conversely, if the standard deviation does not exceed the set threshold, it indicates that the signal within that bin is relatively stable and does not need further subdivision, remaining as is.

[0061] After all necessary subdivisions are completed, the mean of each sub-bin is taken as the compressed sub-band representation. In this way, the data volume can be significantly reduced while ensuring data quality. This method takes into account both the overall characteristics of the signal and local details, achieving a balance between compression ratio and information fidelity. All compressed sub-bands are combined to form the final frequency-domain compressed data block. This process is not simply a data reduction, but through carefully designed algorithm logic, it ensures that the compressed data can still reflect the core characteristics of the original signal.

[0062] During the entire frequency-domain compression process, the choice of tolerance must fully consider the characteristics of different types of data in the power system and the expected application scenarios. A higher tolerance may result in excessive information loss, while a lower tolerance may not achieve effective data compression. In addition, the number and width of sub-bands need to be adjusted flexibly according to actual needs to adapt to different data sets and analysis purposes. For example, in some cases, the number of sub-bands in a certain frequency segment can be appropriately increased to highlight the importance of that segment; in other cases, the requirements for non-critical frequency segments can be appropriately relaxed to simplify the processing flow.

[0063] It is worth noting that before performing frequency-domain compression, data preprocessing is usually required, including removing outliers, smoothing, and other steps to improve the quality of the compression results. These preprocessing measures help to eliminate potential interference factors, making the frequency-domain compression more ideal. At the same time, in order to verify the effectiveness of the compression results, a comparative analysis of the data before and after compression should be performed to check whether the desired compression ratio and key information preservation have been achieved.

[0064] To sum up, the power data mining method based on big data according to the embodiments of the present application is illustrated, which firstly performs recursive classification and data clustering processing on the power data, calculates the direction vector of each data block, and further introduces a semantic consistency analysis algorithm based on self-supervised learning, performs semantic consistency query response analysis on the direction vector of each data block and the direction vector set, to mine the semantic consistency degree of each data block relative to the whole data set, based on which the potential interference data block is identified and removed, and then the frequency domain compression of the effective data block is realized. In this way, the interference data block that is difficult to be found by the traditional statistical analysis method can be effectively identified, and purer and higher quality data basis is provided for subsequent data mining.

[0065] Further, a power data mining system based on big data is also provided.

[0066] Figure 8 A block diagram of the power data mining system based on big data according to the embodiments of the present application is shown. As shown in Figure 8 The power data mining system based on big data 100 according to the embodiments of the present application includes: a power data classification module 110, configured to acquire power data, perform recursive classification based on a tree structure on the power data to obtain a classification tree of the power data; a data clustering module 120, configured to perform data clustering processing on the data values contained in all terminal nodes of the classification tree to obtain a plurality of data blocks with different clustering centers, and calculate the direction vector of each data block to obtain a set of direction vectors; an interference data block removal module 130, configured to remove interference data blocks based on the set of direction vectors; and a data block frequency domain compression module 140, configured to perform frequency domain compression on the data blocks based on the direction vectors of the remaining data blocks, to obtain frequency domain compressed data blocks as the result of data mining. The interference data block removal module 130 is further configured to: perform self-attention guided single-set semantic consistency query response analysis on each direction vector in the set of direction vectors respectively to determine whether the data block corresponding to each direction vector is an interference data block, and remove the interference data block.

[0067] Here, those skilled in the art can understand that the specific operations of each module in the power data mining system based on big data described above have been described in detail above with reference to the description of the power data mining method based on big data Figures 1 to 7 , and therefore the repeated description thereof will be omitted.

[0068] The above describes the basic principles of the present application in combination with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present application are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present application. In addition, the specific details of the above embodiments are only for the purpose of illustration and understanding, and are not limiting, and the above details do not limit the present application to the above specific details.

[0069] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments. In the several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiments described above are only schematic, and the unit division is only a logical function division, and there can be other division ways in actual implementation. The unit described as a separate component can or can not be physically separated, and the components displayed as units can be or can not be physical units, i.e. they can be located in one place, or distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to the actual needs to achieve the purposes of the embodiments.

[0070] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and range of equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be considered as limiting the claims to which they relate.

[0071] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The plurality of units stated in the system claims can also be implemented by one unit through software or hardware.

[0072] Finally, it should be noted that the above description has been given for the purpose of illustration and description. In addition, the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application.

Claims

1. A power data mining method based on big data, characterized in that: include: Acquire power data, and recursively classify the power data based on a tree structure to obtain a classification tree for the power data; Performing data clustering processing on the data values ​​contained in all terminal nodes of the classification tree to obtain a plurality of data blocks with different cluster centers, and calculating the direction vector of each data block to obtain a set of direction vectors; Eliminating interference data blocks based on the set of direction vectors; Perform frequency domain compression on the data blocks based on the direction vectors of the remaining data blocks, and obtain frequency domain compressed data blocks as the result of data mining; Among them, eliminating interference data blocks based on the set of directional vectors includes: performing self-attention-guided monomer-set semantic consistency query response analysis on each directional vector in the set of directional vectors to determine whether the data block corresponding to each directional vector is an interference data block, and eliminating the interference data block.

2. The power data mining method based on big data according to claim 1, characterized in that: Performing a self-attention-guided single-set semantic consistency query response analysis on each direction vector in the set of direction vectors to determine whether a data block corresponding to each direction vector is an interference data block, including: Extracting the direction vector of the first data block from the set of direction vectors as a query direction vector; Inputting the query direction vector and the set of direction vectors into a direction vector monomer semantic query search engine to obtain a query direction vector semantic alignment retrieval result embedding coding vector; Based on the query direction vector semantic alignment retrieval result embedded coding vector, it is determined whether to use the first data block as an interference data block.

3. The power data mining method based on big data according to claim 2, characterized in that: Inputting the query direction vector and the set of direction vectors into a direction vector monomer semantic query search engine to obtain a query direction vector semantic alignment retrieval result embedding coding vector, including: Performing semantic query response encoding on the query direction vector and each direction vector in the set of direction vectors to obtain a set of query direction vector semantic alignment retrieval score encoding vectors; Based on the feature set self-distribution characteristics of the set of query direction vector semantic alignment retrieval score encoding vectors, the set of query direction vector semantic alignment retrieval score encoding vectors is adaptively gated and aggregated to obtain the query direction vector semantic alignment retrieval result embedding encoding vector.

4. The power data mining method based on big data according to claim 3, characterized in that: Performing semantic query response encoding on the query direction vector and each direction vector in the set of direction vectors to obtain a set of query direction vector semantic alignment retrieval score encoding vectors, including: performing deconvolution coding driven feature optimization on the query direction vector to obtain a query direction semantic enhancement coding vector; The query direction semantic enhancement encoding vector is respectively subjected to monomer semantic query encoding with each direction vector in the set of direction vectors to obtain a set of query direction vector semantic alignment retrieval score encoding vectors.

5. The power data mining method based on big data according to claim 4, characterized in that: Performing monomer semantic query encoding on the query direction semantic enhancement encoding vector and each direction vector in the set of direction vectors to obtain a set of query direction vector semantic alignment retrieval score encoding vectors, including: The query direction semantic enhancement encoding vector and the direction vector are cascaded and fused, and then input into a neural network layer based on a tanh function to obtain the query direction vector semantic alignment retrieval score encoding vector.

6. The power data mining method based on big data according to claim 5, characterized in that: Based on the self-distribution characteristics of the feature set of the set of query direction vector semantic alignment retrieval score encoding vectors, adaptively gated aggregation is performed on the set of query direction vector semantic alignment retrieval score encoding vectors to obtain the query direction vector semantic alignment retrieval result embedding encoding vector, including: Based on the feature set self-distribution characteristics of the set of query direction vector semantic alignment retrieval score encoding vectors, determining the monomer semantic matching degree of each query direction vector semantic alignment retrieval score encoding vector in the set of query direction vector semantic alignment retrieval score encoding vectors to obtain a set of query direction vector semantic consistency monomer query matching degrees; Based on the set of query direction vector semantic consistency monomer query matching degrees, gated aggregation is performed on the set of query direction vector semantic alignment retrieval score encoding vectors to obtain the query direction vector semantic alignment retrieval result embedding encoding vector.

7. The power data mining method based on big data according to claim 6, characterized in that: Determining the monomer semantic matching degree of each query direction vector semantic alignment retrieval score encoding vector in the set of query direction vector semantic alignment retrieval score encoding vectors based on the feature set self-distribution characteristics of the set of query direction vector semantic alignment retrieval score encoding vectors to obtain a set of query direction vector semantic consistency monomer query matching degrees, including: Based on the interaction between the query direction semantic enhancement encoding vector and each direction vector in the set of direction vectors, symmetry-constrained optimization is performed on each corresponding query direction vector semantic alignment retrieval score encoding vector to obtain a set of optimized query direction vector semantic alignment retrieval score encoding vectors; The contextual semantic association of each optimized query direction vector semantic alignment retrieval score encoding vector in the set of the optimized query direction vector semantic alignment retrieval score encoding vector relative to other optimized query direction vector semantic alignment retrieval score encoding vectors is calculated as the monomer semantic matching degree to obtain the set of query direction vector semantic consistency monomer query matching degrees.

8. The power data mining method based on big data according to claim 7, characterized in that: Based on the set of query direction vector semantic consistency monomer query matching degrees, gated aggregation is performed on the set of query direction vector semantic alignment retrieval score encoding vectors to obtain the query direction vector semantic alignment retrieval result embedding encoding vector, including: Inputting the set of query direction vector semantic consistency monomer query matching degrees into the relation gating proxy module to obtain a set of query direction vector semantic consistency monomer query self-attention weights; Based on the set of query direction vector semantic consistency monomer query self-attention weights, the set of optimized query direction vector semantic alignment retrieval score encoding vectors is weighted aggregated to obtain the query direction vector semantic alignment retrieval result embedding encoding vector.

9. The power data mining method based on big data according to claim 8, characterized in that: Determining whether to use the first data block as an interference data block based on the semantic alignment of the query direction vector and the embedding of the retrieval result into a coding vector includes: Performing semantic consistency explicit decoding on the query direction vector semantic alignment retrieval result embedding encoding vector to obtain a semantic consistency measurement coefficient; Based on a comparison between the semantic consistency measurement coefficient and a preset threshold, it is determined whether to use the first data block as an interference data block.

10. A power data mining system based on big data, characterized in that: include: The power data classification module is used to obtain power data and perform recursive classification on the power data based on a tree structure to obtain a classification tree for the power data; A data clustering module is used to perform data clustering processing on the data values ​​contained in all terminal nodes of the classification tree to obtain multiple data blocks with different cluster centers, and calculate the direction vector of each data block to obtain a set of direction vectors; an interference data block elimination module, configured to eliminate interference data blocks based on the set of direction vectors; A data block frequency domain compression module is used to perform frequency domain compression on the data block based on the direction vectors of the remaining data blocks, and obtain a frequency domain compressed data block as a result of data mining; Among them, the interference data block elimination module is used to: perform self-attention-guided monomer-set semantic consistency query response analysis on each direction vector in the set of direction vectors to determine whether the data block corresponding to each direction vector is an interference data block, and eliminate the interference data block.

Images