Data fusion sharing method for multi-source heterogeneous power big data

Abstract

The invention provides a data fusion sharing method for multi-source heterogeneous power big data, which comprises the following steps: when receiving multi-source heterogeneous power data, carrying out dynamic protocol adaptation, and converting multi-source heterogeneous power data formats of different protocols into a current system internal standard format; performing data quality evaluation on the data converted into the standard format in the current system, and when a quality evaluation result does not meet a quality requirement, performing data restoration; analyzing the data passing the data quality evaluation or restoration, determining the correlation among different data sources, and fusing the data according to the correlation among the different data sources; and performing distributed storage on the fused data for distributed indexing and access. By implementing the method and the system, efficient fusion and sharing of multi-source heterogeneous data can be realized, the intelligent level of data management and analysis of a power system is improved, and the requirements of an intelligent power grid on data instantaneity, accuracy and safety are met.

Description

Technical Field

[0001] The present invention belongs to the technical field of power data fusion, and in particular relates to a data fusion and sharing method for multi-source heterogeneous power big data. Background Art

[0002] With the rapid development of the power industry and the continuous improvement of its intelligence level, the amount of data generated by the power system is becoming increasingly large and complex. These data come from various sources, including power equipment operation data, user power consumption data, environmental sensor data, and business system data. These data not only have diverse structures, but also different formats, forming typical multi-source heterogeneous data. Therefore, how to efficiently integrate and share these data has become an important challenge facing the power industry. At present, the existing power data processing system has many deficiencies in data acquisition, preprocessing, storage, and sharing, and it is difficult to meet the requirements of smart grids for data real-time, accuracy, and security. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a data fusion and sharing method and device for multi-source heterogeneous power big data to meet the requirements of smart grid for data real-time, accuracy and security.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] The present invention provides a data fusion and sharing method for multi-source heterogeneous power big data, comprising: when receiving multi-source heterogeneous power data, dynamic protocol adaptation is performed to convert the multi-source heterogeneous power data formats of different protocols into the internal standard format of the current system; data quality assessment is performed on the data converted into the internal standard format of the current system, and when the quality assessment result does not meet the quality requirements, data repair is performed; data that has passed the data quality assessment or repair is analyzed to determine the correlation between different data sources, and data is fused according to the correlation of different data sources; the fused data is distributedly stored for distributed indexing and access to achieve data sharing.

[0006] Optionally, when multi-source heterogeneous power data is received, dynamic protocol adaptation is performed to convert the multi-source heterogeneous power data formats of different protocols into the current system internal standard format, including: using a protocol identification algorithm to scan the multi-source heterogeneous power data source to determine the protocol type of the multi-source heterogeneous power data; querying the protocol adapter of the corresponding protocol type in the protocol management module, and loading the protocol adapter using a dynamic loader; receiving multi-source heterogeneous power data through the protocol adapter, and sending the multi-source heterogeneous power data to the protocol converter; the protocol converter converts the multi-source heterogeneous power data formats of different protocols into the current system internal standard format.

[0007] Optionally, perform data quality assessment on the data converted into the current system's internal standard format. When the quality assessment result does not meet the quality requirements, perform data repair, including: performing statistical analysis on the data to detect outliers, missing values, and duplicate values; when outliers exist, use a regression analysis model to make predictions and replace the outliers with the predicted values; when missing values ​​exist, use linear interpolation to fill them; when duplicate values ​​exist, use fingerprint recognition technology to remove duplicates.

[0008] Optionally, the data that has passed the data quality assessment or repair is analyzed to determine the correlation between different data sources, and the data is fused according to the correlation between different data sources, including: extracting feature vectors of the data that has passed the data quality assessment or repair; determining the matching degree between the data sources of multi-source heterogeneous data according to the feature vectors; and fusing the data from different data sources according to a data fusion algorithm based on the matching degree.

[0009] Optionally, determining the matching degree between data sources of multi-source heterogeneous data according to the feature vector includes: calculating the cosine similarity between the data sources represented by the feature vector according to the cosine similarity; determining the combination of data sources whose cosine similarity is higher than a target threshold according to the size of the cosine similarity; and determining the matching degree of the combination of data sources according to the Euclidean distance:

[0010]

[0011] Among them, M ij represents the matching degree of the data source combination, d(v i ,v j ) is the Euclidean distance between data sources.

[0012] Optionally, determining the matching degree between the data sources of the multi-source heterogeneous data according to the feature vector includes: inputting the feature vector into a pre-trained neural network model to determine the matching degree between the data sources of the multi-source heterogeneous data, and the neural network model training process includes: inputting the true matching degree label and the data source into the initial neural network model, using the gradient descent method, iteratively updating the neural network model parameters to minimize the loss function until the preset accuracy requirement is met, and obtaining the trained neural network model; wherein the loss function is defined as:

[0013]

[0014] Among them, L(θ) is the loss function of the model parameter θ, M ij and are the true matching degree and the matching degree predicted by the model, respectively, and N represents the number of samples;

[0015] The gradient update formula is:

[0016]

[0017] Among them, θ t is the model parameter of the tth iteration, η is the learning rate, is the gradient of the loss function.

[0018] Optionally, the fused data is distributedly stored for distributed indexing and access to achieve data sharing, including: calculating the hash value of the physical storage node and mapping the physical storage node to the hash ring; dividing the fused data into multiple fragments, calculating the hash value of each fragment data, locating it on the hash ring according to the hash value, and determining its physical storage node; when a retrieval request is received, using access control lists and role authority management technology to verify the retrieval request; when the verification is passed, using a distributed hash table to index the distributed stored data.

[0019] Optionally, the hash value of the storage node is calculated and the storage node is mapped to a hash ring, including: creating multiple virtual nodes on the hash ring for each physical storage node and associating the physical storage node with the multiple virtual nodes; dividing the fused data into multiple fragments, calculating the hash value of each fragment data, locating it on the hash ring according to the hash value, and determining its storage node, including: dividing the fused data into multiple fragments, calculating the hash value of each fragment data, searching for the first virtual node in a clockwise direction, and storing the fragment data on the physical storage node corresponding to the virtual node.

[0020] Optionally, a data fusion and sharing method for multi-source heterogeneous power big data also includes: initializing the weight of the fused data source; performing weighted averaging on the weight of the data source and the feature vector of the data source to obtain an initial fusion result; optimizing the weight of the fused data according to the gradient descent method to reduce the loss function value until it converges to a target threshold or the number of iterations reaches a preset upper limit, stopping iteration, and outputting the final fusion result and the optimized weight;

[0021] Among them, the loss function is:

[0022]

[0023] Among them, L(w) is the loss function, w=[w1,w2,...,w n ] is the weight of all fused data sources, v i is the feature vector of the ith data source, y j is the reference result, v fusion,j is the result after fusion, m represents the number of samples;

[0024] The method of optimizing the weight of fusion data according to the gradient descent method is as follows:

[0025]

[0026] Where η is the learning rate, is the weight value at the tth iteration, is the weight value at the t+1th iteration, is the gradient of the loss function.

[0027] Optionally, when there are outliers, a regression analysis model is used for prediction, and the predicted value is substituted for the outlier, including: using the following linear regression formula to determine the predicted value:

[0028]

[0029] in, is the predicted value, β0 is the intercept, β1,β2,...,β p are regression coefficients, x1,x2,...,x p As the independent variable, the regression coefficients β0, β1, β2, ..., β p , and then replace the outliers with the output of the regression model;

[0030] When there are missing values, linear interpolation is used to fill them, including:

[0031]

[0032] Among them, y i is the point to be interpolated, y i-1 and i+1 is a known adjacent data point, x i 、x i-1 、x i+1 is the corresponding independent variable value;

[0033] When duplicate values ​​exist, they are removed through fingerprint recognition technology, including:

[0034]

[0035] Among them, F(x) is the fingerprint of data x, h i (x) is a locality sensitive hash function, a i is the weight;

[0036] By calculating fingerprints and comparing fingerprint similarities, duplicate data can be removed.

[0037] The present invention provides a data fusion and sharing method for multi-source heterogeneous power big data, which adopts dynamic protocol adaptation and has strong compatibility and can support multiple data sources and protocols, thereby solving the problem in traditional solutions that each addition of a new protocol requires modification of the system, which increases development and maintenance costs, or the problem of difficulty in docking the system due to protocol incompatibility; the method has high flexibility and, through a dynamic loading mechanism, only loads the corresponding protocol adapter when needed, thereby avoiding unnecessary waste of resources; the method is easy to expand and supports the expansion and update of the protocol adapter to meet the needs of new data sources in the future, and fuses data through the correlation of different data sources, thereby achieving efficient fusion and sharing of multi-source heterogeneous data, improving the intelligent level of power system data management and analysis, and meeting the requirements of smart grids for data real-time, accuracy and security.

[0038] Other advantages, objectives and features of the present invention will be described in the following description and will be apparent to those skilled in the art to some extent, or those skilled in the art may be taught from the practice of the present invention. The objectives and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0040] Figure 1 A specific example flow chart of a data fusion and sharing method of multi-source heterogeneous power big data in the present invention;

[0041] Figure 2 A virtual structure diagram of the dynamic protocol adaptation system in the present invention;

[0042] Figure 3 This is a specific example flow chart of dynamic protocol adaptation from the data flow perspective of the present invention;

[0043] Figure 4 The present invention uses protocol feature matching and signal analysis technology to achieve high-precision protocol recognition code design diagram;

[0044] Figure 5 It is the code design diagram of the dynamic loading algorithm in the present invention;

[0045] Figure 6 A code design diagram for converting the format of received data by the protocol conversion module in the present invention;

[0046] Figure 7 It is a virtual structural diagram of the data quality assessment and repair module in the present invention.

[0047] Figure 8This is a code design diagram for the functions implemented by the data quality assessment and repair module in the present invention.

[0048] Fig. 9 The figure is a specific example flow chart of data repair in the present invention.

[0049] Fig.10 It is a virtual structure diagram of the intelligent data fusion module in the present invention. DETAILED DESCRIPTION

[0050] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0052] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0053] This embodiment provides a data fusion and sharing method for multi-source heterogeneous power big data. Figure 1 As shown, including:

[0054] S101, when receiving multi-source heterogeneous power data, dynamic protocol adaptation is performed to convert the multi-source heterogeneous power data formats of different protocols into the current system internal standard format;

[0055] S102, performing data quality assessment on the data converted into the current system internal standard format, and performing data repair when the quality assessment result does not meet the quality requirements;

[0056] S103, analyzing the data that has passed the data quality assessment or repair, determining the correlation between different data sources, and fusing the data according to the correlation between the different data sources;

[0057] S104, distributively storing the merged data for distributed indexing and access to achieve data sharing.

[0058] Exemplarily, when multi-source heterogeneous power data is received, a protocol identification algorithm is used to scan the multi-source heterogeneous power data source to determine the protocol type of the multi-source heterogeneous power data; a protocol adapter of the corresponding protocol type is queried in the protocol management module, and the protocol adapter is loaded using a dynamic loader; the multi-source heterogeneous power data is received through the protocol adapter, and the multi-source heterogeneous power data is sent to a protocol converter; the protocol converter converts the multi-source heterogeneous power data formats of different protocols into the current system internal standard format.

[0059] From a modular perspective, dynamic protocol adaptation systems, such as Figure 2 As shown, it consists of a protocol management module, a protocol adapter interface, a dynamic loader, and a protocol converter. The design purpose and implementation method of each module are as follows: Protocol management module: manages and maintains all protocols supported in the system, and provides functions such as protocol registration, deregistration, and query. The protocol management module records all supported protocols through a protocol registry, and provides the loading and query functions of protocol adapters through interfaces. Protocol adapter interface: defines the standard interface that all protocol adapters need to implement, including methods such as connection, data reception and transmission, and disconnection. This interface ensures the consistency of different protocol adapters and facilitates dynamic loading and replacement. Dynamic loader: implements dynamic loading through Python's importlib module, and loads and configures the required protocol adapters in real time according to the information provided by the protocol management module. Protocol converter: uniformly converts the data formats of different protocols to make them conform to the standard data format within the system, and implements efficient data format conversion through a rule engine or mapping table.

[0060] From the data flow perspective, Figure 3 As shown in the figure, the dynamic protocol adaptation system is first initialized. When receiving multi-source heterogeneous power data, the protocol management module is started. First, the connected data source is scanned and identified. The protocol identification algorithm determines the type of protocol used by scanning the communication signal of the data source, and uses protocol feature matching and signal analysis technology to achieve high-precision protocol identification. For specific code design, see Figure 4 Then the dynamic loading algorithm queries the corresponding protocol adapter from the protocol management module according to the protocol identification result, loads the adapter class through the dynamic loader, and registers it into the system after instantiation to ensure that the system can effectively communicate with the data source. For the specific code design, see Figure 5 Then the protocol conversion module converts the format of the received data, and uses the rule engine to implement the rule-based conversion logic, converting the data formats of different protocols into the internal standard format of the system to ensure the uniformity of the data, so that the data of different protocols can be uniformly stored and processed. The algorithm code design is as follows Figure 6 .

[0061] After being converted into the data in the current system internal standard format, the data is input into the data quality assessment and repair module, such as Figure 7 As shown in FIG, the data quality assessment and repair module includes a data assessment module and an automatic repair module. Figure 8 As shown, this embodiment provides a code design diagram of the functions implemented by the above-mentioned data quality assessment and repair module.

[0062] The data evaluation module can perform data quality evaluation on the data converted into the current system internal standard format by judging the integrity, accuracy and consistency of the data based on statistical analysis and anomaly detection algorithms. Specifically, statistical methods such as descriptive statistics, probability distribution, hypothesis testing, etc. can be used to analyze the distribution, central tendency, degree of dispersion of the data and other methods to evaluate the integrity, accuracy and consistency of the data. When the quality evaluation results do not meet the quality requirements, that is, any one or more of the integrity, accuracy and consistency are not up to standard, the automatic repair module performs data repair. The methods for repairing the low-quality data found can include missing value filling, outlier correction, duplicate data deletion, etc. This method can discover and autonomously repair most data problems through data quality evaluation and automatic repair, reducing dependence on manual intervention and improving repair efficiency.

[0063] Furthermore, in order to better monitor data quality, this embodiment Figure 7 As shown, it also includes a quality monitoring module that continuously monitors data quality and triggers an alarm when the quality deteriorates. System administrators can view the data quality status in real time, and promptly identify and handle data quality issues to ensure data reliability, as well as the consistency and integrity of data throughout the entire processing process.

[0064] Data fusion based on the correlation of different data sources is achieved through in-depth analysis and intelligent matching of multi-source heterogeneous data to achieve efficient data fusion. Data fusion can be composed of data preprocessing module, intelligent matching module and fusion optimization module. The data preprocessing module performs standardization processing on the original data, including cleaning, normalization, format conversion, etc., to ensure that data from different data sources can enter the fusion process in a unified format. The intelligent matching module uses machine learning algorithms to analyze and match the correlation between different data sources, discover and fuse the same or similar data. The fusion optimization module improves the accuracy and consistency of data after fusion and reduces data redundancy through algorithm optimization.

[0065] The fused data is shared through distributed data sharing. Specifically, by building a distributed data sharing network, efficient sharing and access of data between different nodes is achieved. First, distributed storage is performed to store the fused data on multiple nodes of the distributed network to improve data availability and fault tolerance. Data indexing and retrieval: create an index for each data fragment and distribute it on each node. Users can quickly find the required data through the retrieval interface. Access control: through access control policies, limit the access rights of different users and nodes to data to ensure secure data sharing.

[0066] The present embodiment provides a data fusion and sharing method for multi-source heterogeneous power big data, which adopts dynamic protocol adaptation and has strong compatibility and can support multiple data sources and protocols. It solves the problem in traditional solutions that each addition of a new protocol requires modification of the system, which increases development and maintenance costs, or the problem of difficulty in docking due to protocol incompatibility of the system; it has high flexibility and, through a dynamic loading mechanism, only loads the corresponding protocol adapter when needed, avoiding unnecessary waste of resources; it is easy to expand and supports the expansion and update of protocol adapters to meet the needs of new data sources in the future, and fuses data through the correlation of different data sources, which can achieve efficient fusion and sharing of multi-source heterogeneous data, improve the intelligence level of power system data management and analysis, and meet the requirements of smart grids for data real-time, accuracy and security.

[0067] As an optional implementation, data quality assessment is performed on the data converted into the current system internal standard format. When the quality assessment result does not meet the quality requirements, data repair is performed, including: statistical analysis of the data to detect outliers, missing values, and duplicate values; when outliers exist, a regression analysis model is used for prediction, and the predicted values ​​are used to replace the outliers; when missing values ​​exist, linear interpolation is used to fill them; when duplicate values ​​exist, fingerprint recognition technology is used to deduplicate them.

[0068] For example, Fig. 9 As shown, the process of data repair includes: first accessing the data, performing statistical analysis on the data, and detecting whether there are missing values. If there are missing values, linear interpolation is used to fill them, including:

[0069]

[0070] Among them, y i is the point to be interpolated, y i-1 and i+1 is a known adjacent data point, x i 、x i-1 、x i+1 is the corresponding independent variable value;

[0071] When there are outliers, the following linear regression formula is used to determine the predicted value, replacing the outlier with the predicted value:

[0072]

[0073] in, is the predicted value, β0 is the intercept, β1,β2,...,β p are regression coefficients, x1,x2,...,x p As the independent variable, the regression coefficients β0, β1, β2, ..., β p , and then replace the outliers with the output of the regression model;

[0074] When duplicate values ​​exist, they are removed through fingerprint recognition technology, including:

[0075]

[0076] Among them, F(x) is the fingerprint of data x, h i (x) is a locality sensitive hash function, a i is the weight;

[0077] By calculating fingerprints and comparing fingerprint similarities, duplicate data can be removed.

[0078] This embodiment provides a data fusion and sharing method for multi-source heterogeneous power big data, which significantly improves the integrity and accuracy of the data through an automatic repair mechanism, ensuring the effectiveness of subsequent data processing; reducing manual intervention, the system can automatically complete most of the data repair work, reducing the frequency and workload of manual intervention.

[0079] As an optional implementation, after the data passes the data quality assessment or repair, it is necessary to fuse the data based on correlation. In this embodiment, a specific implementation of the intelligent data fusion module is given, such as Fig.10 As shown, the intelligent data fusion module includes an intelligent matching module. In the intelligent matching module, the data that has passed the data quality assessment or repair is mainly analyzed to determine the correlation between different data sources, and the data is fused according to the correlation between different data sources. Its implementation method may include: extracting the feature vector of the data that has passed the data quality assessment or repair; determining the matching degree between the data sources of multi-source heterogeneous data according to the feature vector; based on the matching degree, according to the data fusion algorithm, fusing the data from different data sources.

[0080] For example, key features are extracted from different data sources as feature vectors for subsequent similarity analysis. Feature vectors represent:

[0081] v i=[f i1 , f i2 …, f im ]

[0082] Among them, v i is the feature vector of data source i, f im is the mth feature of the ith data source, and m is the number of features.

[0083] As an optional implementation, the matching degree between data sources of multi-source heterogeneous data is determined based on the feature vector, including: calculating the cosine similarity between the data sources represented by the feature vector based on the cosine similarity; determining the data source combination whose cosine similarity is higher than the target threshold based on the size of the cosine similarity; and determining the matching degree of the data source combination based on the Euclidean distance.

[0084] Specifically, first, the cosine similarity between the data sources represented by the feature vectors is calculated according to the cosine similarity as follows:

[0085]

[0086] Among them, sim(v i ,v j ) represents the cosine similarity between data sources i and j, v i and v j are the feature vectors of data sources i and j respectively, ||v i || is the vector v i The modulus of ||v j || is the vector v j Model.

[0087] Since cosine similarity can well measure the similarity of two vectors in direction, after determining the similarity in direction, it is necessary to further measure the difference in magnitude of the data sources. As the most direct distance measurement method, Euclidean distance can reflect the absolute difference of data points in multidimensional space. The combination of the two can provide a more comprehensive data source similarity assessment. Therefore, after calculating the cosine similarity between data sources, the data sources with cosine similarity higher than the target threshold are combined. The target threshold can be set according to actual needs, such as 80%, which is not limited in this embodiment. And the following formula is used to calculate the Euclidean distance between data sources in the data source combination:

[0088]

[0089] Among them, Mi j represents the matching degree of the data source combination, d(v i ,v j ) is the Euclidean distance between data sources. Calculating the matching degree of the data source combination in the above manner is simple and fast, and the amount of calculation is small.

[0090] As another optional implementation, according to the feature vector, determining the matching degree between the data sources of the multi-source heterogeneous data includes: inputting the feature vector into a pre-trained neural network model to determine the matching degree between the data sources of the multi-source heterogeneous data, and the neural network model training process includes: inputting the real matching degree label and the data source into the initial neural network model, using the gradient descent method, iteratively updating the neural network model parameters to minimize the loss function until the preset accuracy requirement is met, and obtaining the trained neural network model; wherein the loss function is defined as:

[0091]

[0092] Among them, L(θ) is the loss function of the model parameter θ, M ij and are the true matching degree and the matching degree predicted by the model, respectively, and N represents the number of samples; the true matching degree can be obtained by manual annotation according to expert suggestions, or by analysis through a similarity analysis method. This embodiment does not limit the method for obtaining the true matching degree, and those skilled in the art can determine it as needed.

[0093] The gradient update formula is:

[0094]

[0095] Among them, θ t is the model parameter of the tth iteration, η is the learning rate, is the gradient of the loss function. The application of the gradient descent method makes the fusion process gradually approach the optimal solution, thereby improving the quality and reliability of data fusion. Therefore, the above neural network model can efficiently analyze the correlation between different data sources. In order to achieve intelligent matching between data sources, machine learning algorithms are used to analyze and match the correlation between different data sources. The use of machine learning algorithms can effectively eliminate the contradictions between data, ensure the consistency of fused data, and have wide adaptability. It can adapt to different types of data sources and complex data structures to meet diverse data fusion needs.

[0096] After determining the matching degree between different data sources by any of the above methods, based on the matching degree and according to the data fusion algorithm, the specific method of fusing the data from different data sources can be: Since the weights of different data sources reflect the contribution of them to the final fusion result, the calculation of the similarity of the data sources can help determine the weights. For example, after normalizing the matching degree, the normalized matching degree is used as the weight to perform data fusion. The specific fusion method is shown in the following formula:

[0097]

[0098] Among them, v fusion is the fused feature vector, v i is the feature vector of the ith data source, w i is the weight of data source i.

[0099] As an optional implementation, in order to ensure the accuracy and consistency of the data fusion results in the intelligent matching module, Fig.10 As shown, it also includes a fusion optimization module, which is designed to continuously adjust the fusion parameters to achieve the goal of improving the accuracy and consistency of the data fusion results. The process of adjusting the fusion parameters by the fusion optimization module includes:

[0100] First, initialize the weights of the fused data sources. In data fusion, the weights of different data sources reflect their contribution to the final fusion result. The weights of the initialized fused data sources meet the following conditions: w i is the weight of the ith data source, and n is the number of data sources for parameter fusion.

[0101] The weights of the data sources and their feature vectors are weighted averaged to obtain the initial fusion result:

[0102]

[0103] Among them, v fusion is the fused feature vector, v i is the feature vector of the i-th data source.

[0104] Optimize the weight of the fused data according to the gradient descent method to reduce the loss function value until it converges to the target threshold or the number of iterations reaches the preset upper limit, then stop the iteration and output the final fusion result and optimized weight;

[0105] Among them, the loss function is:

[0106]

[0107] Among them, L(w) is the loss function, w=[w1,w2,...,w n ] is the weight of all fused data sources, v i is the feature vector of the ith data source, y j is the reference result, v fusion,j is the predicted value after fusion, and m represents the number of samples;

[0108] The method of optimizing the weight of fusion data according to the gradient descent method is as follows:

[0109]

[0110] Where η is the learning rate, is the weight value at the tth iteration, is the weight value at the t+1th iteration, is the gradient of the loss function, which is calculated as follows:

[0111]

[0112] v ij Input the corresponding feature or input variable for the jth sample, indicating the value of the input data on feature i; y j is the true label or value of the jth sample, which is used to compare with the predicted value to calculate the error; v fusion,j is the fused prediction value of the jth sample, representing the current prediction result of the model; m is the number of samples in the dataset, which is used to standardize the gradient so that each update is not too large or too small; w i is the i-th weight, and the gradient calculation is to adjust these weights to reduce the loss function.

[0113] When the loss function value converges to a certain threshold or the number of iterations reaches the preset upper limit, the iteration stops and the final fusion result and optimized weights are output. The convergence condition can be:

[0114] |L(w (t+1) )-L(w (t) )|<∈

[0115] Among them, ∈ is the preset convergence threshold.

[0116] The above method can dynamically adjust the fusion parameters during the data fusion process to ensure the accuracy and consistency of the final fusion result. The application of the gradient descent method makes the fusion process gradually approach the optimal solution, thereby improving the quality and reliability of data fusion. This embodiment provides a data fusion and sharing method for multi-source heterogeneous power big data. Through further optimization and fusion, the fusion weight parameters are continuously adjusted to achieve the optimal fusion result, ensure the accuracy and consistency of the data fusion result, and reduce data redundancy.

[0117] As an optional implementation, the data that has been evaluated or repaired for data quality is analyzed to determine the correlation between different data sources, and before the data is fused according to the correlation between different data sources, that is, in the intelligent data fusion module, such as Fig.10As shown, it also includes a data preprocessing module, which is located before the intelligent matching module and the fusion optimization module. In the data preprocessing module, in order to ensure that data from different data sources can enter the intelligent matching module and the fusion optimization module in a unified format, the data from different data sources are standardized, including cleaning, normalization, and format conversion. The goal of cleaning is to process missing values, outliers, and redundant data in the data set.

[0118] Remove records with missing values:

[0119]

[0120] Among them, D is the original data set, D' is the cleaned data set, and x i is a record in the data set, x ij For record x i The j-th attribute value in .

[0121] Fill missing values ​​with the mean:

[0122]

[0123] Where n is the total number of non-missing values ​​of attribute j.

[0124] Outlier detection based on Z-Score:

[0125]

[0126] Among them, Z ij is the Z-Score value of the jth attribute of the ith record, μ j and σ j are the mean and standard deviation of attribute j respectively. If |Z ij |>|threshold|, then x ij are considered outliers.

[0127] Z-Score normalization converts data of different dimensions to the same dimension range:

[0128]

[0129] in, is the normalized value, μ j and σ j are the mean and standard deviation of attribute j respectively.

[0130] Numerical to categorical conversion, using binning to convert continuous values ​​into discrete categories:

[0131]

[0132] Among them, C ij are the converted category values, [a1,b1), [a2,b2), [a k ,b k ] is the binning interval, c1, c2...c k is the corresponding category label.

[0133] Text to numeric conversion, using One-Hot encoding to convert categorical data into binary vectors:

[0134] v i =[v i1 ,v i2 ,...,v im ],v ij ∈{0,1};

[0135] Among them, v i is the One-Hot vector of categorical data, and m is the number of categories.

[0136] Data cleaning improves data quality by processing missing values, outliers, and deduplication. Normalization unifies the scale of data so that data of different attributes are at the same level, and format conversion ensures that the structure of the data meets the input requirements of the model or algorithm. These preprocessing steps play a vital role in the data processing process to ensure the consistency and effectiveness of the data for subsequent analysis.

[0137] As an optional implementation method, the fused data is distributedly stored for distributed indexing and access to achieve data sharing, including: calculating the hash value of the physical storage node and mapping the physical storage node to the hash ring; dividing the fused data into multiple fragments, calculating the hash value of each fragment data, locating it on the hash ring according to the hash value, and determining its physical storage node; when a retrieval request is received, using access control lists and role authority management technology to verify the retrieval request; when the verification is passed, using a distributed hash table to index the distributed stored data.

[0138] Exemplarily, from a modular perspective, distributed indexing and access realize the function of data sharing, which is composed of a distributed storage module, a data indexing and retrieval module, and an access control module. The distributed storage module stores the merged data fragments on different nodes, and adopts redundant storage and copy strategies to improve the system's fault tolerance and data availability. The data indexing and retrieval module indexes the stored data and supports efficient data retrieval through technologies such as distributed hash tables. The access control module ensures the security of data during the sharing process through access control lists (ACLs) and role-based authority management (RBAC).

[0139] From the perspective of data processing, a hash value is calculated for each physical storage node. This hash value is usually calculated through the node identifier (such as IP address or host name). For example, the node identifier such as node IP address, MAC address or node ID can be hashed through a hash function. The hash function used to calculate the hash value can be SHA-256. The hash value determines the position of the node on the hash ring. At the same time, the data is divided into multiple fragments, and each fragment will calculate a hash value based on its key value. The hash value is used to locate on the hash ring which physical storage node the data fragment should be stored on.

[0140] The above process describes the process of mapping physical storage nodes and data to the hash ring. The overall mapping formula is as follows:

[0141] H(x) = hash(x);

[0142] Among them, x is the key value of the data or the identifier of the node, and H(x) is the position on the hash ring.

[0143] That is, for all nodes, the node identifier N i The formula for hashing and mapping to the hash ring is as follows:

[0144] H(N i )=hash(N i );

[0145] Among them, H(N i ) indicates that the node ID is N i The position of the node in the hash ring.

[0146] Hash all data and map them to the hash ring as follows:

[0147] H(D i )=hash(D i );

[0148] Among them, H(D i ) indicates that the data key value is D i The position of the data in the hash ring.

[0149] Then, the hash ring starts from H(D i ) and go clockwise to find the first i ) of nodes N k As a data storage node.

[0150] N k = min{H(N i )|H(N i )≥H(D j )};

[0151] If H(D j ) is greater than the hash value of all nodes, then data D j Stored on the smallest node on the hash ring.

[0152] When a data retrieval request is received, the system uses access control lists (ACLs) and role-based permission management (RBAC) technologies to verify the legitimacy of the request. The access control list defines which users or roles can access which data, and the role permission management defines different roles and their permissions. The access control list checks whether the retrieval request contains valid user information and permissions, and the role permission management verifies whether the user has the right to access the requested data based on the user role, ensuring the security of the data during the sharing process.

[0153] When verified through access control lists and role-based permission management technology, the hash value of the data fragment is used to query in the distributed hash table, the storage node information of the data fragment is obtained based on the query result, and the data is retrieved from the node.

[0154] This embodiment provides a data fusion and sharing method for multi-source heterogeneous power big data. Distributed storage improves the fault tolerance of the system and ensures that data is still available when any node fails. Through the data indexing mechanism, users can quickly retrieve and access the required data, which improves the efficiency of data sharing. Through strict access control policies, data is protected from unauthorized access and data security is ensured.

[0155] As an optional implementation, the hash value of the storage node is calculated and the storage node is mapped to a hash ring, including: creating multiple virtual nodes on the hash ring for each physical storage node and associating the physical storage node with the multiple virtual nodes; dividing the fused data into multiple fragments, calculating the hash value of each fragment data, locating it on the hash ring according to the hash value, and determining its storage node, including: dividing the fused data into multiple fragments, calculating the hash value of each fragment data, searching for the first virtual node in a clockwise direction, and storing the fragment data on the physical storage node corresponding to the virtual node.

[0156] For example, in order to further balance data distribution, virtual nodes are introduced, each actual node corresponds to multiple virtual nodes, and each virtual node has an independent position on the hash ring. Virtual nodes enable data to be evenly distributed on more nodes, further reducing the possibility of uneven load.

[0157] The virtual node mapping method is:

[0158] H(V ik )=hash(N i +k);

[0159] Among them, V ik Indicates the node ID N i The kth virtual node of the corresponding node, where k is the number of the virtual node.

[0160] Data storage location determination:

[0161] N k = min{H(V ij )|H(V ij )≥H(D j )};

[0162] This data sharding algorithm uses consistent hashing to achieve efficient and balanced sharding and storage of data in a distributed system, avoiding the problem of large-scale data migration caused by node changes, and the introduction of virtual nodes further improves the system's load balancing capabilities.

[0163] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A data fusion and sharing method for multi-source heterogeneous power big data, characterized in that: include: When receiving multi-source heterogeneous power data, dynamic protocol adaptation is performed to convert the multi-source heterogeneous power data formats of different protocols into the current system internal standard format; Perform data quality assessment on the data converted into the current system internal standard format. If the quality assessment result does not meet the quality requirements, perform data repair. Analyze the data that has passed data quality assessment or repair, determine the correlation between different data sources, and fuse the data based on the correlation between different data sources; The fused data is stored in a distributed manner for distributed indexing and access to achieve data sharing.

2. According to claim 1, a data fusion and sharing method for multi-source heterogeneous power big data is characterized in that: When receiving multi-source heterogeneous power data, dynamic protocol adaptation is performed to convert the multi-source heterogeneous power data formats of different protocols into the current system internal standard format, including: A protocol identification algorithm is used to scan multi-source heterogeneous power data sources and determine the protocol type of multi-source heterogeneous power data; Query the protocol adapter of the corresponding protocol type in the protocol management module, and use the dynamic loader to load the protocol adapter; receiving multi-source heterogeneous power data through the protocol adapter and sending the multi-source heterogeneous power data to the protocol converter; The protocol converter converts the multi-source heterogeneous power data formats of different protocols into the current system internal standard format.

3. The data fusion and sharing method of multi-source heterogeneous power big data according to claim 1 is characterized in that: Perform data quality assessment on the data converted to the current system internal standard format. If the quality assessment result does not meet the quality requirements, perform data repair, including: Perform statistical analysis on the data to detect outliers, missing values, and duplicate values; When there are outliers, the regression analysis model is used for prediction, and the predicted values ​​replace the outliers; When there are missing values, linear interpolation is used to fill them; When there are duplicate values, they are removed through fingerprint recognition technology.

4. The data fusion and sharing method of multi-source heterogeneous power big data according to claim 1 is characterized in that: Analyze the data that has passed data quality assessment or repair, determine the correlation between different data sources, and fuse the data based on the correlation of different data sources, including: Extract feature vectors of data that have passed data quality assessment or repair; According to the feature vector, the matching degree between the data sources of multi-source heterogeneous data is determined; Based on the matching degree and data fusion algorithm, data from different data sources are fused.

5. The data fusion and sharing method of multi-source heterogeneous power big data according to claim 4 is characterized in that: According to the feature vector, the matching degree between the data sources of multi-source heterogeneous data is determined, including: Calculate the cosine similarity between the data sources represented by the feature vectors according to the cosine similarity; According to the size of the cosine similarity, determine the data source combination whose cosine similarity is higher than the target threshold; According to the Euclidean distance, determine the matching degree of the data source combination: Among them, M ij represents the matching degree of the data source combination, d(v i ,v j ) is the Euclidean distance between data sources.

6. The data fusion and sharing method of multi-source heterogeneous power big data according to claim 4 is characterized in that: According to the feature vector, the matching degree between the data sources of multi-source heterogeneous data is determined, including: The feature vector is input into the pre-trained neural network model to determine the matching degree between the data sources of multi-source heterogeneous data. The neural network model training process includes: The true matching degree label and the data source are input into the initial neural network model, and the neural network model parameters are iteratively updated using the gradient descent method to minimize the loss function until the preset accuracy requirements are met, thereby obtaining a trained neural network model; Among them, the loss function is defined as: Among them, L(θ) is the loss function of the model parameter θ, M ij and are the true matching degree and the matching degree predicted by the model, respectively, and N represents the number of samples; The gradient update formula is: Among them, θ t is the model parameter of the tth iteration, η is the learning rate, is the gradient of the loss function.

7. The data fusion and sharing method of multi-source heterogeneous power big data according to claim 1 is characterized in that: The fused data is distributedly stored for distributed indexing and access to achieve data sharing, including: Calculate the hash value of the physical storage node and map the physical storage node to the hash ring; Divide the merged data into multiple fragments, calculate the hash value of each fragment, locate it on the hash ring according to the hash value, and determine its physical storage node; When a search request is received, the search request is verified using access control lists and role-based permission management techniques; When verified, a distributed hash table is used to index the distributed stored data.

8. The data fusion and sharing method of multi-source heterogeneous power big data according to claim 7 is characterized in that: Calculate the storage node hash value and map the storage node to the hash ring, including: Each physical storage node creates multiple virtual nodes on the hash ring, and associates the physical storage node with the multiple virtual nodes; Divide the merged data into multiple fragments, calculate the hash value of each fragment, locate it on the hash ring according to the hash value, and determine its storage node, including: The fused data is divided into multiple fragments, the hash value of each fragment data is calculated, the first virtual node is searched in a clockwise direction, and the fragment data is stored in the physical storage node corresponding to the virtual node.

9. A data fusion and sharing method for multi-source heterogeneous power big data according to claim 5 or 6, characterized in that: Also includes: Initialize the weight of the fused data source; Perform weighted averaging on the weights of the data sources and the feature vectors of the data sources to obtain the initial fusion result; Optimize the weight of the fused data according to the gradient descent method to reduce the loss function value until it converges to the target threshold or the number of iterations reaches the preset upper limit, then stop the iteration and output the final fusion result and optimized weight; Among them, the loss function is: Among them, L(w) is the loss function, w=[w1,w2,...,w n ] is the weight of all fused data sources, v i is the feature vector of the ith data source, y j is the reference result, v fusion,j is the result after fusion, m represents the number of samples; The method of optimizing the weight of fusion data according to the gradient descent method is as follows: Where η is the learning rate, is the weight value at the tth iteration, is the weight value at the t+1th iteration, is the gradient of the loss function.

10. The data fusion and sharing method of multi-source heterogeneous power big data according to claim 3 is characterized in that: When there are outliers, the regression analysis model is used for prediction, and the predicted values ​​replace the outliers, including: The following linear regression formula was used to determine the predicted value: in, is the predicted value, β0 is the intercept, β1,β2,...,β p are regression coefficients, x1,x2,...,x p As the independent variable, the regression coefficients β0, β1, β2, ..., β p , and then replace the outliers with the output of the regression model; When there are missing values, linear interpolation is used to fill them, including: Among them, y i is the point to be interpolated, y i-1 and i+1 is a known adjacent data point, x i 、x i-1 、x i+1 is the corresponding independent variable value; When duplicate values ​​exist, they are removed through fingerprint recognition technology, including: Among them, F(x) is the fingerprint of data x, h i (x) is a locality sensitive hash function, a i is the weight; By calculating fingerprints and comparing fingerprint similarities, duplicate data can be removed.

Images