Multi-level heterogeneous data processing method and device based on big data processing technology

Through neural network models, multi-level heterogeneous data are cleaned and fused, and combined with clustering technology, the efficiency and accuracy of processing multi-level heterogeneous data in traditional methods is solved, and the automated data processing and pattern recognition are realized.

CN119089114AActive Publication Date: 2024-12-06SHAANXI YIZHINUO INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411136132.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-06
Estimated Expiration
2044-08-19

AI Technical Summary

Technical Problem

Traditional methods are difficult to effectively process multi-level heterogeneous data, resulting in slow processing speed and error-prone, complex data integration, lost information, and incomplete and accurate decision-making, and inability to capture complex patterns or hidden trends.

Method used

Multi-level heterogeneous data is obtained through multiple sensor devices, data cleaning and fusion are used using neural network models, and combined with data clustering technology to generate a standardized pattern data set.

Benefits of technology

It improves data quality and accuracy, reduces manual intervention, realizes automated processing of data, unifies data format and attributes, improves the comprehensive expression and decision-making accuracy of data, and can discover potential patterns and trends.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089114A_ABST
    Figure CN119089114A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data processing, in particular to a multi-level heterogeneous data processing method and device based on the big data processing technology, and the method comprises the steps: obtaining multi-level heterogeneous data through a plurality of sensor devices, and representing the multi-level heterogeneous data in a data table form, obtaining the attribute of the heterogeneous data based on the type of the sensor equipment; and calculating the similarity among the multi-level heterogeneous data based on the attributes of the heterogeneous data, and performing data cleaning on the multi-level heterogeneous data through a trained neural network-based data cleaning model to obtain standard multi-level heterogeneous data. According to the method, data cleaning is performed on the multi-level heterogeneous data through the trained neural network model, and noise and errors in the data can be automatically processed, so that the quality and accuracy of the data are improved, manual intervention is reduced, and the processing efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-level heterogeneous data processing method and device based on big data processing technology. Background Art

[0002] Multi-level heterogeneous data refers to data from different sources and types, which may differ in structure, format, attributes and hierarchy. It includes structured data, semi-structured data and unstructured data, which usually need to be integrated and utilized through advanced processing and analysis techniques.

[0003] Traditional methods often rely on manual cleaning or rule-based methods, which are slow and error-prone. The formats and attributes of different data sources of traditional methods are inconsistent, making data integration and comparison complicated and cumbersome. Traditional methods may not be able to effectively handle multi-level and multi-type data fusion and are prone to information loss. Traditional methods may not be comprehensive and accurate in making decisions based on a single data source or simple processing. Traditional methods may not be able to capture complex patterns or hidden trends in data clustering and pattern recognition. Summary of the invention

[0004] The purpose of the present invention is to address the problems existing in the background technology and to propose a multi-level heterogeneous data processing method and device based on big data processing technology.

[0005] The technical solution of the present invention is a multi-level heterogeneous data processing method based on big data processing technology, comprising:

[0006] Acquire multi-level heterogeneous data through multiple sensor devices, represent the multi-level heterogeneous data in a data table format, and acquire attributes of the heterogeneous data based on the type of the sensor device;

[0007] Calculating the similarity between the multi-level heterogeneous data based on the attributes of the heterogeneous data, and performing data cleaning on the multi-level heterogeneous data through a trained neural network-based data cleaning model to obtain standard multi-level heterogeneous data;

[0008] Performing data fusion on the standard multi-level heterogeneous data to obtain a first data fusion result, and performing feature fusion on the standard multi-level heterogeneous data to obtain a second data fusion result;

[0009] Performing decision fusion on the standard multi-level heterogeneous data based on the first data fusion result and the second data fusion result to obtain a third data fusion result;

[0010] The third data fusion result is clustered to obtain a plurality of clustering clusters corresponding to the standard multi-level heterogeneous data, and the standard multi-level heterogeneous data is divided based on the plurality of clustering clusters to obtain a plurality of pattern data sets.

[0011] Preferably, the similarity calculation formula between the multi-level heterogeneous data is as follows:

[0012]

[0013] in, represents the similarity between heterogeneous data i and heterogeneous data j, represents the normalized value of the Manhattan distance between heterogeneous data i and heterogeneous data j, and S represents the heterogeneous data dimension, Represents the attributes of heterogeneous data, and They respectively represent the attributes corresponding to the data table corresponding to heterogeneous data i and the attributes corresponding to the data table corresponding to heterogeneous data j.

[0014] Preferably, the neural network-based data cleaning model includes an input layer, multiple hidden layers and an output layer, the number of neurons in the input layer corresponds to the number of attributes of the multi-level heterogeneous data, the node weights between the input layer and the hidden layer correspond to the similarity between the multi-level heterogeneous data, and the output layer is used to output standard multi-level heterogeneous data.

[0015] Preferably, the training method of the neural network-based data cleaning model includes:

[0016] Construct a node weight matrix between the levels of the neural network-based data cleaning model, and the node weight matrix is ​​as follows:

[0017]

[0018] Among them, ξ represents the node weight matrix, ω Sd represents the weight vector between the Sth node of the input layer and the dth hidden layer in the data cleaning model based on the neural network, and d represents the number of hidden layers in the data cleaning model based on the neural network;

[0019] Based on the node weight matrix, a node energy function of the multi-level heterogeneous data is obtained, and the node energy function is as follows:

[0020]

[0021] Among them, α i represents the i-th node data in the input layer, β j represents the jth node data in the input layer, δi represents the bias coefficient of the i-th node data in the input layer, γ i represents the bias coefficient of the jth node data in the input layer, A(α i ,β j ) represents the node energy between the i-th node data and the j-th node data in the input layer;

[0022] The data distribution probability of the nodes in the input layer and the nodes in the hidden layer is obtained based on the node energy function. The data distribution probability calculation formula is as follows:

[0023]

[0024] Among them, ε represents the distribution factor, Q(α i ,β j ) represents the data distribution probability;

[0025] After activating the nodes in the hidden layer of the neural network-based data cleaning model based on the above operation, the node data in the input layer performs back propagation on the neural network-based data cleaning model to obtain a trained neural network-based data cleaning model.

[0026] Preferably, performing data fusion on the standard multi-level heterogeneous data to obtain a first data fusion result includes:

[0027] The total mean square error of the sensor device is calculated, and the total mean square error calculation formula is as follows:

[0028]

[0029] Among them, ΔS 2 represents the total mean square error, X represents the unbiased estimate of multi-level heterogeneous data, ω i Represents x i The corresponding weight, ω j Represents x j The corresponding weight, x i represents the metadata collected by the i-th sensor device, x j Represents the metadata collected by the jth sensor device;

[0030] The optimal weighting factor of the sensor device is calculated based on the total mean square error of the standard multi-level heterogeneous data. The optimal weighting factor calculation formula of the sensor device is as follows:

[0031]

[0032] in, represents the optimal weighting factor of the i-th sensor device;

[0033] The first data fusion result is calculated based on the historical data mean of the sensor device, the optimal weighting factor and the minimum total mean square error. The minimum total mean square error calculation formula is as follows:

[0034]

[0035] in, represents the minimum total mean square error;

[0036] The calculation formula of the first data fusion result is as follows:

[0037]

[0038] in, represents the first data fusion result, Represents the mean value of historical data of the i-th sensor device.

[0039] Preferably, performing feature fusion on the standard multi-level heterogeneous data to obtain a second data fusion result includes:

[0040] Based on the variance maximization criterion, the feature matrix composed of the p-dimensional feature vectors of the standard multi-level heterogeneous data is linearly combined by the principal component analysis method to obtain p principal components of the feature. The p principal components of the feature are expressed as follows:

[0041]

[0042] Among them, f p represents the pth principal component of the feature, y p represents the pth eigenvector of the feature matrix, a kp represents the coefficients of the feature matrix;

[0043] The projections of the p principal components of the feature in the space are calculated. The projection calculation formula in the space is as follows:

[0044]

[0045] in, and denote the projection in space, τ and Respectively represent the vectors corresponding to the projection in space, y k represents the kth eigenvector, y l represents the lth eigenvector;

[0046] A feature component is obtained based on the projection of the p principal components of the feature in space, and the feature vector and the feature component are fused to obtain a second data fusion result. The calculation formula of the second data fusion result is as follows:

[0047]

[0048] in, represents the second data fusion result, and They represent the characteristic components of the kth eigenvector and the lth eigenvector respectively.

[0049] Preferably, the third data fusion result calculation formula is as follows:

[0050]

[0051] Wherein, U represents the third data fusion result.

[0052] Preferably, clustering the third data fusion result to obtain a plurality of clusters corresponding to the standard multi-level heterogeneous data includes:

[0053] The separation degree of the third data fusion result under the q-dimensional feature is analyzed by the data clustering silhouette coefficient, and the data clustering silhouette coefficient is as follows:

[0054]

[0055] Among them, a q h represents the interval of clustering stability in q-dimensional features, b q h represents the interval of cluster transition state in q-dimensional features, h represents the number of cluster sequences in stable state, represents the data clustering silhouette coefficient of the heterogeneous data points in the third data fusion result in the q-dimensional feature, r e represents the total number of clustering sequences in the stable state, and D represents the total dimension of the features of the third data fusion result;

[0056] A sliding window is set and the average value of the sliding window feature quantity is calculated. The sliding window setting formula is as follows:

[0057]

[0058] in, Represents the internal difference of the p-dimensional feature, Y i represents the sliding window, and N represents the width of the sliding window;

[0059] The average value calculation formula of the sliding window feature quantity is as follows:

[0060]

[0061] in, Represents the average value of the sliding window feature;

[0062] The data range in the sliding window is limited based on the data clustering silhouette coefficient. The data range expression is as follows:

[0063]

[0064] in, Represents the extreme value of the p-dimensional feature within the sliding window.

[0065] Preferably, clustering the third data fusion result to obtain a plurality of clusters corresponding to the standard multi-level heterogeneous data further includes:

[0066] The feature difference is calculated based on the average value of the sliding window feature value, and the cluster center is set based on the feature difference. The feature difference calculation formula is as follows:

[0067]

[0068] in, represents the eigenvalue of the initial cluster p-point sliding window, Represents the feature difference between sliding windows;

[0069] The clustering condition that the cluster center needs to satisfy is set based on the feature difference and the data range in the sliding window, and the clustering condition is as follows:

[0070]

[0071] Among them, θ represents the data clustering standard threshold;

[0072] The clustering result is calculated based on the clustering condition, and the clustering result calculation formula is as follows:

[0073]

[0074] Among them, ||X i -X n || represents the Euclidean distance from data space category i to the nth data sample point, n represents the data sample point, F(n) represents the sample point density function, τ(n) represents the clustering result, and r represents the effective radius of the cluster center.

[0075] The technical solution of the present invention is a multi-stage heterogeneous data processing device based on big data processing technology, which is applicable to the multi-stage heterogeneous data processing method based on big data processing technology, and is characterized by comprising:

[0076] A data acquisition module, the data acquisition module is used to acquire multi-level heterogeneous data through multiple sensor devices, and represent the data in a data table form, and acquire attributes of the heterogeneous data based on the type of sensor device;

[0077] A data cleaning module, wherein the data cleaning module is used to calculate the similarity between the multi-level heterogeneous data based on the attributes of the heterogeneous data, and to perform data cleaning on the multi-level heterogeneous data through a trained neural network-based data cleaning model to obtain standard multi-level heterogeneous data;

[0078] A data fusion module, the data fusion module is used to perform data fusion on the standard multi-level heterogeneous data to obtain a first data fusion result, and perform feature fusion on the standard multi-level heterogeneous data to obtain a second data fusion result;

[0079] A decision fusion module, the decision fusion module is used to perform decision fusion on the standard multi-level heterogeneous data based on the first data fusion result and the second data fusion result to obtain a third data fusion result;

[0080] A data clustering module is used to cluster the third data fusion result to obtain a plurality of clustering clusters corresponding to the standard multi-level heterogeneous data, and divide the standard multi-level heterogeneous data based on the plurality of clustering clusters to obtain a plurality of pattern data sets.

[0081] Compared with the prior art, the above technical solution of the present invention has the following beneficial technical effects:

[0082] 1. The present invention performs data cleaning on multi-level heterogeneous data through a trained neural network model, and can automatically process noise and errors in the data, thereby improving the quality and accuracy of the data, thereby reducing manual intervention and improving processing efficiency. The multi-level heterogeneous data is standardized into standard data, making subsequent analysis and processing more consistent and comparable, thereby helping to unify data formats and attributes and reducing problems caused by data inconsistency. In addition, through data fusion and feature fusion methods, data from different sources and different types are comprehensively processed. This multi-level fusion method can extract key information from the data from different angles and improve the comprehensive expressiveness of the data.

[0083] 2. The present invention combines data fusion results and feature fusion results to perform decision fusion, so as to obtain more comprehensive analysis results, thereby improving the accuracy of decision-making and reducing the deviation that may be caused by a single data source. The fused data can be clustered to divide the data into different clusters. Through this method, potential patterns and trends in the data can be discovered, so as to perform more targeted analysis and application. Moreover, through the analysis of the clustering results, multiple pattern data sets can be identified. These patterns can be used in various application scenarios, such as anomaly detection, trend prediction, personalized recommendation, etc., providing data support for business decisions and improving data-driven decision-making capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 A schematic flow chart of an overall method in an embodiment of the present invention;

[0085] Figure 2 A schematic diagram of the overall process of an embodiment of the present invention.

[0086] Figure numerals: 1. Data acquisition module; 2. Data cleaning module; 3. Data fusion module; 4. Decision fusion module; 5. Data clustering module. DETAILED DESCRIPTION

[0087] Embodiment 1, as Figure 1 As shown, the present invention proposes a multi-level heterogeneous data processing method based on big data processing technology, comprising:

[0088] S1. Acquire multi-level heterogeneous data through multiple sensor devices, represent the multi-level heterogeneous data in a data table form, and acquire attributes of the heterogeneous data based on the type of sensor device;

[0089] S2. Calculate the similarity between multi-level heterogeneous data based on the attributes of the heterogeneous data, and clean the multi-level heterogeneous data using a trained neural network-based data cleaning model to obtain standard multi-level heterogeneous data;

[0090] S3, performing data fusion on the standard multi-level heterogeneous data to obtain a first data fusion result, and performing feature fusion on the standard multi-level heterogeneous data to obtain a second data fusion result;

[0091] S4, performing decision fusion on the standard multi-level heterogeneous data based on the first data fusion result and the second data fusion result to obtain a third data fusion result;

[0092] S5. Clustering the third data fusion result to obtain a plurality of clustering clusters corresponding to the standard multi-level heterogeneous data, and dividing the standard multi-level heterogeneous data based on the plurality of clustering clusters to obtain a plurality of pattern data sets.

[0093] In the present invention, multi-level heterogeneous data refers to a data set from different sources, formats, and structures; data table format refers to structuring data into a table format for easy processing and analysis; attributes refer to the characteristics or descriptions of data, such as temperature sensor readings, timestamps, etc.; similarity calculation refers to measuring the degree of similarity between data to determine the relevance or consistency of data; pattern data set refers to a grouped data set generated based on clustering results, each set represents a pattern or category; clustering refers to grouping data according to similarity to identify natural classifications or patterns in the data; data fusion refers to integrating data from different sources into a unified data set to enhance the overall value of the data.

[0094] Embodiment 2, a multi-level heterogeneous data processing method based on big data processing technology proposed by the present invention, compared with embodiment 1, this embodiment also includes: the similarity calculation formula between multi-level heterogeneous data is as follows:

[0095]

[0096] in, represents the similarity between heterogeneous data i and heterogeneous data j, represents the normalized value of the Manhattan distance between heterogeneous data i and heterogeneous data j, and S represents the heterogeneous data dimension, Represents the attributes of heterogeneous data, and They respectively represent the attributes corresponding to the data table corresponding to heterogeneous data i and the attributes corresponding to the data table corresponding to heterogeneous data j.

[0097] In this embodiment, Manhattan distance refers to the distance calculation method between two points in multidimensional space, which is equal to the sum of the absolute values ​​of their differences in each dimension; normalization refers to adjusting the data to a standard range to eliminate the impact of different data scales for easy comparison; dimension refers to the number of features of the data. For example, the dimensions of a data point may include different measurements such as temperature and humidity; attributes refer to the columns in the data table, representing different features of the data, such as the type of sensor reading, timestamp, etc.

[0098] In an optional embodiment, a neural network-based data cleaning model includes an input layer, multiple hidden layers and an output layer, the number of neurons in the input layer corresponds to the number of attributes of the multi-level heterogeneous data, the node weights between the input layer and the hidden layer correspond to the similarity between the multi-level heterogeneous data, and the output layer is used to output standard multi-level heterogeneous data.

[0099] It should be noted that a neural network refers to a computational model that mimics the neuron structure of the human brain and is used to process and analyze complex data.

[0100] In an optional embodiment, a training method for a data cleaning model based on a neural network includes:

[0101] A1. Construct a node weight matrix between the levels of the neural network-based data cleaning model. The node weight matrix is ​​as follows:

[0102]

[0103] Among them, ξ represents the node weight matrix, ω Sdrepresents the weight vector between the Sth node of the input layer and the dth hidden layer in the data cleaning model based on the neural network, and d represents the number of hidden layers in the data cleaning model based on the neural network;

[0104] A2. Based on the node weight matrix, the node energy function of multi-level heterogeneous data is obtained. The node energy function is as follows:

[0105]

[0106] Among them, α i represents the i-th node data in the input layer, β j represents the jth node data in the input layer, δ i represents the bias coefficient of the i-th node data in the input layer, γ i represents the bias coefficient of the jth node data in the input layer, A(α i ,β j ) represents the node energy between the i-th node data and the j-th node data in the input layer;

[0107] A3. Based on the node energy function, the data distribution probability of the nodes in the input layer and the nodes in the hidden layer is obtained. The data distribution probability calculation formula is as follows:

[0108]

[0109] Among them, ε represents the distribution factor, Q(α i ,β j ) represents the data distribution probability;

[0110] A4. After activating the nodes in the hidden layer of the neural network-based data cleaning model based on the above operations, the node data in the input layer performs back propagation on the neural network-based data cleaning model to obtain a trained neural network-based data cleaning model.

[0111] In an optional embodiment, data fusion is performed on standard multi-level heterogeneous data to obtain a first data fusion result, including:

[0112] B1. Calculate the total mean square error of the sensor equipment. The total mean square error calculation formula is as follows:

[0113]

[0114] Among them, ΔS 2 represents the total mean square error, X represents the unbiased estimate of multi-level heterogeneous data, ω i Represents x i The corresponding weight, ω j Represents x j The corresponding weight, x irepresents the metadata collected by the i-th sensor device, x j Represents the metadata collected by the jth sensor device;

[0115] B2. Calculate the optimal weighting factor of the sensor device based on the total mean square error of the standard multi-level heterogeneous data. The optimal weighting factor calculation formula of the sensor device is as follows:

[0116]

[0117] in, represents the optimal weighting factor of the i-th sensor device;

[0118] B3. Calculate the first data fusion result based on the historical data mean of the sensor device, the optimal weighting factor and the minimum total mean square error. The minimum total mean square error calculation formula is as follows:

[0119]

[0120] in, represents the minimum total mean square error;

[0121] The calculation formula of the first data fusion result is as follows:

[0122]

[0123] in, represents the first data fusion result, Represents the mean value of historical data of the i-th sensor device.

[0124] In an optional embodiment, feature fusion is performed on standard multi-level heterogeneous data to obtain a second data fusion result, including:

[0125] C1. Based on the variance maximization criterion, the feature matrix composed of the p-dimensional feature vectors of standard multi-level heterogeneous data is linearly combined by principal component analysis to obtain the p principal components of the feature. The p principal components of the feature are expressed as follows:

[0126]

[0127] Among them, f p represents the pth principal component of the feature, y p represents the pth eigenvector of the feature matrix, a kp represents the coefficients of the feature matrix;

[0128] C2. Calculate the projection of the p principal components of the feature in space. The projection calculation formula in space is as follows:

[0129]

[0130] in, and denote the projection in space, τ and Respectively represent the vectors corresponding to the projection in space, y k represents the kth eigenvector, y l represents the lth eigenvector;

[0131] C3. Obtain feature components based on the projection of the p principal components of the features in space, fuse the feature vectors and feature components to obtain a second data fusion result. The calculation formula for the second data fusion result is as follows:

[0132]

[0133] in, represents the second data fusion result, and They represent the characteristic components of the kth eigenvector and the lth eigenvector respectively.

[0134] It should be noted that the variance maximization criterion is used as the standard for selecting principal components. The purpose is to select the direction that can maximize the data variance, so that as much data variation information as possible can be retained. In principal component analysis (PCA), the selection of principal components is based on this criterion; principal component analysis (PCA) refers to a dimensionality reduction technique that transforms data from the original feature space to a new feature space through linear transformation, where the new features (principal components) are linear combinations of the original features, and these principal components are arranged according to the size of the variance. The goal of PCA is to reduce the dimension of the data while retaining the original information of the data as much as possible; principal components refer to the new features obtained through PCA transformation, which represent the information with the largest variance in the data, and each principal component is a linear combination of some original features in the feature matrix; projection in space refers to the projection of data points in the direction of the principal components, that is, mapping data from the original space to the principal component space, and projection is the performance of data in the direction of the principal components.

[0135] In an optional embodiment, the third data fusion result calculation formula is as follows:

[0136]

[0137] Wherein, U represents the third data fusion result.

[0138] In an optional embodiment, clustering is performed on the third data fusion result to obtain a plurality of clusters corresponding to the standard multi-level heterogeneous data, including:

[0139] D1. Analyze the separation degree of the third data fusion result under the q-dimensional feature through the data clustering silhouette coefficient. The data clustering silhouette coefficient is as follows:

[0140]

[0141] Among them, a q h represents the interval of clustering stability in q-dimensional features, b q h represents the interval of cluster transition state in q-dimensional features, h represents the number of cluster sequences in stable state, represents the data clustering silhouette coefficient of the heterogeneous data points in the third data fusion result in the q-dimensional feature, r e represents the total number of clustering sequences in the stable state, and D represents the total dimension of the features of the third data fusion result;

[0142] D2. Set the sliding window and calculate the average value of the sliding window feature quantity. The sliding window setting formula is as follows:

[0143]

[0144] in, Represents the internal difference of the p-dimensional feature, Y i represents the sliding window, and N represents the width of the sliding window;

[0145] The calculation formula for the average value of the sliding window feature quantity is as follows:

[0146]

[0147] in, Represents the average value of the sliding window feature;

[0148] D3. Limit the data range within the sliding window based on the data clustering silhouette coefficient. The data range expression is as follows:

[0149]

[0150] in, Represents the extreme value of the p-dimensional feature within the sliding window.

[0151] In an optional embodiment, clustering the third data fusion result to obtain a plurality of clusters corresponding to the standard multi-level heterogeneous data further includes:

[0152] D4. Calculate the feature difference based on the average value of the sliding window feature quantity, and set the cluster center based on the feature difference. The feature difference calculation formula is as follows:

[0153]

[0154] in, represents the eigenvalue of the initial cluster p-point sliding window, Represents the feature difference between sliding windows;

[0155] D5. Set the clustering conditions that the cluster center must meet based on the feature difference and the data range in the sliding window. The clustering conditions are as follows:

[0156]

[0157] Among them, θ represents the data clustering standard threshold;

[0158] D6. Calculate the clustering results based on the clustering conditions. The clustering result calculation formula is as follows:

[0159]

[0160] Among them, ||X i -X n || represents the Euclidean distance from data space category i to the nth data sample point, n represents the data sample point, F(n) represents the sample point density function, τ(n) represents the clustering result, and r represents the effective radius of the cluster center.

[0161] Embodiment three, as Figure 2 As shown, a multi-level heterogeneous data processing device based on big data processing technology is applicable to the above-mentioned multi-level heterogeneous data processing method based on big data processing technology, including:

[0162] The data acquisition module 1 is used to acquire multi-level heterogeneous data through multiple sensor devices, represent the data in a data table format, and acquire attributes of the heterogeneous data based on the type of sensor device;

[0163] Data cleaning module 2, which is used to calculate the similarity between multi-level heterogeneous data based on the attributes of the heterogeneous data, and to clean the multi-level heterogeneous data through a trained neural network-based data cleaning model to obtain standard multi-level heterogeneous data;

[0164] The data fusion module 3 is used to perform data fusion on the standard multi-level heterogeneous data to obtain a first data fusion result, and perform feature fusion on the standard multi-level heterogeneous data to obtain a second data fusion result;

[0165] A decision fusion module 4, which is used to perform decision fusion on the standard multi-level heterogeneous data based on the first data fusion result and the second data fusion result to obtain a third data fusion result;

[0166] The data clustering module 5 is used to cluster the third data fusion result to obtain a plurality of clustering clusters corresponding to the standard multi-level heterogeneous data, and divide the standard multi-level heterogeneous data based on the plurality of clustering clusters to obtain a plurality of pattern data sets.

[0167] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A multi-level heterogeneous data processing method based on big data processing technology, characterized in that: include: Acquire multi-level heterogeneous data through multiple sensor devices, represent the multi-level heterogeneous data in a data table format, and acquire attributes of the heterogeneous data based on the type of the sensor device; Calculating the similarity between the multi-level heterogeneous data based on the attributes of the heterogeneous data, and performing data cleaning on the multi-level heterogeneous data through a trained neural network-based data cleaning model to obtain standard multi-level heterogeneous data; Performing data fusion on the standard multi-level heterogeneous data to obtain a first data fusion result, and performing feature fusion on the standard multi-level heterogeneous data to obtain a second data fusion result; Performing decision fusion on the standard multi-level heterogeneous data based on the first data fusion result and the second data fusion result to obtain a third data fusion result; The third data fusion result is clustered to obtain a plurality of clustering clusters corresponding to the standard multi-level heterogeneous data, and the standard multi-level heterogeneous data is divided based on the plurality of clustering clusters to obtain a plurality of pattern data sets.

2. The multi-level heterogeneous data processing method based on big data processing technology according to claim 1 is characterized by: The similarity calculation formula between the multi-level heterogeneous data is as follows: in, represents the similarity between heterogeneous data i and heterogeneous data j, represents the normalized value of the Manhattan distance between heterogeneous data i and heterogeneous data j, and S represents the heterogeneous data dimension, Represents the attributes of heterogeneous data, and They respectively represent the attributes corresponding to the data table corresponding to heterogeneous data i and the attributes corresponding to the data table corresponding to heterogeneous data j.

3. The multi-level heterogeneous data processing method based on big data processing technology according to claim 2 is characterized by: The neural network-based data cleaning model includes an input layer, multiple hidden layers and an output layer. The number of neurons in the input layer corresponds to the number of attributes of the multi-level heterogeneous data, the node weights between the input layer and the hidden layer correspond to the similarity between the multi-level heterogeneous data, and the output layer is used to output standard multi-level heterogeneous data.

4. The multi-level heterogeneous data processing method based on big data processing technology according to claim 3 is characterized by: The training method of the neural network-based data cleaning model includes: Construct a node weight matrix between the levels of the neural network-based data cleaning model, and the node weight matrix is ​​as follows: Among them, ξ represents the node weight matrix, ω Sd represents the weight vector between the Sth node of the input layer and the dth hidden layer in the data cleaning model based on the neural network, and d represents the number of hidden layers in the data cleaning model based on the neural network; Based on the node weight matrix, a node energy function of the multi-level heterogeneous data is obtained, and the node energy function is as follows: Among them, α i represents the i-th node data in the input layer, β j represents the jth node data in the input layer, δ i represents the bias coefficient of the i-th node data in the input layer, γ i represents the bias coefficient of the jth node data in the input layer, A(α i ,β j ) represents the node energy between the i-th node data and the j-th node data in the input layer; The data distribution probability of the nodes in the input layer and the nodes in the hidden layer is obtained based on the node energy function. The data distribution probability calculation formula is as follows: Among them, ε represents the distribution factor, Q(α i ,β j ) represents the data distribution probability; After activating the nodes in the hidden layer of the neural network-based data cleaning model based on the above operations, the node data in the input layer performs back propagation on the neural network-based data cleaning model to obtain a trained neural network-based data cleaning model.

5. The multi-level heterogeneous data processing method based on big data processing technology according to claim 1 is characterized by: Performing data fusion on the standard multi-level heterogeneous data to obtain a first data fusion result includes: The total mean square error of the sensor device is calculated, and the total mean square error calculation formula is as follows: Among them, ΔS 2 represents the total mean square error, X represents the unbiased estimate of multi-level heterogeneous data, ω i Represents x i The corresponding weight, ω j Represents x j The corresponding weight, x i represents the metadata collected by the i-th sensor device, x j Represents the metadata collected by the jth sensor device; The optimal weighting factor of the sensor device is calculated based on the total mean square error of the standard multi-level heterogeneous data. The optimal weighting factor calculation formula of the sensor device is as follows: in, represents the optimal weighting factor of the i-th sensor device; The first data fusion result is calculated based on the historical data mean of the sensor device, the optimal weighting factor and the minimum total mean square error. The minimum total mean square error calculation formula is as follows: in, represents the minimum total mean square error; The calculation formula of the first data fusion result is as follows: in, represents the first data fusion result, Represents the mean value of historical data of the i-th sensor device.

6. The multi-level heterogeneous data processing method based on big data processing technology according to claim 5 is characterized by: Performing feature fusion on the standard multi-level heterogeneous data to obtain a second data fusion result includes: Based on the variance maximization criterion, the feature matrix composed of the p-dimensional feature vectors of the standard multi-level heterogeneous data is linearly combined by the principal component analysis method to obtain p principal components of the feature. The p principal components of the feature are expressed as follows: Among them, f p represents the pth principal component of the feature, y p represents the pth eigenvector of the feature matrix, a kp represents the coefficients of the feature matrix; The projections of the p principal components of the feature in the space are calculated. The projection calculation formula in the space is as follows: in, and denote the projection in space, τ and Respectively represent the vectors corresponding to the projection in space, y k represents the kth eigenvector, y l represents the lth eigenvector; A feature component is obtained based on the projection of the p principal components of the feature in space, and the feature vector and the feature component are fused to obtain a second data fusion result. The calculation formula of the second data fusion result is as follows: in, represents the second data fusion result, and They represent the characteristic components of the kth eigenvector and the lth eigenvector respectively.

7. The multi-level heterogeneous data processing method based on big data processing technology according to claim 6 is characterized by: The third data fusion result calculation formula is as follows: Wherein, U represents the third data fusion result.

8. The multi-level heterogeneous data processing method based on big data processing technology according to claim 1 is characterized by: Clustering the third data fusion result to obtain a plurality of clusters corresponding to the standard multi-level heterogeneous data includes: The separation degree of the third data fusion result under the q-dimensional feature is analyzed by the data clustering silhouette coefficient, and the data clustering silhouette coefficient is as follows: Among them, a q h represents the interval of clustering stability in q-dimensional features, b q h represents the interval of cluster transition state in q-dimensional features, h represents the number of cluster sequences in stable state, represents the data clustering silhouette coefficient of the heterogeneous data points in the third data fusion result in the q-dimensional feature, r e represents the total number of clustering sequences in the stable state, and D represents the total dimension of the features of the third data fusion result; A sliding window is set and the average value of the sliding window feature quantity is calculated. The sliding window setting formula is as follows: in, Represents the internal difference of the p-dimensional feature, Y i represents the sliding window, and N represents the width of the sliding window; The average value calculation formula of the sliding window feature quantity is as follows: in, Represents the average value of the sliding window feature; The data range in the sliding window is limited based on the data clustering silhouette coefficient. The data range expression is as follows: in, Represents the extreme value of the p-dimensional feature within the sliding window.

9. The multi-level heterogeneous data processing method based on big data processing technology according to claim 8, characterized in that: Clustering the third data fusion result to obtain a plurality of clusters corresponding to the standard multi-level heterogeneous data, further comprising: The feature difference is calculated based on the average value of the sliding window feature value, and the cluster center is set based on the feature difference. The feature difference calculation formula is as follows: in, represents the eigenvalue of the initial cluster p-point sliding window, Represents the feature difference between sliding windows; The clustering condition that the cluster center needs to satisfy is set based on the feature difference and the data range in the sliding window, and the clustering condition is as follows: Among them, θ represents the data clustering standard threshold; The clustering result is calculated based on the clustering condition, and the clustering result calculation formula is as follows: Among them, ||X i -X n || represents the Euclidean distance from data space category i to the nth data sample point, n represents the data sample point, F(n) represents the sample point density function, τ(n) represents the clustering result, and r represents the effective radius of the cluster center.

10. A multi-stage heterogeneous data processing device based on big data processing technology, which is applicable to a multi-stage heterogeneous data processing method based on big data processing technology as claimed in any one of claims 1 to 9, characterized in that: include: A data acquisition module (1), the data acquisition module (1) is used to acquire multi-level heterogeneous data through multiple sensor devices, represent the data in a data table format, and acquire attributes of the heterogeneous data based on the type of sensor device; A data cleaning module (2), the data cleaning module (2) being used to calculate the similarity between the multi-level heterogeneous data based on the attributes of the heterogeneous data, and to perform data cleaning on the multi-level heterogeneous data using a trained neural network-based data cleaning model to obtain standard multi-level heterogeneous data; A data fusion module (3), the data fusion module (3) is used to perform data fusion on the standard multi-level heterogeneous data to obtain a first data fusion result, and perform feature fusion on the standard multi-level heterogeneous data to obtain a second data fusion result; A decision fusion module (4), the decision fusion module (4) is used to perform decision fusion on the standard multi-level heterogeneous data based on the first data fusion result and the second data fusion result to obtain a third data fusion result; A data clustering module (5), the data clustering module (5) is used to cluster the third data fusion result to obtain a plurality of clustering clusters corresponding to the standard multi-level heterogeneous data, and divide the standard multi-level heterogeneous data based on the plurality of clustering clusters to obtain a plurality of pattern data sets.

Citation Information

Patent Citations

  • Multi-source heterogeneous big data fusion method and system in aerospace manufacturing process

    CN114841252A

  • Fault diagnosis method and device based on transformer oil chromatographic analysis

    CN115563563A

  • Multi-source heterogeneous data cleaning method based on generative adversarial network

    CN116166650A

  • Data real-time efficient fusion processing method based on FCM clustering algorithm model

    CN117112871A

  • Data decomposition / reduction method for visualizing data clusters / sub-clusters

    WO2000016250A1

Cited By

  • Data cleaning method, device and system for edge end and storage medium

    CN120216884A