A fusion intelligent lake warehouse system and method based on trusted data

Abstract

The application discloses a kind of fusion intelligent lake warehouse systems and methods based on trusted data, it is related to data fusion technical field, the present application is trained to obtain classifier using three types of trusted data in history, the mixed trusted data of real-time acquisition in data is classified by format;Respectively extract the main information of each trusted data, according to the main information of each trusted data, the content classification of all real-time trusted data collected;Trusted data after content classification is stored using blockchain technology;Build data fusion model, and the comprehensive feature vector of the feature of all different formats trusted data is fused to generate;Respectively extract the feature of three types of trusted data and input into data fusion model, and calculate abnormal threshold value;Extract the trusted data of each content classification stored, after obtaining comprehensive feature vector by data fusion model, real-time comprehensive feature vector is judged using abnormal threshold value, when judging that trusted data exists abnormal risk, send early warning.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of data fusion, and particularly relates to a fusion intelligent lake warehouse system and method based on trusted data. BACKGROUND

[0002] Traditional lake warehouses are computation-centered, and have limited support for AI-driven real-time analysis, incremental computation and the like, and cannot meet the data processing requirements in the era of large models. With the deepening of digital transformation, the trend of data security and privacy protection is becoming more and more strict, and enterprises need to be compatible with historical systems (such as relational databases) and emerging architectures (such as lake warehouse integration) to achieve seamless management in a hybrid environment through logical unification and virtualization technology. Cloud computing provides on-demand allocation of computing and storage resources, reducing the hardware construction cost of enterprises; distributed file systems and distributed database technologies are also improving, and can support distributed storage and parallel processing of massive data, laying a solid foundation for building a large-scale trusted data lake warehouse system; the federal architecture (such as the digital object identification resolution protocol) realizes data sharing and trusted management among multiple subjects, breaking down organizational boundaries. However, the existing technology cannot effectively balance the strictness of data governance, the flexibility of architecture and the real-time performance of intelligent analysis, resulting in challenges for enterprises in dealing with massive heterogeneous data, AI fusion and cross-domain collaboration. SUMMARY

[0003] The application aims to provide a fusion intelligent lake warehouse system and method based on trusted data to solve the problems in the prior art.

[0004] To achieve the above-mentioned purpose, the application provides the following technical scheme.

[0005] A fusion intelligent lake warehouse method based on trusted data, the method comprising the following steps:

[0006] S100, collect all trusted data collected by sensors, judge the format of the collected trusted data, and divide the trusted data into numerical trusted data, textual trusted data and video trusted data; use the three types of trusted data in the history to train a classifier, and classify the mixed trusted data collected in real time according to the format;

[0007] Further, the specific steps of classifying the mixed trusted data collected in real time according to the format are as follows:

[0008] S101, collect trusted data collected by sensors, judge the format of the collected data, and divide the data into numerical trusted data, textual data and video data; extract the format state of the three types of data as the format features of the three types of data, and use the format features of the three types of data as a training set, and calculate the information entropy of each type of data, the formula being:

[0009] ;

[0010] In the formula, H(X) represents the information entropy of each type of data calculated, p(x i ) represents the probability of the x i th type of data, and n represents the number of data in the training set of each type of data;

[0011] S102, after calculating the information entropy of each type of data, the information gain of the three data format features is calculated, and the formula is:

[0012] ;

[0013] In the formula, Gain(X, S) represents the information gain of each format feature in the three data format features calculated, S represents the format feature, and the information gain of the three data format features is calculated; and the format feature with the maximum information gain is selected as the classification standard;

[0014] S103, the selected classification standard is used as a classification node, and the classification node is input into the decision tree to construct a classifier; after real-time collection of all data in the trusted space to obtain mixed data, the mixed data is input into the classifier to automatically classify the collected mixed data.

[0015] In the face of massive and complex trusted data, different format data storage, access requirements are different, and after the mixed data is classified by the classifier according to the format, the product authorization key table and the interface calling key table of the encryption type can be quickly stored in the relational database, and relying on the mature index system, the target data can be accurately positioned within milliseconds; the product use log of the text type is put into the document database, and when a specific product use is only a certain period of log, it can be quickly screened based on the document identifier; the video has a dedicated storage area, which is convenient for management. In the classification process, it is easier to find data with format disorder and not conforming to the standard, and to clean up or correct in time to maintain data quality.

[0016] S200, after classifying the real-time collected mixed trusted data, the three types of trusted data after classification are analyzed, the main information of each trusted data is extracted, and the content of all real-time trusted data collected is classified according to the main information of each trusted data;

[0017] Further, the specific steps of classifying the content of all real-time trusted data collected according to the main information of each data are as follows:

[0018] S201, after classifying the real-time collected mixed data, all kinds of information values in the numerical type data are extracted, the average value of all kinds of information values is calculated as Zp, and then the importance of each information value is calculated, and the formula is:

[0019] ;

[0020] In the formula, Zd represents the importance of the calculated information value, Z represents each information value, and Zp represents the average of all calculated information values. After calculating the importance of each information value, the information value with the largest importance is selected as the main information corresponding to the numerical data;

[0021] S202, in the text type data, all kinds of words contained in the text type data are extracted, and the importance of each kind of word is calculated, and the formula is:

[0022] ;

[0023] In the formula, Wd represents the importance of each word calculated, m j represents the number of the jth word in the document, M represents the total number of words in the document; N represents the total number of documents, and L j represents the number of documents containing the jth word in the document; the importance of all kinds of words in the text type data is calculated repeatedly, and the word with the largest importance is selected as the main information of the text type data;

[0024] S203, in the video type data, the video type data is processed by frame, and all key frames in the video type data are extracted by using the CNN model. The objects in all key frame images are extracted, the total number of extracted object types is p, the importance of each object in the video type data is calculated, and the formula is:

[0025] ;

[0026] In the formula, Td represents the importance of each object in the video type data calculated, k represents the number of occurrences of each object in all key frames, and kz represents the total number of occurrences of all objects in all key frames; r min and r max represent the minimum and maximum gray values of each object in all key frames, R min and R max represent the minimum and maximum gray values of each key frame in all key frames; after calculating the importance of each object, the object with the largest importance value is selected as the main information of the video type data;

[0027] S204, after calculating the main information of the numerical data, the text type data and the video type data respectively, the information content corresponding to the main information is mapped to obtain the information content of each data main information. In the three types of data, the information content of the main information is used for division, and the data with the same information content of the main information is divided into the same content data.

[0028] After the content is subdivided, the manager can quickly lock the key business board, accurately grasp the operation details of the specific business process, understand the data transaction process in the trusted space; through content classification, hidden connections between different businesses can be found; data content reflects the difference in sensitivity, and content classification facilitates the implementation of more detailed security policies.

[0029] S300, after the content classification of the collected real-time trusted data, the content classified trusted data is stored by using the blockchain technology;

[0030] Further, the specific steps of storing the content classified data by using the blockchain technology are:

[0031] S301, after the content classification of the data in the trusted space, the data with the same content is stored in the same block by using the blockchain technology, and the hash value of each data is calculated by using the hash algorithm when storing the data, the formula is: HA=SHA256(D), in the formula, HA represents the hash value of each data, and D represents the original data of each data. The hash value of each data is stored correspondingly when storing.

[0032] S400, after storing the content classified trusted data, the features of all different format trusted data in the same content are extracted, the extracted features are mapped in the same space, the features in the same space are reduced in dimension, a data fusion model is constructed, and the features of all different format trusted data are fused to generate a comprehensive feature vector;

[0033] Further, the specific steps of fusing the features of all different format data to generate a comprehensive feature vector are:

[0034] S401, after storing the classified data, the stored data is analyzed by using the deep learning algorithm, the features of all data in the same content category are extracted, the number of features extracted in each data is g, each extracted feature is standardized to form a feature matrix, the covariance matrix of each feature matrix after standardization is calculated, and the formula is:

[0035] ;

[0036] In the formula, F represents the covariance matrix of each feature matrix, e represents the number of each feature, V represents the feature matrix, and V T represents the transpose matrix of the feature matrix; the characteristic equation is constructed, the eigenvalue and eigenvector of the characteristic equation are calculated according to the covariance matrix, and the eigenvalue with the maximum value is selected as the main feature of the corresponding data;

[0037] S402, the main features of the three types of data in the same content category are T1, T2 and T3 extracted through S401, the extracted features are mapped in the same space, the features in the same space are reduced in dimension, the three main features in the abnormal records in the trusted space in the history are collected, the weights of the three main features are calculated, the data in the same content category all contain three types of data, and the formula is:

[0038] ;

[0039] In the formula, β represents the weight of each main feature calculated, Y i represents the number of records in which the main features change when the anomaly occurs, and Yz represents the total number of abnormal records;

[0040] S403, a data fusion model is constructed by using the weights of the main features of the three types of data, and the model is:

[0041] ;

[0042] In the formula, Zt represents the comprehensive feature vector after fusion, β1, β2 and β3 represent the weights of the main features of the numerical data, the text data and the video data, and Tz1, Tz2 and Tz3 represent the main features of the numerical data, the text data and the video data.

[0043] S500, collect the abnormal records occurring in the trusted space in the history, extract the three types of trusted data in the abnormal records, respectively extract the features of the three types of trusted data and input them into the data fusion model, and calculate the abnormal threshold;

[0044] Further, the specific steps of calculating the abnormal threshold are:

[0045] S501, collect the abnormal records occurring in the trusted space in the history, extract the three types of data in the abnormal records, input the features of the three types of data into the data fusion model by using the deep learning model, obtain the comprehensive feature vector in each record, calculate the abnormal threshold, and the formula is:

[0046] ;

[0047] In the formula, In represents the calculated abnormal threshold, Zsp represents the average value of the comprehensive feature vector in the history, and Zst represents the standard deviation of the comprehensive feature vector in the history.

[0048] S600, extract the trusted data of each content category stored, obtain the comprehensive feature vector through the data fusion model, and judge the real-time comprehensive feature vector by using the abnormal threshold, and issue a warning when it is judged that the trusted data has an abnormal risk.

[0049] Further, the specific steps of issuing a warning when judging that the trusted data has an abnormal risk are as follows:

[0050] S601, extract the stored data of each content classification, obtain a real-time comprehensive feature vector Zt through a data fusion model, and judge the real-time comprehensive feature vector using an abnormal threshold value; when Zt≥In, it is judged that the trusted data has an abnormal risk, and a warning is issued; when Zt<In, it is judged that the trusted data has no abnormal risk.

[0051] After the three types of data are fused by using the data fusion model, it is judged whether the trusted data has an abnormal risk, the data transaction operation is reviewed from multiple dimensions, and the abnormal signs are captured in a comprehensive manner instead of being limited to a one-sided perspective of a single data type. Moreover, isolated data types are difficult to reveal deep connections, and after fusion, the correlation between the generation of data products and related data elements and the frequent data transactions leading to data missing problems can be found, thereby creating conditions for accurately locating the abnormal source. Different types of data have their own advantages and disadvantages, and after fusion, they can be verified and supplemented with each other, reduce misjudgment, and significantly improve the accuracy of abnormal judgment.

[0052] A fusion intelligent lake warehouse system based on trusted data, the fusion intelligent lake warehouse system comprises a data collection module, a classification module, a storage module, a data fusion module, a threshold value calculation module and an abnormal judgment module;

[0053] The data collection module is used to collect all data when an abnormality occurs in the trusted space in the history;

[0054] The classification module is used to generate a classifier by judging the format of all data in the trusted space, classify the mixed data by format, and extract the main information of each data for content classification;

[0055] The storage module is used to store the trusted data classified by content by using blockchain technology;

[0056] The data fusion module is used to extract the features of all different format data in the same content, map the extracted features in the same space, reduce the dimension of the features in the same space, and construct a data fusion model;

[0057] The threshold value calculation module is used to extract three types of data in the abnormal record, extract the features of the three types of data respectively, input the features into the data fusion model, and calculate an abnormal threshold value;

[0058] The abnormal judgment module is used to judge the real-time comprehensive feature vector using the abnormal threshold value, and issue a warning when judging that the trusted data has an abnormal risk.

[0059] The classification module comprises a format classification unit and a content classification unit;

[0060] The format classification unit is used for judging the format of all data in the trusted space, and classifying the mixed data according to the format;

[0061] The content classification unit is used for extracting the main information of each data respectively, and classifying all real-time data collected according to the main information of each data.

[0062] The data fusion module comprises a main feature extraction unit and a model construction unit;

[0063] The main feature extraction unit is used for extracting the features of all data in the same content, calculating the feature value of each feature, and selecting the maximum feature value as the main feature;

[0064] The model construction unit is used for calculating the weight of each main feature, and constructing the data fusion model of the three types of data by using the weight.

[0065] Compared with the prior art, the beneficial effects of the present application are:

[0066] 1. The present application classifies the mixed data according to the format by using the classifier, and in the classification process, it is easier to find the data with format disorder and not conforming to the standard, so that the data quality can be maintained by cleaning or correcting in time.

[0067] 2. The present application can also find the hidden relationship between different services through content classification; the data content reflects the difference in sensitivity, and the more detailed security policy can be implemented according to the content classification.

[0068] 3. The present application judges whether the trusted data has abnormal risk by fusing the three types of data by using the data fusion model, and the operation of the product in the trusted space is examined from multiple dimensions, so that the abnormal signs can be captured from all directions instead of being limited to the one-sided view of a single data type. BRIEF DESCRIPTION OF DRAWINGS

[0069] Fig. 1 It is a module distribution diagram of the present application of a fusion intelligent lake warehouse system based on trusted data;

[0070] Fig. 2 It is a step schematic diagram of the present application of a fusion intelligent lake warehouse method based on trusted data. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0072] Example: Figs. 1-2 As shown, the present invention provides a technical solution.

[0073] A fusion-based intelligent lake warehouse method based on trusted data, the method comprising the following steps:

[0074] S100: Collect trusted data from all sensors in the trusted space, determine the format of the collected trusted data, and classify it into numerical trusted data, text trusted data, and video trusted data; train a classifier using the three types of trusted data from history, and classify the format of the real-time collected mixed trusted data.

[0075] The specific steps for format classification of real-time acquired mixed trusted data are as follows:

[0076] S101. Collect data from all sensors in the trusted space, determine the format of the collected data, and classify it into numerical data, text data, and video data; extract the format status of each type of data as its format features, use the format features of the three types of data as the training set, and calculate the information entropy of each type of data using the following formula:

[0077] ;

[0078] In the formula, H(X) represents the information entropy of each type of data, and p(x) represents the information entropy of each data type. i ) represents the xth i The probability of data type, where n represents the number of data points of each data type in the training set;

[0079] S102. After calculating the information entropy of each type of data, calculate the information gain of the three data format features using the following formula:

[0080] ;

[0081] In the formula, Gain(X, S) represents the information gain of each of the three data format features, S represents the format feature, the information gain of the three data format features is calculated, and the format feature with the largest information gain is selected as the classification criterion.

[0082] S103. The selected classification criteria are used as classification nodes, and the classification nodes are input into the decision tree to construct a classifier; after collecting all data in the confidence space in real time to obtain mixed data, the mixed data is input into the classifier to automatically classify the collected mixed data.

[0083] Faced with massive and complex trusted data, different data storage and access requirements vary across formats. By using a classifier to categorize mixed data according to format, encrypted product authorization key tables and API call key tables can be quickly stored in a relational database. Relying on its mature indexing system, target data can be accurately located within milliseconds. Text-based products use logs in a document database, allowing for rapid filtering based on document identifiers when querying logs from a specific product or time period. Videos have dedicated storage areas for easy management. During the classification process, it is easier to identify data with disordered or non-compliant formats, allowing for timely cleaning or correction and maintaining data quality.

[0084] S200. After classifying the real-time collected mixed trusted data, analyze the three types of trusted data after classification, extract the main information of each trusted data, and classify the content of all collected real-time trusted data according to the main information of each trusted data.

[0085] The specific steps for categorizing all collected real-time trusted data based on the key information of each trusted data point are as follows:

[0086] S201. After classifying the real-time collected mixed data, extract the information values ​​of all categories from the numerical data, calculate the average value of all categories of information values ​​as Zp, and then calculate the importance of each information value using the following formula:

[0087] ;

[0088] In the formula, Zd represents the importance of the calculated information value, Z represents each information value, and Zp represents the average value of all calculated information values. After calculating the importance of each information value, the information value with the highest importance is selected as the main information of the corresponding numerical data.

[0089] S202. In text data, extract all types of words contained in the text data, and calculate the importance of each type of word using the following formula:

[0090] ;

[0091] In the formula, Wd represents the importance of each word / character being calculated, and m j L represents the number of times the j-th word appears in the document, M represents the total number of words in the document, N represents the total number of documents, and L represents the total number of words in the document. j This indicates the number of documents containing the j-th type of word; the importance of all types of words in the text data is repeatedly calculated, and the word with the highest importance is selected as the main information of the text data;

[0092] S203, in the video type data, the video type data is frame processed, all key frames in the video type data are extracted by using the CNN model, objects in all key frame images are extracted, the total number of extracted object types is p, the importance of each object in the video type data is calculated, and the formula is:

[0093] ;

[0094] In the formula, Td represents the importance of each object in the calculated video type data, k represents the number of occurrences of each object in all key frames, and kz represents the total number of occurrences of all objects in all key frames; r min And r max represent the minimum and maximum gray values of each object in all key frames, R min And R max represent the minimum and maximum gray values of each key frame in all key frames; after calculating the importance of each object, the object with the largest importance value is selected as the main information of the video type data;

[0095] S204, when the main information of the numerical type data, the text type data and the video type data is calculated respectively, the information content corresponding to the main information is mapped to obtain the information content of each data main information, and the information content of the main information is used for division in the three types of data. The data with the same information content of the main information is divided into the same content data.

[0096] After being subdivided according to content, the manager can quickly lock the key business board, accurately grasp the operation details of specific business processes, and understand the data transaction process in the trusted space. Through content classification, hidden connections between different businesses can also be found. The data content reflects the difference in sensitivity, and the content classification facilitates the implementation of more detailed security policies.

[0097] S300, when the collected real-time trusted data is classified by content, the trusted data classified by content is stored by using the blockchain technology;

[0098] The specific steps of storing the trusted data classified by content by using the blockchain technology are as follows:

[0099] S301, after classifying the data in the trusted space by content, the data with the same content is stored in the same block by using the blockchain technology, and the hash value of each data is calculated by using the hash algorithm when storing the data, and the formula is: HA=SHA256(D), in the formula, HA represents the hash value of each data calculated, and D represents the original data of each data. When storing, the hash value of each data is stored correspondingly.

[0100] S400, after storing the trusted data classified according to the content, extracting the features of all different format trusted data in the same content, mapping the extracted features in the same space, reducing the dimension of the features in the same space, constructing a data fusion model, and fusing the features of all different format trusted data to generate a comprehensive feature vector;

[0101] The specific steps of fusing the features of all different format trusted data to generate a comprehensive feature vector are:

[0102] S401, after storing the data classified in real time, analyzing the stored data using a deep learning algorithm, extracting the features of all data in the same content category, setting the number of features extracted in each data as g, standardizing each extracted feature to form a feature matrix, and calculating the covariance matrix of each standardized feature matrix, the formula is:

[0103] ;

[0104] In the formula, F represents the covariance matrix of each feature matrix, e represents the number of each feature, V represents the feature matrix, V T represents the transpose matrix of the feature matrix; a characteristic equation is constructed, the eigenvalues and eigenvectors of the characteristic equation are calculated according to the covariance matrix, and the feature with the largest eigenvalue is selected as the main feature of the corresponding data;

[0105] S402, the main features of the three types of data in the same content category extracted through S401 are T1, T2 and T3; the extracted features are mapped in the same space, and the features in the same space are reduced in dimension; the three main features of the trusted data in the abnormal record in the history are collected, the weights of the three main features are calculated, and the data in the same content category all contain three types of data, the formula is:

[0106] ;

[0107] In the formula, β represents the weight of each main feature calculated, Y i represents the number of records in which the main feature changes when an anomaly occurs, and Yz represents the total number of abnormal records;

[0108] S403, a data fusion model is constructed using the weights of the main features of the three types of data, the model is:

[0109] ;

[0110] In the formula, Zt represents the comprehensive feature vector after fusion, β1, β2 and β3 represent the weights of the main features of the numerical data, the text data and the video data, and Tz1, Tz2 and Tz3 represent the main features of the numerical data, the text data and the video data.

[0111] S500, collect the abnormal records occurring in the trusted space in the history, extract three types of trusted data in the abnormal records, input the features of the three types of trusted data into the data fusion model respectively, and calculate an abnormal threshold value;

[0112] The specific steps of calculating the abnormal threshold value are as follows:

[0113] S501, collect the abnormal records occurring in the trusted space in the history, extract three types of data in the abnormal records, input the features of the three types of data into the data fusion model by using a deep learning model, obtain a comprehensive feature vector in each record, calculate an abnormal threshold value, and the formula is as follows:

[0114] ;

[0115] In the formula, In represents the calculated abnormal threshold value, Zsp represents the average value of the comprehensive feature vector in the history, and Zst represents the standard deviation of the comprehensive feature vector in the history.

[0116] S600, extract the trusted data of each content classification stored, obtain a comprehensive feature vector after the data fusion model, and judge the real-time comprehensive feature vector by using the abnormal threshold value, and issue a warning when it is judged that the trusted data has an abnormal risk.

[0117] The specific steps of issuing a warning when it is judged that the trusted data has an abnormal risk are as follows:

[0118] S601, extract the data of each content classification stored, obtain a real-time comprehensive feature vector Zt after the data fusion model, judge the real-time comprehensive feature vector by using the abnormal threshold value, and issue a warning when it is judged that the trusted data has an abnormal risk; and when Zt<In, it is judged that the trusted data has no abnormal risk.

[0119] After the three types of data are fused by using the data fusion model, it is judged whether the trusted data has an abnormal risk, the data transaction operation is reviewed from multiple dimensions, and the abnormal signs are captured in a full range. And the isolated data type is difficult to show the deep connection, after fusion, the correlation between the data product online and the generation of related data elements and the frequent data transaction leading to the data missing problem can be found, conditions for accurately positioning the abnormal root cause are created; different types of data have advantages and disadvantages, and after fusion, they are mutually verified and supplemented, false positives are reduced, and the accuracy of abnormal judgment is significantly improved.

[0120] A fusion intelligent lake warehouse system based on trusted data, the fusion intelligent lake warehouse system comprises a data collection module, a classification module, a storage module, a data fusion module, a threshold value calculation module, and an abnormal judgment module;

[0121] The data collection module is used for collecting all data when an anomaly occurs in the trusted space in history;

[0122] The classification module is used for generating a classifier by judging the format of all data in the trusted space, classifying the mixed data by format, and extracting main information of each data for content classification;

[0123] The storage module is used for storing the trusted data classified by content by using a blockchain technology;

[0124] The data fusion module is used for extracting features of all different format data in the same content, mapping the extracted features in the same space, reducing the features in the same space, and constructing a data fusion model;

[0125] The threshold calculation module is used for extracting three types of data in the anomaly record, extracting features of the three types of data respectively and inputting the features into the data fusion model, and calculating an anomaly threshold;

[0126] The anomaly judgment module is used for judging real-time comprehensive feature vectors by using the anomaly threshold, and issuing a warning when it is judged that the trusted data has an abnormal risk.

[0127] The classification module includes a format classification unit and a content classification unit;

[0128] The format classification unit is used for judging the format of all data in the trusted space, and classifying the mixed data by format;

[0129] The content classification unit is used for extracting main information of each data respectively, and classifying all collected real-time data by content according to the main information of each data.

[0130] The data fusion module includes a main feature extraction unit and a model construction unit;

[0131] The main feature extraction unit is used for extracting features of all data in the same content, calculating a feature value of each feature, and selecting a maximum feature value as a main feature;

[0132] The model construction unit is used for calculating a weight of each main feature, and constructing a data fusion model of three types of data by using the weight.

[0133] Embodiment: A trusted space is monitored in real time for an anomaly, and numerical, text and video data in the trusted space are extracted respectively; all data are classified into product, transaction and personnel categories according to the content of the data; each content classification includes numerical, text and video data;

[0134] When collecting the trusted data in real time, the data of a product class is provided, including product record table, transaction log and operation record; the main features of the three kinds of data are extracted as transaction time, transaction record and operation video; the three main features are mapped in the same space as transaction time 8:00, operation time 50 min and data product number 80; the three kinds of data are converted into feature vectors;

[0135] The abnormal threshold is calculated as 80 through the abnormal record in the history;

[0136] The weight is calculated through the abnormal record in the history, the data fusion model Zt is obtained as Zt=β1×Tz1+β2×Tz1+β3×Tz1; the real-time comprehensive feature vector 95 is calculated; and it is judged that the trusted data has abnormal risk.

[0137] It is apparent for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all aspects as illustrative and not restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended to encompass all changes falling within the meaning and scope of the equivalent elements of the claims. Any reference signs in the claims should not be considered as limiting the involved claims.

Claims

1. A fusion-based intelligent lakehouse method based on trusted data, characterized in that: The method includes the following steps: S100: Collect reliable data from all sensors, determine the format of the collected reliable data, and classify it into numerical reliable data, text reliable data, and video reliable data; train a classifier using the three types of reliable data from history, and classify the format of the real-time collected mixed reliable data. S200. After classifying the real-time collected mixed trusted data, analyze the three types of trusted data after classification, extract the main information of each trusted data, and classify the content of all collected real-time trusted data according to the main information of each trusted data. The specific steps for categorizing all collected real-time trusted data based on the key information of each trusted data point are as follows: S201. After classifying the real-time collected mixed trusted data, extract all types of information values ​​from the numerical trusted data, calculate the average value of all types of information values ​​as Zp, and then calculate the importance of each type of information value using the following formula: ; In the formula, Zd represents the importance of the calculated information value, Z represents each information value, and Zp represents the average value of all calculated information values. After calculating the importance of each information value, the information value with the highest importance is selected as the main information of the corresponding numerical reliable data. S202. In text-based trusted data, extract all types of words contained in the text-based trusted data, and calculate the importance of each type of word. The formula is as follows: ； In the formula, Wd represents the importance of each word / character being calculated, and m j L represents the number of occurrences of the j-th word in the document, M represents the total number of characters in the document, N represents the total number of documents, and L represents the total number of occurrences of the word. j This indicates the number of documents containing the j-th type of word; the importance of all types of words in the text-based trusted data is repeatedly calculated, and the word with the highest importance is selected as the main information of the text-based trusted data; S203. In the trusted video data, the trusted video data is processed by frame segmentation. A CNN model is used to extract all keyframes from the trusted video data, and objects are extracted from all keyframe images. Let the total number of extracted object types be p. Calculate the importance of each object type in the trusted video data using the following formula: ； In the formula, Td represents the importance of each object in the calculated video-based reliable data, k represents the number of times each object appears in all keyframes, and kz represents the total number of times all objects appear in all keyframes; r min and r max R represents the minimum and maximum grayscale values ​​of each object in all keyframes. min and R max This represents the minimum and maximum grayscale values ​​of each keyframe in all keyframes; after calculating the importance of each object, the object with the highest importance value is selected as the main information of the video-type reliable data; S204. After calculating the main information of numerical trusted data, text trusted data and video trusted data respectively, the information content corresponding to the main information is used for mapping to obtain the information content of the main information of each trusted data. The information content of the main information is used to divide the three types of trusted data, and trusted data with the same information content of the main information are divided into trusted data with the same content. S300. After the collected real-time trusted data is classified, the trusted data after content classification is stored using blockchain technology. S400. After storing the credible data after content classification, extract the features of all data in different formats in the same content, map the extracted features to the same space, reduce the dimensionality of the features in the same space, construct a data fusion model, and fuse the features of all credible data in different formats to generate a comprehensive feature vector. S500: Collect abnormal records of credible data occurrences in history, extract three types of credible data from the abnormal records, extract the features of the three types of credible data respectively and input them into the data fusion model, and calculate the abnormal threshold. S600: Extract trusted data for each content category from the storage, obtain a comprehensive feature vector through the data fusion model, and use an anomaly threshold to judge the real-time comprehensive feature vector. When it is judged that there is an anomaly risk in the trusted data, an early warning is issued.

2. The method for a fusion-based intelligent lakehouse based on trusted data according to claim 1, characterized in that: The specific steps for format classification of the real-time acquired mixed reliable data in S100 are as follows: S101. Collect all reliable data acquired by the sensors, and determine the format of the collected reliable data, classifying it into numerical reliable data, text reliable data, and video reliable data. The format states of three types of reliable data are extracted as three data format features. These three types of reliable data format features are used as the training set. The information entropy of each type of reliable data is calculated using the following formula: ； In the formula, H(X) represents the information entropy of each type of reliable data, and p(x) represents the information entropy of each type of reliable data. i ) represents the xth i The probability of a type of trusted data, where n represents the number of data points in the training set for each type of trusted data; S102. After calculating the information entropy of each type of trusted data, calculate the information gain of the three trusted data format features, using the following formula: ； In the formula, Gain(X, S) represents the information gain of each of the three reliable data format features, S represents the format feature, the information gain of the three reliable data format features is calculated, and the format feature with the largest information gain is selected as the classification criterion. S103. Use the selected classification criteria as classification nodes, and input the classification nodes into the decision tree to construct a classifier; After obtaining mixed trusted data based on all trusted data collected in real time, the mixed trusted data is input into a classifier to automatically classify the collected mixed trusted data.

3. The method for a fusion-based intelligent lakehouse based on trusted data according to claim 2, characterized in that: The specific steps in S300 for storing trusted data after content classification using blockchain technology are as follows: S301. After classifying the trusted data, blockchain technology is used to store trusted data with the same content in the same block. When storing trusted data, a hash algorithm is used to calculate the hash value of each data. The formula is: HA=SHA256(D). In the formula, HA represents the hash value of each type of trusted data and D represents the original data of each type of trusted data. When storing, the hash value of each type of trusted data is stored accordingly.

4. The method for a fusion-based intelligent lakehouse based on trusted data according to claim 3, characterized in that: The specific steps in S400 for fusing features from all trusted data in different formats to generate a comprehensive feature vector are as follows: S401. After classifying and storing the real-time collected reliable data, a deep learning algorithm is used to analyze the stored reliable data, extracting features from all reliable data within the same content category. Let g be the number of features extracted from each reliable data. Each extracted feature is standardized to form a feature matrix. The covariance matrix of each standardized feature matrix is ​​calculated using the following formula: ； In the formula, F represents the covariance matrix of each feature matrix, e represents the number of each feature, and V represents the feature matrix. T The transpose of the characteristic matrix; Construct a feature equation, calculate the eigenvalues ​​and eigenvectors of the feature equation based on the covariance matrix, and select the feature with the largest eigenvalue as the main feature of the corresponding reliable data. S402. Following step S401, extract the main features of three types of credible data within the same content category as T1, T2, and T3; map the extracted features to the same space, and perform dimensionality reduction on the features within the same space; collect the three main features from historical records of anomalous credible data, calculate the weights of the three main features, and ensure that the data within the same content category contains all three types of credible data. The formula is: ； In the formula, β represents the weight of each principal feature, and Y... i Yz represents the number of records whose main characteristics changed when an anomaly occurred; Yz represents the total number of anomaly records. S403. Construct a data fusion model using the weights of the main features of the three types of reliable data. The model is as follows: ； In the formula, Zt represents the fused comprehensive feature vector, β1, β2, and β3 represent the weights of the main features of numerical credible data, text credible data, and video credible data, and Tz1, Tz2, and Tz3 represent the main features of numerical credible data, text credible data, and video credible data.

5. The method for a fusion-based intelligent lakehouse based on trusted data according to claim 4, characterized in that: The specific steps for calculating the anomaly threshold in S500 are as follows: S501. Collect the anomaly records of historical credible data, extract the three types of credible data in the anomaly records, input the features of the three types of credible data into the data fusion model using a deep learning model, obtain the comprehensive feature vector for each record, and calculate the anomaly threshold. The formula is: ； In the formula, In represents the calculated anomaly threshold, Zsp represents the average value of the comprehensive feature vector in history, and Zst represents the standard deviation of the comprehensive feature vector in history.

6. The method for a fusion-based intelligent lakehouse based on trusted data according to claim 5, characterized in that: The specific steps for issuing a warning when it is determined that there is an anomaly risk in credible data in S600 are as follows: S601. Extract the credible data of each content classification stored, obtain the real-time comprehensive feature vector Zt through the data fusion model, and use the anomaly threshold to judge the real-time comprehensive feature vector. When Zt ≥ In, it is determined that there is an anomaly risk in the credible data, and a warning is issued; when Zt < In, it is determined that there is no anomaly risk in the credible data.

7. A fusion intelligent lakehouse system based on trusted data, applying the fusion intelligent lakehouse method based on trusted data as described in any one of claims 1-6, characterized in that: The fusion intelligent lakehouse system includes a data collection module, a classification module, a storage module, a data fusion module, a threshold calculation module, and an anomaly judgment module; The data collection module is used to collect all credible data when an anomaly occurs historically; The classification module is used to generate a classifier by judging the formats of all credible data, classify the formats of the mixed credible data; and extract the main information of each type of credible data for content classification; The storage module is used to store the content-classified credible data using blockchain technology; The data fusion module is used to extract the features of all credible data in the same content with different formats, map the extracted features to the same space, reduce the dimensions of the features in the same space, and construct a data fusion model; The threshold calculation module is used to extract the three types of credible data in the anomaly records, respectively extract the features of the three types of credible data and input them into the data fusion model, and calculate the anomaly threshold; The anomaly judgment module is used to judge the real-time comprehensive feature vector using the anomaly threshold, and issue a warning when it is determined that there is an anomaly risk in the credible data.

8. The fusion intelligent lake warehouse system based on trusted data according to claim 7, characterized in that: The classification module includes a format classification unit and a content classification unit; The format classification unit is used to judge the data formats of all credible data and classify the formats of the mixed credible data; The content classification unit is used to respectively extract the main information of each credible data, and classify the content of all the collected real-time data according to the main information of each credible data.

9. The fusion intelligent lake warehouse system based on trusted data according to claim 7, characterized in that: The data fusion module includes a main feature extraction unit and a model construction unit; The main feature extraction unit is used to extract the features of all credible data in the same content, calculate the feature values of each feature, and select the one with the largest feature value as the main feature; The model building unit is used to calculate the weight of each main feature and use the weights to build a data fusion model for three types of reliable data.

Images