Data discovery and identification method

Abstract

The invention relates to the technical field of data processing, in particular to a data discovery and identification method, which comprises the following steps of: scanning and identifying a data source in a network environment through a hybrid algorithm based on machine learning and deep learning, and extracting key characteristics of the data source by adopting an automatic characteristic engineering technology; the method comprises the following steps: acquiring data sources, storing the data sources in a characteristic library, classifying the data sources into structured data, semi-structured data and unstructured data by utilizing an adaptive classification algorithm in combination with fuzzy logic and a Bayesian classifier, automatically marking the data sources, performing quality evaluation on the classified data sources, generating a priority list, and storing the priority list into a database; data extraction, data conversion and data loading are carried out on the data sources with high priorities, and it is ensured that the integrated data has high consistency and high reliability. According to the method, intelligent discovery, accurate classification, dynamic evaluation and efficient integration of data sources are realized through a systematic process, and the method is suitable for data management and business decision support in a big data environment.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for data discovery and identification. Background Art

[0002] With the rapid development of information technology, the amount of data generated and processed by enterprises and organizations in their daily operations has exploded. These data sources include structured data (such as database records), semi-structured data (such as JSON, XML files), and unstructured data (such as text files, images, videos, etc.). In the big data environment, how to efficiently and accurately discover, identify and integrate these diverse data sources has become the key to improving business insights and decision support capabilities. However, due to the diversity and complexity of data sources, traditional data management methods are difficult to cope with the rapid growth and changes of data in modern enterprise environments, which poses new challenges to data discovery, classification, quality assessment and integration.

[0003] Existing technologies face several significant technical problems when dealing with diverse data sources. First, traditional data discovery methods rely on manual rules or simple algorithms, which make it difficult to efficiently and accurately identify and classify various data sources, especially in complex network environments. Second, the feature extraction and quality assessment of data sources often lack intelligent and automated means, resulting in inefficient data processing and prone to errors and inconsistencies. In addition, when dealing with massive amounts of data, existing integration methods often lack effective prioritization and resource optimization strategies, resulting in delayed processing of key data and failure to support business decisions in a timely manner. Summary of the invention

[0004] Based on the above objectives, the present invention provides a method for data discovery and identification.

[0005] A method for data discovery and identification comprises the following steps:

[0006] S1, data source identification and scanning: Scan and identify data sources in the network environment through a hybrid algorithm based on machine learning and deep learning;

[0007] S2, automatic extraction of data source characteristics: Extract characteristics of the identified data source, including data format, data type, data structure and data scale. Automatic feature engineering technology is used to automatically extract the characteristics of structured data and unstructured data through natural language processing and image recognition technology, and the characteristics are stored in the feature library.

[0008] S3, data source classification and labeling: Based on the extracted features, an adaptive classification algorithm is used to classify data sources into structured data, semi-structured data, and unstructured data. During the classification process, fuzzy logic and Bayesian classifiers are combined to improve the accuracy and robustness of classification. At the same time, data sources are automatically labeled and label information is generated to facilitate subsequent processing and management;

[0009] S4, data source quality assessment: perform quality assessment on the classified data sources. The assessment indicators include data integrity, data consistency, data accuracy, and data freshness. During the quality assessment process, statistical analysis and machine learning models are combined to dynamically adjust the assessment parameters to ensure the accuracy and real-time nature of the assessment results.

[0010] S5, data source prioritization: prioritize data sources based on quality assessment results and data usage requirements, and use the analytic hierarchy process based on multi-factor decision analysis to generate a priority list by comprehensively considering data quality, data importance, and data availability;

[0011] S6, data source integration: Integrate high-priority data sources, including data extraction, data transformation and data loading.

[0012] Optionally, the S1 specifically includes:

[0013] S11, data collection and preprocessing: collect data from network traffic, file systems and databases, and perform preprocessing;

[0014] S12, Feature Extraction and Modeling: Extract features from network traffic using convolutional neural networks and classify file system and database data using support vector machines;

[0015] S13, hybrid algorithm integration: the outputs of the convolutional neural network and the support vector machine are fused to form the final recognition model;

[0016] S14, data source scanning and identification: Apply hybrid algorithms to comprehensively scan the network environment and identify potential data sources.

[0017] Optionally, the S2 specifically includes:

[0018] S21, data format identification: data format identification includes file format identification and database format identification, wherein;

[0019] File format recognition: Use file header information (such as the first few bytes of a file) to identify common file formats;

[0020] Database format identification: Identify the database format by querying the database metadata, such as table structure, field type, etc.;

[0021] S22, data type identification: data type identification includes structured data type identification and unstructured data type identification, wherein;

[0022] Structured data type identification: Identify data types through statistical characteristics of data (such as value distribution, mean, standard deviation, etc.). For example, integers and floating-point numbers have different distribution characteristics, string data usually contains non-numeric characters, and date data conforms to a specific date format.

[0023] Unstructured data type identification: For text data, natural language processing techniques such as word frequency analysis and named entity recognition are used to identify the type of text content. For image data, pre-trained convolutional neural network models are used to identify the category of images by inputting images and extracting their feature vectors.

[0024] S23, data structure identification: data structure identification includes table structure identification and nested structure identification, wherein;

[0025] Table structure identification: By querying the metadata of the database, the structural information of the table is extracted, including the table name, field name, field type, primary key and foreign key. This information is used to describe the structure and relationship of the table;

[0026] Nested structure recognition: For JSON and XML nested data structures, use a recursive algorithm to parse the nesting level and node information to identify the nested structure and hierarchical relationship of the data;

[0027] S24, data scale estimation: data scale estimation includes data record number estimation and file size estimation, wherein;

[0028] Estimation of the number of data records: For CSV files, estimate the number of records by reading the number of file lines; for database tables, obtain the number of records by executing SQL query statements;

[0029] File size estimation: Get the file size information through the file attribute interface provided by the operating system;

[0030] S25, Automatic feature engineering technology: Automatic feature engineering technology includes natural language processing feature extraction and image recognition feature extraction, among which;

[0031] Natural language processing feature extraction: Use the bag-of-words model and word vector model to extract features of text data;

[0032] Image recognition feature extraction: Use convolutional neural networks to extract features from image data;

[0033] S26, storing the features in the feature library: selecting a relational database MySQL to store the extracted features, and storing the extracted features in the feature library in the form of key-value pairs.

[0034] Optionally, the S3 specifically includes:

[0035] S31, feature standardization: Before classification, the extracted features are standardized so that different features can be compared under the same dimension;

[0036] S32, adaptive classification algorithm: The adaptive classification algorithm dynamically adjusts the classification rules according to the characteristics of the data source;

[0037] S33, fuzzy logic processing: In order to improve the accuracy of classification, fuzzy logic is introduced for the unclear data source characteristics, the characteristic values ​​are converted into fuzzy sets, and the degree of the characteristics belonging to a certain category is expressed by the membership function;

[0038] S34, Bayesian classifier: The robustness of classification is improved by combining the Bayesian classifier, which determines the category of the data source by calculating the posterior probability;

[0039] S35, automatic marking and label generation: After the classification is completed, the data source is automatically marked to generate label information.

[0040] Optionally, in S4, the quality of the classified data sources is evaluated, and the evaluation indicators include data integrity, data consistency, data accuracy and data freshness, specifically including:

[0041] S41, Data integrity: Data integrity refers to whether there are missing or null values ​​in the data set, and whether all the data are collected and saved;

[0042] S42, Data consistency: Data consistency refers to whether data is consistent across different sources or at different points in time;

[0043] S43, Data accuracy: Data accuracy refers to the degree of consistency between data and the actual situation, which needs to be calculated by comparing with reliable reference data;

[0044] S44, Data freshness: Data freshness refers to the timeliness of data, evaluating whether the data is updated in a timely manner.

[0045] Optionally, in the quality assessment process in S4, statistical analysis and machine learning models are combined to dynamically adjust assessment parameters, specifically including:

[0046] S45, Statistical analysis: Statistical analysis methods are used to evaluate the basic conditions of various quality indicators and judge the data quality through the mean and variance of the data;

[0047] S46, Machine Learning Model: In order to improve the accuracy of the evaluation, a machine learning model is used to predict and correct data quality indicators. Through supervised learning methods, the model learns the relationship between data characteristics and quality, and dynamically adjusts the evaluation parameters;

[0048] S47, real-time and accuracy assurance: through real-time monitoring and online learning, the evaluation parameters are continuously optimized to adapt to changes in data sources;

[0049] S48, quality score output: Comprehensively calculate the overall quality score of the data source by combining various quality indicators;

[0050] S49, Quality report generation: Generate a report based on the quality assessment results, point out the advantages and disadvantages of the data source, and make suggestions for improvement.

[0051] Optionally, the S5 specifically includes:

[0052] S51, constructing a hierarchical model: The first step of the AHP is to construct a hierarchical model, dividing the factors affecting the priority of data sources into the target layer, the criterion layer, and the scheme layer;

[0053] S52, constructing a judgment matrix: comparing the relative importance of each pair of criteria and constructing a judgment matrix;

[0054] S53, calculating a weight vector: calculating a weight vector of each criterion by normalizing and weighted summing the judgment matrix;

[0055] S54, consistency check: In order to ensure the consistency of the judgment matrix, a consistency check is required, and the consistency ratio of the judgment matrix must be less than 0.1;

[0056] S55, calculating the comprehensive score of the data source: for each data source, calculating the comprehensive score according to its score under each criterion and the weight of the criterion;

[0057] S56, generating a priority list: sorting the data sources according to their comprehensive scores to obtain a priority list of data sources.

[0058] Optionally, the S6 specifically includes:

[0059] S61, data extraction: obtain full or incremental data from high-priority data sources and extract it to intermediate storage or processing platforms;

[0060] S62, data conversion: clean, format convert and normalize the extracted data to ensure data consistency and applicability;

[0061] S63, data loading: Load the converted data into the target storage system to ensure that it is available for subsequent analysis and query.

[0062] Optionally, the high-priority data sources in S6 are obtained by calculating the comprehensive scores of the data sources in S55, and data sources with scores in the top 20% are regarded as high priority.

[0063] Beneficial effects of the present invention:

[0064] The present invention realizes the automatic identification and scanning of various data sources in the network environment through a hybrid algorithm based on machine learning and deep learning. It combines the adaptive classification algorithm with fuzzy logic and Bayesian classifier, which not only improves the accuracy and robustness of data classification, but also makes the discovery, classification and management process of data sources more intelligent and efficient, reduces the need for manual intervention, and greatly improves the efficiency of overall data processing.

[0065] The present invention can effectively screen and manage high-priority data sources through systematic data source quality assessment and priority sorting, ensuring that important data is given priority and resources are allocated during the integration process. This not only improves the accuracy and efficiency of data integration, but also enables data to better serve business decision-making and analysis, enhancing the company's ability to respond and competitive advantage in a rapidly changing market environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of the S5 process of an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0070] It should be noted that the references to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0071] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0072] like Figure 1-Figure 2 As shown, a method for data discovery and identification includes the following steps:

[0073] S1, data source identification and scanning: Scan and identify data sources in the network environment through a hybrid algorithm based on machine learning and deep learning;

[0074] S2, automatic extraction of data source characteristics: Extract characteristics of the identified data source, including data format, data type, data structure and data scale. Automatic feature engineering technology is used to automatically extract the characteristics of structured data and unstructured data through natural language processing and image recognition technology, and the characteristics are stored in the feature library.

[0075] S3, data source classification and labeling: Based on the extracted features, an adaptive classification algorithm is used to classify data sources into structured data, semi-structured data, and unstructured data. During the classification process, fuzzy logic and Bayesian classifiers are combined to improve the accuracy and robustness of classification. At the same time, data sources are automatically labeled and label information is generated to facilitate subsequent processing and management;

[0076] S4, data source quality assessment: perform quality assessment on the classified data sources. The assessment indicators include data integrity, data consistency, data accuracy, and data freshness. During the quality assessment process, statistical analysis and machine learning models are combined to dynamically adjust the assessment parameters to ensure the accuracy and real-time nature of the assessment results.

[0077] S5, data source prioritization: prioritize data sources based on quality assessment results and data usage requirements, and use the analytic hierarchy process based on multi-factor decision analysis to generate a priority list by comprehensively considering data quality, data importance, and data availability;

[0078] S6, data source integration: Integrate high-priority data sources, including data extraction, data transformation and data loading.

[0079] S1 specifically includes:

[0080] S11, data collection and preprocessing: collect data from network traffic, file system and database, and preprocess them. The specific steps are as follows:

[0081] Network traffic data collection: Use network monitoring tools (such as Wireshark) to capture network traffic data, including HTTP requests, FTP traffic, SMTP traffic, etc.

[0082] File system data collection: collects file system metadata, including file type, file size, creation time, etc.

[0083] Database data collection: extract metadata from the database, such as table structure, field type, number of records, etc.;

[0084] Data preprocessing: remove noise and redundant information, clean the data, and standardize and normalize the data;

[0085] S12, feature extraction and modeling: Use convolutional neural networks to extract features from network traffic and use support vector machines to classify file system and database data. The specific steps are as follows:

[0086] (1) Constructing a convolutional neural network model:

[0087] Network architecture design includes:

[0088] Input layer: receives network traffic data;

[0089] Convolution layer: extract features through convolution kernel;

[0090] Pooling layer: downsampling to reduce feature dimensions;

[0091] Fully connected layer: integrates the extracted features;

[0092] Output layer: output recognition results;

[0093] Model training: Use large-scale labeled datasets to train convolutional neural network models, and use back-propagation algorithms to optimize model parameters;

[0094] (2) Constructing the support vector machine model:

[0095] Feature selection: Select key features from file system and database data, such as file type, field type, etc.

[0096] Model training: Use labeled data sets to train the support vector machine model and use kernel functions (such as RBF kernel) to improve classification results;

[0097] The support vector machine decision function is expressed as:

[0098] Among them, α i is the Lagrange multiplier, y i is the training sample label, K(x i ,x) is the kernel function, b is the bias term;

[0099] The kernel function (taking RBF kernel as an example) is expressed as: K(x i ,x j ) = exp(-γ||x i -x j || 2 );

[0100] where γ is the kernel parameter, ||x i -x j || 2 is the square of the Euclidean distance;

[0101] S13, hybrid algorithm integration: The outputs of the convolutional neural network and the support vector machine are integrated to form the final recognition model, including:

[0102] (1) Feature fusion: The network traffic features extracted by the convolutional neural network are fused with the file system and database features selected by the support vector machine to generate a comprehensive feature vector;

[0103] (2) Integrated model: Use integrated learning methods (such as random forest) to combine the output results of convolutional neural networks and support vector machines to improve recognition accuracy and robustness;

[0104] Random Forest: Decision tree construction is represented as:

[0105] Where M is the number of decision trees, h m is the mth decision tree;

[0106] The loss function is expressed as:

[0107] Where N is the number of samples, y i is the true label, is the predicted value;

[0108] S14, data source scanning and identification: Apply a hybrid algorithm to comprehensively scan the network environment and identify potential data sources. The specific steps are as follows:

[0109] (1) Initialization: load the pre-trained convolutional neural network and SVM model, and set the scanning parameters and thresholds;

[0110] (2) Real-time monitoring and data collection: real-time monitoring of network traffic, capturing and parsing data packets, scanning file systems and databases, and extracting metadata;

[0111] (3) Feature extraction and classification: Use convolutional neural networks to extract features from network traffic data, use support vector machines to classify file system and database data, and input the extracted features into an integrated model for comprehensive classification and identification.

[0112] (4) Result output and display: Output the recognition results to a log file or database, generate a recognition report, and display the recognition results in real time.

[0113] S2 specifically includes:

[0114] S21, data format identification: data format identification includes file format identification and database format identification, wherein;

[0115] File format identification: Use file header information (such as the first few bytes of a file) to identify common file formats (such as CSV, JSON, XML, PDF, JPEG, etc.). Regular expressions can be used to match the structural features of the file content to further confirm the file format, expressed as:

[0116] File_format=identify_format(file_header,regex);

[0117] Among them, file_header is the header information of the file, and regex is a regular expression;

[0118] Database format identification: The database format can be identified by querying the database metadata, such as table structure, field type, etc. The database management system provides interfaces for obtaining metadata, which can be used to extract table structure information;

[0119] S22, data type identification: data type identification includes structured data type identification and unstructured data type identification, wherein;

[0120] Structured data type identification: Identify data types through statistical characteristics of data (such as value distribution, mean, standard deviation, etc.). For example, integers and floating-point numbers have different distribution characteristics, string data usually contains non-numeric characters, and date data conforms to a specific date format.

[0121] Unstructured data type identification: For text data, natural language processing techniques such as word frequency analysis and named entity recognition are used to identify the type of text content. For image data, pre-trained convolutional neural network models are used to identify the category of images by inputting images and extracting their feature vectors.

[0122] S23, data structure identification: data structure identification includes table structure identification and nested structure identification, wherein;

[0123] Table structure identification: By querying the metadata of the database, the structural information of the table is extracted, including the table name, field name, field type, primary key and foreign key. This information is used to describe the structure and relationship of the table;

[0124] Nested structure recognition: For JSON and XML nested data structures, use a recursive algorithm to parse the nesting level and node information to identify the nested structure and hierarchical relationship of the data;

[0125] S24, data scale estimation: data scale estimation includes data record number estimation and file size estimation, wherein;

[0126] Estimation of the number of data records: For CSV files, estimate the number of records by reading the number of file lines; for database tables, obtain the number of records by executing SQL query statements;

[0127] File size estimation: Get the file size information through the file attribute interface provided by the operating system;

[0128] S25, Automatic feature engineering technology: Automatic feature engineering technology includes natural language processing feature extraction and image recognition feature extraction, among which;

[0129] Natural language processing feature extraction: Use the bag-of-words model and word vector model to extract features of text data. The bag-of-words model represents text as word frequency vectors, while the word vector model represents words as real number vectors to capture the semantic relationship between words.

[0130] Image recognition feature extraction: Use convolutional neural networks to extract features of image data. Convolutional neural networks extract features at different levels from images through multi-layer convolution and pooling operations, and finally obtain high-level image feature vectors;

[0131] S26, features are stored in the feature library: the relational database MySQL is selected to store the extracted features, and the extracted features are stored in the feature library in the form of key-value pairs. For example, the file path can be used as the key to store the corresponding feature values ​​such as data format, data type, data structure and data scale. The records in the feature library should include the unique identifier of the data source, the extracted features and their values.

[0132] S3 specifically includes:

[0133] S31, feature standardization: Before classification, the extracted features are standardized so that different features can be compared under the same dimension, expressed as:

[0134]

[0135] Where x is the original characteristic value, μ is the mean value of the characteristic, σ is the standard deviation of the characteristic, and x′ is the standardized characteristic value;

[0136] S32, Adaptive classification algorithm: The adaptive classification algorithm dynamically adjusts the classification rules according to the characteristics of the data source. The following are the main steps of classification:

[0137] (1) Structured data classification: Determine whether the data source is in a tabular format (such as a database table, CSV file, etc.) and whether there is a clear relationship between fields (such as a primary key-foreign key relationship), expressed as:

[0138]

[0139] Where x is the data source feature, is_tabular(x) determines whether the data source is in tabular form, and has_relations(x) determines whether there is a relationship between fields.

[0140] (2) Semi-structured data classification: Determine whether the data source has a certain hierarchical structure, such as JSON and XML files. These data sources have some structured information, but the field relationship is not as strict as that of table data, which can be expressed as:

[0141]

[0142] Among them, has_hierarchy(x) determines whether the data source has a hierarchical structure;

[0143] (3) Unstructured data classification: Determine whether the data source lacks a clear structure, such as text, images, audio, etc. These data sources do not meet the characteristics of structured or semi-structured data, expressed as:

[0144]

[0145] Among them, is_unstructured(x) determines whether the data source is unstructured data;

[0146] S33, fuzzy logic processing: In order to improve the accuracy of classification, fuzzy logic is introduced for the unclear data source characteristics, the characteristic value is converted into a fuzzy set, and the degree of the characteristic belonging to a certain category is expressed by the membership function. The membership function formula is expressed as:

[0147]

[0148] Among them, μ A (x) represents the membership of data source feature x to category A, a, b, c are fuzzy logic parameters, which control the slope, shape and center position of the membership respectively;

[0149] Fuzzy rules: fuzzy reasoning is performed based on the membership of the feature, and the data source is classified into the category with the highest membership;

[0150] S34, Bayesian classifier: The robustness of classification is improved by combining the Bayesian classifier. The Bayesian classifier determines the category of the data source by calculating the posterior probability. The Bayesian formula is expressed as:

[0151]

[0152] Among them, P(C i |x) is the data source belongs to category C given feature x i The posterior probability of i ) is a characteristic in category C i The conditional probability under i ) is Category C i The prior probability of feature x; P(x) is the marginal probability of feature x;

[0153] Maximum a posteriori probability classification: The data source is classified into the category with the largest a posteriori probability, expressed as:

[0154]

[0155] S35, automatic labeling and label generation: After the classification is completed, the data source is automatically labeled to generate label information. The specific steps are as follows:

[0156] (1) Label generation: Generate corresponding labels based on the classification results of the data source and attach them to the metadata of the data source, expressed as:

[0157] Label(x)=generate label (C(x));

[0158] Among them, Label(x) is the label of data source x, and C(x) is the classification result of the data source;

[0159] Label structure: Labels can include information such as the category of the data source, key features, and classification confidence, expressed as:

[0160] Label={Category:C(x),Key Features:{f1,f2,…,f n},Confidence:P(C i |x)};

[0161] Through the above steps, data sources can be classified into structured data, semi-structured data and unstructured data according to their characteristics. Fuzzy logic and Bayesian classifiers are combined to handle unclear and complex classification situations, further improving the accuracy and robustness of classification. After the classification is completed, label information is automatically generated and attached to the metadata of the data source for subsequent processing and management.

[0162] In S4, the quality of the classified data sources is evaluated. The evaluation indicators include data integrity, data consistency, data accuracy and data freshness, including:

[0163] S41, Data integrity: Data integrity refers to whether there are missing values ​​or null values ​​in the data set, whether all data are collected and saved, and the integrity measurement formula is expressed as:

[0164]

[0165] Where I is the data integrity percentage, N valid is the number of valid records (no missing values), N total is the total number of records;

[0166] S42, Data consistency: Data consistency refers to whether data is consistent across different sources or at different time points. For example, whether the values ​​of the same field in multiple data sources are the same. The consistency measurement formula is expressed as:

[0167]

[0168] Among them, C is the consistency measure, M is the number of matching pairs, and x i and i are the values ​​of the same field from different sources 1(x i =y i ) is the indicator function, when x i and i If they are equal, the value is 1, otherwise it is 0;

[0169] S43, Data accuracy: Data accuracy refers to the degree of consistency between data and the actual situation, which needs to be calculated by comparing with reliable reference data. The accuracy measurement formula is expressed as:

[0170]

[0171] Among them, A is the data accuracy, N is the total number of records, and x i is the true value of the i-th record in the data set, is the corresponding reference value, is the indicator function, when x i and If they are equal, the value is 1, otherwise it is 0;

[0172] S44, Data freshness: Data freshness refers to the timeliness of data, which evaluates whether the data is updated in a timely manner. The freshness measurement formula is expressed as:

[0173]

[0174] Where F is the data freshness, and Δt is the difference between the last updated time of the data and the current time, which can be expressed in hours or days.

[0175] In the quality assessment process of S4, statistical analysis and machine learning models are combined to dynamically adjust the assessment parameters, including:

[0176] S45, Statistical analysis: Statistical analysis methods are used to evaluate the basic conditions of various quality indicators. The data quality is judged by the mean and variance of the data, which is expressed as:

[0177]

[0178] Among them, μ x is the data mean, is the variance;

[0179] S46, machine learning model: In order to improve the accuracy of the evaluation, a machine learning model is used to predict and correct data quality indicators. Through supervised learning methods, the model learns the relationship between data features and quality, and dynamically adjusts the evaluation parameters. The specific steps are as follows:

[0180] (1) Model training:

[0181] Input: data feature vector X (such as missing value rate, field consistency, etc.);

[0182] Output: quality score Y (such as completeness score, consistency score, etc.);

[0183] The regression model formula is expressed as:

[0184]

[0185] Where f is the regression function (such as linear regression, random forest, etc.), θ is the model parameter;

[0186] (2) Dynamically adjust evaluation parameters: According to the prediction results of the model and the changes in real-time data, dynamically adjust various evaluation parameters, such as integrity threshold and consistency tolerance, expressed as:

[0187]

[0188] Among them, θ t is the evaluation parameter at the current time point, L is the loss function, and η is the learning rate, indicating the adjustment speed;

[0189] S47, real-time and accuracy guarantee: Through real-time monitoring and online learning, the evaluation parameters are continuously optimized to adapt to the changes in data sources and ensure the real-time and accuracy of the evaluation results. The online learning formula is expressed as:

[0190]

[0191] Among them, θ t+1 is the evaluation parameter at the next time point, L t is the loss function at the current time point;

[0192] S48, quality score output: Comprehensively consider various quality indicators and calculate the overall quality score Q of the data source, expressed as:

[0193] Q=w1·I+w2·C+w3·A+w4·F;

[0194] Among them, w1, w2, w3, w4 are the weights of various quality indicators, satisfying w1+w2+w3+w4=1;

[0195] S49, quality report generation: Generate a report based on the quality assessment results, point out the advantages and disadvantages of the data source, and make suggestions for improvement;

[0196] By conducting quality assessments on the integrity, consistency, accuracy, and freshness of data sources, combined with statistical analysis and machine learning models, the assessment parameters can be dynamically adjusted to ensure the accuracy and real-time nature of the assessment results. The final quality scores and reports provide an important basis for the management and optimization of data sources.

[0197] S5 specifically includes:

[0198] S51, build a hierarchical model: The first step of the hierarchical analysis method is to build a hierarchical model, dividing the factors affecting the priority of data sources into the target layer, the criterion layer and the scheme layer. The specific steps are as follows:

[0199] (1) Target layer:

[0200] Determine the end goal: prioritize data sources;

[0201] (2) Standard level:

[0202] Data quality (Q): A comprehensive score that includes data completeness, consistency, accuracy, and freshness;

[0203] Data importance (I): reflects the relative importance of the data source to the business or system;

[0204] Data availability (A): includes the difficulty of obtaining data, access speed, and timeliness of data;

[0205] (3) Solution layer: each data source to be sorted;

[0206] S52, constructing a judgment matrix: comparing the relative importance of each pair of criteria, constructing a judgment matrix, wherein the elements of the judgment matrix represent the relative importance of each pair of criteria, and are represented by a scale of 1 to 9;

[0207] The formula of judgment matrix A is expressed as:

[0208] Among them, a ij Indicates the importance of the i-th criterion relative to the j-th criterion;

[0209] S53, calculating the weight vector: by normalizing and weighted summing the judgment matrix, the weight vector of each criterion is calculated, including:

[0210] (1) Normalization: Each column of the judgment matrix A is normalized, expressed as:

[0211]

[0212] (2) Calculate weights: Calculate the weight of each criterion (i.e., the row mean of the normalized matrix), expressed as:

[0213]

[0214] Among them, w i is the weight of the i-th criterion;

[0215] S54, consistency check: In order to ensure the consistency of the judgment matrix, a consistency check is required. The consistency ratio of the judgment matrix must be less than 0.1, including:

[0216] (1) Calculate the consistency index (CI), expressed as:

[0217] Among them, λmax is the maximum eigenvalue of the judgment matrix, n is the order of the matrix;

[0218] (2) Calculate the consistency ratio (CR), expressed as:

[0219] Among them, RI is the random consistency index. If CR<0.1, the consistency test is passed;

[0220] S55, calculating the comprehensive score of the data source: for each data source, according to its score under each criterion and the weight of the criterion, calculate the comprehensive score, which is expressed as:

[0221]

[0222] Among them, S j is the comprehensive score of the jth data source, w i is the weight of the ith criterion, x ij is the score of the j-th data source under the i-th criterion;

[0223] S56, generating a priority list: sorting the data sources according to their comprehensive scores to obtain a priority list of data sources;

[0224] Through the hierarchical analysis method, data quality, data importance and data availability are comprehensively considered to scientifically prioritize data sources. This method combines multi-factor decision analysis with actual business needs to ensure that those data sources that are most important to the business, have the highest quality and are most easily accessible are given priority, thereby optimizing the efficiency of data management and use.

[0225] S6 specifically includes:

[0226] S61, Data Extraction: Obtain full or incremental data from high-priority data sources and extract it to an intermediate storage or processing platform, including:

[0227] (1) Define the extraction strategy:

[0228] Full extraction: Applicable to the initial integration or when the amount of data in the data source is small, extract the entire data set;

[0229] Incremental extraction: Applicable to situations where data sources are updated regularly, and only new or updated data since the last integration is extracted;

[0230] Incremental extraction formula: ΔD=D t -D t-1 ;

[0231] Among them, ΔD is the incremental data set, D t is the data set at time t, D t-1 This is the data set from the last extraction;

[0232] (2) Data extraction tools: Use distributed data extraction tools such as Apache Kafka, Apache Nifi, or Talend to support real-time data stream processing and efficient data transmission;

[0233] (3) Extraction execution: In a distributed environment, start multi-task parallel extraction and use a distributed computing framework (such as Apache Spark) to accelerate the data extraction process;

[0234] Parallel extraction formula:

[0235] Among them, T extraction is the parallel extraction time, T single is the extraction time of a single task, and N is the number of parallel tasks;

[0236] S62, Data Conversion: Clean, format and normalize the extracted data to ensure data consistency and applicability, including:

[0237] (1) Conversion rule definition:

[0238] Format conversion: convert the data source format to the target format (such as CSV to Parquet, JSON to Avro, etc.);

[0239] Data cleaning: processing missing values ​​and outliers, and unifying field naming and coding formats;

[0240] Data normalization: normalize or standardize numerical data to ensure that the data is analyzed under the same dimension;

[0241] The normalized formula is expressed as:

[0242] Among them, x is the original data, and x′ is the normalized data;

[0243] The standardized formula is expressed as:

[0244] Among them, μ x is the data mean, σ x is the data standard deviation;

[0245] (2) Distributed transformation execution:

[0246] Use distributed data processing frameworks (such as Apache Spark and Apache Flink) to batch transform and process data;

[0247] Distributed computing can split the conversion task into multiple subtasks and assign them to different nodes in the cluster for parallel processing;

[0248] The distributed data processing formula is expressed as:

[0249] Among them, T transformation is the total conversion time, T i is the processing time on each node, and N is the number of nodes;

[0250] (3) Conversion result verification:

[0251] Verify whether the converted data meets the predefined rules to ensure the correctness and consistency of data conversion;

[0252] Use data consistency verification technology (such as checksum, hash value) to compare the data before and after conversion.

[0253] S63, data loading: Load the converted data into the target storage system to ensure that it can be used for subsequent analysis and query, including:

[0254] (1) Data loading strategy: including full loading and incremental loading, among which;

[0255] Full load: Applicable to the case of small data volume or first loading, all data is loaded into the target storage;

[0256] Incremental loading: For an existing data set, only the newly added data or updated data is loaded;

[0257] (2) Target storage selection: including cloud storage and distributed databases, among which;

[0258] Cloud storage: Load data into cloud storage services (such as Amazon S3 and Google Cloud Storage) to take advantage of their high availability and elastic expansion.

[0259] Distributed database: Load into a distributed database (such as Apache HBase, Cassandra, MongoDB) to ensure high concurrent reading and writing and high availability of data;

[0260] (3) Distributed data loading: Use a distributed computing framework to split data into small blocks and load them in parallel to the target storage, reducing the bottleneck of single-point loading. The distributed loading formula is expressed as:

[0261]

[0262] Among them, T loading is the loading time, T single_node is the single node loading time, N nodes The number of nodes involved in loading.

[0263] The high-priority data sources in S6 are obtained by calculating the comprehensive scores of the data sources in S55, and the data sources with scores in the top 20% are considered high priority.

[0264] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0265] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for data discovery and identification, characterized in that: The following steps are involved: S1, data source identification and scanning: Scan and identify data sources in the network environment through a hybrid algorithm based on machine learning and deep learning; S2, automatic extraction of data source characteristics: Extract characteristics of the identified data source, including data format, data type, data structure and data scale. Automatic feature engineering technology is used to automatically extract the characteristics of structured data and unstructured data through natural language processing and image recognition technology, and the characteristics are stored in the feature library. S3, data source classification and labeling: Based on the extracted features, an adaptive classification algorithm is used to classify data sources into structured data, semi-structured data, and unstructured data. During the classification process, fuzzy logic and Bayesian classifiers are combined to improve the accuracy and robustness of classification. At the same time, data sources are automatically labeled and label information is generated to facilitate subsequent processing and management; S4, data source quality assessment: perform quality assessment on the classified data sources. The assessment indicators include data integrity, data consistency, data accuracy, and data freshness. During the quality assessment process, statistical analysis and machine learning models are combined to dynamically adjust the assessment parameters. S5, data source prioritization: prioritize data sources based on quality assessment results and data usage requirements, and use the analytic hierarchy process based on multi-factor decision analysis to generate a priority list by comprehensively considering data quality, data importance, and data availability; S6, data source integration: Integrate high-priority data sources, including data extraction, data transformation and data loading.

2. A method for data discovery and identification according to claim 1, characterized in that: The S1 specifically includes: S11, data collection and preprocessing: collect data from network traffic, file systems and databases, and perform preprocessing; S12, Feature Extraction and Modeling: Extract features from network traffic using convolutional neural networks and classify file system and database data using support vector machines; S13, hybrid algorithm integration: the outputs of the convolutional neural network and the support vector machine are fused to form the final recognition model; S14, data source scanning and identification: Apply a hybrid algorithm to comprehensively scan the network environment and identify the data source.

3. A method for data discovery and identification according to claim 2, characterized in that: The S2 specifically includes: S21, data format identification: data format identification includes file format identification and database format identification, wherein; File format identification: Use file header information to identify the file format; Database format identification: Identify the database format by querying the database metadata; S22, data type identification: data type identification includes structured data type identification and unstructured data type identification, wherein; Structured data type identification: Identify data types through statistical characteristics of data; Unstructured data type recognition: For text data, natural language processing technology is used to identify the type of text content. For image data, a pre-trained convolutional neural network model is used to identify the category of the image by inputting the image and extracting its feature vector. S23, data structure identification: data structure identification includes table structure identification and nested structure identification, wherein; Table structure identification: By querying the metadata of the database, the structural information of the table is extracted, including the table name, field name, field type, primary key and foreign key. This information is used to describe the structure and relationship of the table; Nested structure recognition: For JSON and XML nested data structures, use a recursive algorithm to parse the nesting level and node information to identify the nested structure and hierarchical relationship of the data; S24, data scale estimation: data scale estimation includes data record number estimation and file size estimation, wherein; Estimation of the number of data records: For CSV files, estimate the number of records by reading the number of file lines; for database tables, obtain the number of records by executing SQL query statements; File size estimation: Get the file size information through the file attribute interface provided by the operating system; S25, Automatic feature engineering technology: Automatic feature engineering technology includes natural language processing feature extraction and image recognition feature extraction, among which; Natural language processing feature extraction: Use the bag-of-words model and word vector model to extract features of text data; Image recognition feature extraction: Use convolutional neural networks to extract features from image data; S26, storing the features in the feature library: selecting a relational database MySQL to store the extracted features, and storing the extracted features in the feature library in the form of key-value pairs.

4. A method for data discovery and identification according to claim 3, characterized in that: The S3 specifically includes: S31, feature standardization: Before classification, the extracted features are standardized so that different features can be compared under the same dimension; S32, adaptive classification algorithm: The adaptive classification algorithm dynamically adjusts the classification rules according to the characteristics of the data source; S33, fuzzy logic processing: converting the characteristic value into a fuzzy set and using the membership function to express the degree to which the characteristic belongs to a certain category; S34, Bayesian classifier: The robustness of classification is improved by combining the Bayesian classifier, which determines the category of the data source by calculating the posterior probability; S35, automatic marking and label generation: After the classification is completed, the data source is automatically marked to generate label information.

5. A method for data discovery and identification according to claim 4, characterized in that: In S4, the quality of the classified data sources is evaluated, and the evaluation indicators include data integrity, data consistency, data accuracy and data freshness, specifically including: S41, Data integrity: Data integrity refers to whether there are missing or null values ​​in the data set, and whether all the data are collected and saved; S42, Data consistency: Data consistency refers to whether data is consistent across different sources or at different points in time; S43, Data accuracy: Data accuracy refers to the degree of consistency between data and the actual situation, which needs to be calculated by comparing with reliable reference data; S44, Data freshness: Data freshness refers to the timeliness of data, evaluating whether the data is updated in a timely manner.

6. A method for data discovery and identification according to claim 5, characterized in that: In the quality assessment process of S4, statistical analysis and machine learning models are combined to dynamically adjust the assessment parameters, including: S45, Statistical analysis: Statistical analysis methods are used to evaluate the basic conditions of various quality indicators and judge the data quality through the mean and variance of the data; S46, Machine Learning Model: Use machine learning models to predict and correct data quality indicators. Through supervised learning methods, the model learns the relationship between data characteristics and quality, and dynamically adjusts evaluation parameters; S47, real-time and accuracy assurance: through real-time monitoring and online learning, the evaluation parameters are continuously optimized to adapt to changes in data sources; S48, quality score output: Comprehensively calculate the overall quality score of the data source by combining various quality indicators; S49, Quality report generation: Generate a report based on the quality assessment results, point out the advantages and disadvantages of the data source, and make suggestions for improvement.

7. A method for data discovery and identification according to claim 6, characterized in that: The S5 specifically includes: S51, construct a hierarchical model: construct a hierarchical model to divide the factors affecting the priority of data sources into the target layer, the criterion layer and the scheme layer; S52, constructing a judgment matrix: comparing the relative importance of each pair of criteria and constructing a judgment matrix; S53, calculating a weight vector: calculating a weight vector of each criterion by normalizing and weighted summing the judgment matrix; S54, consistency check: perform consistency check, and the consistency ratio of the judgment matrix must be less than 0.1; S55, calculating the comprehensive score of the data source: for each data source, calculating the comprehensive score according to its score under each criterion and the weight of the criterion; S56, generating a priority list: sorting the data sources according to their comprehensive scores to obtain a priority list of data sources.

8. A method for data discovery and identification according to claim 7, characterized in that: The S6 specifically includes: S61, data extraction: obtain full or incremental data from high-priority data sources and extract it to intermediate storage or processing platforms; S62, data conversion: clean, format convert and normalize the extracted data to ensure data consistency and applicability; S63, data loading: Load the converted data into the target storage system to ensure that it is available for subsequent analysis and query.

9. A method for data discovery and identification according to claim 8, characterized in that: The high-priority data sources in S6 are obtained by calculating the comprehensive scores of the data sources in S55, and data sources with scores in the top 20% are regarded as high priority.

Images