Data processing method for data management platform

Abstract

The invention discloses a data processing method for a data management platform, and relates to the technical field of big data storage and management, and the method comprises the steps: collecting original data from a plurality of industrial data sources, carrying out format standardization conversion and preliminary cleaning on the original data, and generating a standardized data set, performing feature extraction on each piece of data in the standardized data set, classifying the data according to the extracted features, determining a data category label and an access frequency level of each piece of data, performing quality evaluation on the classified data, performing data cleaning and repairing operation according to an evaluation result, and generating a high-quality data set. According to the data category labels and the access frequency levels, the high-quality data sets are allocated to the corresponding storage levels to be stored, and multi-level indexes are constructed for the stored data, efficient collection, intelligent classification, quality optimization and hierarchical storage of industrial data are achieved, and the big data management efficiency and the storage resource utilization rate are improved.

Description

Technical Field

[0001] This invention relates to the field of big data storage and management technology, specifically a data processing method for a data management platform. Background Technology

[0002] With the rapid development of the Industrial Internet and intelligent manufacturing, the amount of industrial data of enterprises has shown an explosive growth trend. Industrial big data covers all kinds of data generated in all stages of the entire product life cycle, from customer demand to sales, orders, planning, R&D, design, manufacturing, procurement, supply, inventory, delivery, after-sales service, operation and maintenance. This data is not only huge in scale, but also diverse in type, including structured data, semi-structured data and unstructured data.

[0003] Existing industrial data management technologies have the following shortcomings:

[0004] First, traditional data storage systems have performance bottlenecks when dealing with massive amounts of industrial data. The traditional centralized storage management model relies heavily on the performance of a single machine, which limits the scalability of storage capacity and cannot effectively support the low-latency access and high-concurrency access requirements of massive amounts of unstructured data.

[0005] Second, the existing data classification and management mechanism is not refined enough. Data generated in industrial scenarios has different access frequencies and importance levels. Existing systems cannot perform differentiated storage and management according to the characteristics of the data, resulting in unreasonable storage resource allocation and high storage costs.

[0006] Third, there is a lack of data quality management capabilities. Industrial data is prone to generating redundant, invalid, and abnormal data during the collection and transmission process. Existing systems lack effective data cleaning and quality assessment mechanisms, which affects the accuracy of subsequent data analysis and mining.

[0007] Fourth, the ability to integrate and process multi-source heterogeneous data is limited. Industrial enterprises typically have multiple production lines and multiple information systems. Data from different sources have different formats and structures, and existing systems struggle to achieve efficient integration and unified management of multi-source heterogeneous data.

[0008] Therefore, there is an urgent need for a data management platform that can efficiently process massive amounts of industrial data, achieve intelligent classification and storage, and ensure data quality. Summary of the Invention

[0009] To address the problems existing in the prior art, this invention provides a data processing method for a data management platform, which can achieve efficient collection, intelligent classification, quality optimization, and secure storage of industrial data.

[0010] The technical solution adopted in this invention is as follows:

[0011] A data processing method for a data management platform includes the following steps:

[0012] Data Acquisition and Preprocessing: Raw data is collected from multiple industrial data sources, and the raw data is standardized and preliminarily cleaned to generate a standardized dataset;

[0013] Data feature extraction and classification: Features are extracted for each data point in the standardized dataset, and the data is classified according to the extracted features to determine the data category label and access frequency level for each data point;

[0014] Data quality assessment and optimization: The quality of the classified data is assessed, and data cleaning and repair operations are performed based on the assessment results to generate a high-quality dataset;

[0015] Tiered storage and index building: Based on data category labels and access frequency levels, high-quality datasets are allocated to corresponding storage tiers for storage, and multi-level indexes are built for the stored data.

[0016] Furthermore, the method for collecting raw data from multiple industrial data sources in the data acquisition and preprocessing process includes:

[0017] Establish a data connection channel with industrial data sources, including sensor acquisition devices, industrial control systems, and business management systems;

[0018] Raw data is acquired from each data connection channel according to the preset acquisition cycle, and timestamps and data source identifiers are added to the acquired raw data.

[0019] The original data with added tags is stored in the data cache area, where a format standardization conversion is performed to convert data of different formats into the platform's standard data format.

[0020] Furthermore, in the data feature extraction and classification, the methods for classifying the data include:

[0021] Extract the data feature vector for each data point, which includes data type features, data size features, timeliness features, and correlation features;

[0022] Based on data type characteristics, data is divided into three basic categories: structured data, semi-structured data, and unstructured data.

[0023] The predicted access frequency of the data is calculated based on the timeliness characteristics. The predicted access frequency is then compared with a preset frequency threshold to determine the access frequency level of the data. The access frequency level includes three levels: high-frequency access, medium-frequency access, and low-frequency access.

[0024] Furthermore, in the data feature extraction and classification, the method for calculating the predicted access frequency of the data includes:

[0025] Obtain the historical access count and historical access time interval sequence of the data within the historical time window;

[0026] Calculate the predicted access frequency based on the historical access count and historical access time interval sequence;

[0027] The formula for calculating the predicted access frequency is:

[0028] ;

[0029] in, This represents the predicted access frequency. This indicates the number of historical visits within a historical time window. Indicates the duration of a historical time window, in days. This represents the average value of a series of historical access time intervals, in hours. This is the first weighting coefficient, with a value of 0.6. This is the second weighting coefficient, with a value of 0.4.

[0030] Furthermore, in the data quality assessment and optimization, the methods for assessing the quality of the classified data include:

[0031] Perform an integrity check on each piece of data to determine if there are any missing values ​​in the required fields, and record the integrity check results;

[0032] Perform a consistency check on each piece of data to determine whether the data format and value range conform to the preset data specifications, and record the consistency check results;

[0033] Perform accuracy checks on each data point, determine the accuracy of the data values ​​by cross-validating with related data, and record the accuracy check results;

[0034] Based on the integrity test results, consistency test results, and accuracy test results, calculate the overall quality score for each data point, and mark data with an overall quality score lower than the preset quality threshold as data to be processed.

[0035] Furthermore, in the data quality assessment and optimization, the method for calculating the overall quality score of each data point includes:

[0036] Integrity score is determined based on integrity test results. Integrity score is 1 when integrity test passes and 0 when integrity test fails.

[0037] A consistency score is determined based on the consistency test results. A consistency score of 1 is given when the consistency test passes and 0 is given when the consistency test fails.

[0038] An accuracy score is determined based on the accuracy test results. An accuracy score of 1 is given when the accuracy test is passed, and an accuracy score of 0 is given when the accuracy test is failed.

[0039] The formula for calculating the overall quality score is as follows:

[0040] ;

[0041] in, Indicates the overall quality score. Indicates the completeness score. Indicates the consistency score. Indicates the accuracy score. The integrity weight is set to 0.4. This is the consistency weight, with a value of 0.35. The accuracy weight is set to 0.25.

[0042] Furthermore, in the hierarchical storage and index construction, the method for allocating high-quality datasets to corresponding storage levels for storage includes:

[0043] A three-tier storage architecture is established. The first storage tier is the memory storage tier, which is used to store high-frequency access data. The second storage tier is the solid-state storage tier, which is used to store mid-frequency access data. The third storage tier is the disk storage tier, which is used to store low-frequency access data.

[0044] Based on the access frequency level of each piece of data, the data is allocated to the corresponding storage level;

[0045] Establish a data migration mechanism between different storage tiers. When the access frequency of data changes, the data will be migrated to the storage tier corresponding to the new access frequency level.

[0046] Furthermore, in the hierarchical storage and index construction, the method for constructing multi-level indexes for the stored data includes:

[0047] Construct a primary index, which is a categorized index based on data category labels, to support data retrieval by data category;

[0048] Construct a secondary index, which is a time-series index based on timestamps, to support data retrieval by time range;

[0049] A three-level index is constructed, which is a full-text index based on data content keywords, to support data retrieval by keywords;

[0050] The first-level index, second-level index, and third-level index are linked together to form a multi-level composite index structure.

[0051] The beneficial effects of this invention are as follows:

[0052] 1. This invention establishes data connection channels with multiple industrial data sources, thereby achieving unified collection and standardized processing of industrial data and solving the problem of fusion processing of multi-source heterogeneous data.

[0053] 2. This invention achieves intelligent classification and management of data by extracting and classifying data features and determining access frequency levels based on the predicted access frequency values ​​of the data, thereby improving the utilization efficiency of storage resources.

[0054] 3. This invention comprehensively evaluates data quality through three dimensions: integrity detection, consistency detection, and accuracy detection, and cleans and repairs low-quality data, effectively ensuring data quality.

[0055] 4. This invention achieves hierarchical storage management of data by establishing a three-level storage architecture and a data migration mechanism, storing frequently accessed data on high-speed storage media, reducing data access latency, and optimizing storage costs.

[0056] 5. This invention supports multi-dimensional data retrieval by constructing a multi-level composite index structure, thereby improving the query efficiency of large-scale data. Attached Figure Description

[0057] Figure 1 This is an overall flowchart of a data processing method for a data management platform according to the present invention;

[0058] Figure 2 This is a flowchart of the data acquisition and preprocessing process of this invention;

[0059] Figure 3 This is a flowchart of the data feature extraction and classification process of this invention;

[0060] Figure 4 This is a flowchart of the data quality assessment and optimization process of this invention;

[0061] Figure 5 This is a schematic diagram of the hierarchical storage architecture of the present invention;

[0062] Figure 6 This is a schematic diagram of the multi-level index structure of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0064] Example 1

[0065] This embodiment provides a data processing method for a data management platform, applied to an industrial big data management platform, to achieve efficient processing and intelligent management of industrial data, such as... Figure 1 As shown, the method includes the following steps:

[0066] Step-by-step data acquisition and preprocessing

[0067] like Figure 2 As shown, the specific implementation method of this step is as follows:

[0068] First, establish a data connection channel with the industrial data source. In this embodiment, the industrial data source includes three types: sensor acquisition equipment, industrial control system and business management system. The sensor acquisition equipment is used to collect temperature, pressure, humidity, vibration, current and voltage parameters during the production process.

[0069] Industrial control systems include PLC control systems and SCADA systems, which are used to collect production control data;

[0070] Business management systems include MES, ERP, and CRM systems, used to collect business process data.

[0071] Then, raw data is acquired from each data connection channel according to a preset acquisition cycle. In this embodiment, the acquisition cycle is set to 100 milliseconds for the sensor acquisition device, 1 second for the industrial control system, and 1 minute for the business management system. After acquiring the raw data, a timestamp and a data source identifier are added to each data item. The timestamp uses a uniform time format, and the data source identifier is used to indicate the source system of the data.

[0072] Finally, the tagged original data is stored in a data cache, where format standardization conversion is performed. This embodiment uses a unified platform standard data format to convert data from different sources into this standard format, including field name standardization, data type standardization, and encoding format standardization.

[0073] Step-by-step data feature extraction and classification

[0074] like Figure 3 As shown, the specific implementation method of this step is as follows:

[0075] First, extract the data feature vector for each data point. The data feature vector includes four dimensions of features: data type features, which describe the structural form of the data;

[0076] Data size characteristics are used to describe the storage space required for data;

[0077] Timeliness is a feature used to describe the time sensitivity of data;

[0078] Association features are used to describe the relationships between data and other data.

[0079] Then, based on data type characteristics, the data is divided into three basic categories: structured data includes database table data and standardized business records;

[0080] Semi-structured data includes log files and configuration files;

[0081] Unstructured data includes image files, video files, and document files.

[0082] Next, based on the timeliness characteristics, the predicted access frequency of the data is calculated. This involves obtaining the historical access count N and historical access time interval series over the past 30 days, calculating the average value of the historical access time interval series, and then using the access frequency prediction formula to calculate the predicted access frequency. :

[0083] ;

[0084] in, This represents the predicted access frequency. This indicates the number of historical visits within a historical time window. In this embodiment, the duration of the historical time window is indicated. The value is 30 days. This represents the average value of a series of historical access time intervals, in hours. This is the first weighting coefficient, with a value of 0.6. This is the second weighting coefficient, with a value of 0.4.

[0085] Finally, the predicted access frequency value is compared with the preset frequency threshold to determine the access frequency level of the data. In this embodiment, two frequency thresholds are set: the first frequency threshold is 10 and the second frequency threshold is 2. When the predicted access frequency value is greater than the first frequency threshold, the access frequency level is determined to be high-frequency access.

[0086] When the predicted access frequency value is greater than the second frequency threshold and less than the first frequency threshold, the access frequency level is determined to be medium frequency access.

[0087] When the predicted access frequency is less than the second frequency threshold, the access frequency level is determined to be low-frequency access.

[0088] Step-by-step data quality assessment and optimization

[0089] like Figure 4 As shown, the specific implementation method of this step is as follows:

[0090] First, perform an integrity check on each piece of data. Based on the data specifications of the data category, determine the list of required fields for that category of data. Check each required field for missing values. If all required fields have valid values, the integrity check passes and the integrity score is 1.

[0091] If any required field is missing, the integrity check fails and the integrity score is 0.

[0092] Then, a consistency check is performed on each piece of data. According to the field format requirements and value range requirements defined in the data specification, the value of each field of the data is checked to see if it conforms to the specification. If the format and value range of all fields conform to the specification, the consistency check passes and the consistency score is 1.

[0093] Otherwise, the consistency check fails, and the consistency score is 0.

[0094] Next, the accuracy of each data point is checked. The accuracy of the data value is determined by cross-validation with related data. The cross-validation method used in this embodiment includes: comparing and verifying with data from adjacent data sources within the same time period, and comparing and verifying with the statistical patterns of similar historical data. If the cross-validation passes, the accuracy score is 1.

[0095] Otherwise, the accuracy score is 0.

[0096] Finally, the overall quality score is calculated based on the results of the three tests. :

[0097] ;

[0098] in, Indicates the overall quality score. Indicates the completeness score. Indicates the consistency score. Indicates the accuracy score. The integrity weight is set to 0.4. This is the consistency weight, with a value of 0.35. For accuracy weighting, a value of 0.25 is used.

[0099] In this embodiment, the preset quality threshold is set to 0.7. When the overall quality score is lower than 0.7, the data is marked as data to be processed, and data cleaning and repair operations are performed on the data to be processed, including missing value filling, format correction, and outlier repair.

[0100] Hierarchical storage and index building

[0101] like Figure 5 As shown, this embodiment establishes a three-tier storage architecture:

[0102] The first storage tier is the memory storage tier, which uses high-speed memory as the storage medium to store frequently accessed data, providing microsecond-level data access response times.

[0103] The second storage tier is the solid-state storage tier, which uses solid-state drives (SSDs) as the storage medium to store data with intermediate-frequency access, providing millisecond-level data access response times.

[0104] The third storage tier is the disk storage tier, which uses hard disk drives (HDDs) as the storage medium. It stores low-frequency access data, providing second-level data access response times, and has the advantages of low cost and large capacity.

[0105] Based on the access frequency level of each data item, the data is allocated to the corresponding storage level for storage. Simultaneously, a data migration mechanism is established: the system periodically recalculates the predicted access frequency value for each data item. When the access frequency level of data changes, the data is automatically migrated to the storage level corresponding to the new access frequency level. In this embodiment, the migration detection cycle is set to 24 hours.

[0106] like Figure 6 As shown, this embodiment constructs a multi-level index structure:

[0107] The primary index is a categorical index based on data category labels, using a hash index structure, and supports fast retrieval based on three categories: structured data, semi-structured data, and unstructured data.

[0108] The secondary index is a timestamp-based time-series index using a B+ tree structure. It supports data retrieval by time range and is suitable for range queries of time-series data.

[0109] The third-level index is a full-text index based on data content keywords. It adopts an inverted index structure, supports data retrieval by keywords, and is suitable for content retrieval scenarios of unstructured data.

[0110] By linking first-level, second-level, and third-level indexes to form a multi-level composite index structure, data retrieval can be performed by selecting a single index based on query conditions, or by using index join queries to achieve multi-condition combined retrieval.

[0111] Example 2

[0112] This embodiment provides an application of a data management platform's data processing method in an industrial cloud storage scenario.

[0113] In industrial cloud storage scenarios, the data management platform is deployed on a cloud server, and industrial data sources are distributed across multiple factory workshops. This embodiment adds the following processing steps based on Embodiment 1:

[0114] In data acquisition and preprocessing, industrial IoT gateways are used to enable data transmission between the edge and the cloud. These gateways are deployed in various factory workshops and are responsible for collecting data from local sensors and industrial control systems. After compressing and encoding the data, it is transmitted to the cloud data management platform via the network.

[0115] In tiered storage and index construction, a distributed storage architecture is adopted to realize cloud data storage. Data is sharded and stored on multiple storage nodes, with each data shard having three copies distributed across different storage nodes, ensuring data reliability and availability.

[0116] This embodiment can meet the centralized data management needs of industrial enterprises with multiple factories and workshops, and realize unified cloud storage and management of industrial data.

[0117] In summary, the data processing method for a data management platform provided by this invention achieves efficient management of industrial big data through four stages: data acquisition and preprocessing, data feature extraction and classification, data quality assessment and optimization, and hierarchical storage and index construction. This invention effectively solves the problems of insufficient storage scalability, imprecise data classification management, weak data quality assurance capabilities, and difficulties in integrating multi-source heterogeneous data in the prior art, and has significant technological progress and practical value.

Claims

1. A data processing method for a data management platform, characterized in that: Includes the following steps: Data Acquisition and Preprocessing: Raw data is collected from multiple industrial data sources, and the raw data is standardized and preliminarily cleaned to generate a standardized dataset; Data feature extraction and classification: Features are extracted for each data point in the standardized dataset, and the data is classified according to the extracted features to determine the data category label and access frequency level for each data point; Data quality assessment and optimization: The quality of the classified data is assessed, and data cleaning and repair operations are performed based on the assessment results to generate a high-quality dataset; Tiered storage and index building: Based on data category labels and access frequency levels, high-quality datasets are allocated to corresponding storage tiers for storage, and multi-level indexes are built for the stored data.

2. The data processing method for a data management platform according to claim 1, characterized in that: The methods for collecting raw data from multiple industrial data sources in the data acquisition and preprocessing process include: Establish a data connection channel with industrial data sources, including sensor acquisition devices, industrial control systems, and business management systems; Raw data is acquired from each data connection channel according to the preset acquisition cycle, and timestamps and data source identifiers are added to the acquired raw data. The original data with added tags is stored in the data cache area, where a format standardization conversion is performed to convert data of different formats into the platform's standard data format.

3. The data processing method for a data management platform according to claim 1, characterized in that: In the data feature extraction and classification, the methods for classifying data include: Extract the data feature vector for each data point, which includes data type features, data size features, timeliness features, and correlation features; Based on data type characteristics, data is divided into three basic categories: structured data, semi-structured data, and unstructured data. The predicted access frequency of the data is calculated based on the timeliness characteristics. The predicted access frequency is then compared with a preset frequency threshold to determine the access frequency level of the data. The access frequency level includes three levels: high-frequency access, medium-frequency access, and low-frequency access.

4. The data processing method for a data management platform according to claim 3, characterized in that: In the data feature extraction and classification, the method for calculating the predicted access frequency of data includes: Obtain the historical access count and historical access time interval sequence of the data within the historical time window; Based on the historical number of visits and the historical visit time interval sequence, the predicted visit frequency is calculated using a formula.

5. The data processing method for a data management platform according to claim 1, characterized in that: In the data quality assessment and optimization, the methods for assessing the quality of the classified data include: Perform an integrity check on each piece of data to determine if there are any missing values ​​in the required fields, and record the integrity check results; Perform a consistency check on each piece of data to determine whether the data format and value range conform to the preset data specifications, and record the consistency check results; Perform accuracy checks on each data point, determine the accuracy of the data values ​​by cross-validating with related data, and record the accuracy check results; Based on the integrity test results, consistency test results, and accuracy test results, calculate the overall quality score for each data point, and mark data with an overall quality score lower than the preset quality threshold as data to be processed.

6. The data processing method for a data management platform according to claim 5, characterized in that: In the data quality assessment and optimization, the method for calculating the overall quality score of each data point includes: Integrity score is determined based on integrity test results. Integrity score is 1 when integrity test passes and 0 when integrity test fails. A consistency score is determined based on the consistency test results. A consistency score of 1 is given when the consistency test passes and 0 is given when the consistency test fails. An accuracy score is determined based on the accuracy test results. An accuracy score of 1 is given when the accuracy test is passed, and an accuracy score of 0 is given when the accuracy test is failed. The overall quality score is calculated using a formula.

7. The data processing method for a data management platform according to claim 1, characterized in that: In the hierarchical storage and index construction, the method for allocating high-quality datasets to corresponding storage levels includes: A three-tier storage architecture is established. The first storage tier is the memory storage tier, which is used to store high-frequency access data. The second storage tier is the solid-state storage tier, which is used to store mid-frequency access data. The third storage tier is the disk storage tier, which is used to store low-frequency access data. Based on the access frequency level of each piece of data, the data is allocated to the corresponding storage level; Establish a data migration mechanism between different storage tiers. When the access frequency of data changes, the data will be migrated to the storage tier corresponding to the new access frequency level.

8. The data processing method for a data management platform according to claim 1, characterized in that: In the hierarchical storage and index construction, the method for constructing multi-level indexes for the stored data includes: Construct a primary index, which is a categorized index based on data category labels, to support data retrieval by data category; Construct a secondary index, which is a time-series index based on timestamps, to support data retrieval by time range; A three-level index is constructed, which is a full-text index based on data content keywords, to support data retrieval by keywords; The first-level index, second-level index, and third-level index are linked together to form a multi-level composite index structure.

Images