Data acquisition method and system based on industrial internet operating system

Abstract

The invention relates to the technical field of industrial internet data acquisition, in particular to a data acquisition method and system based on an industrial internet operating system, and the method comprises the following steps: S1, detecting and recognizing data sources of a plurality of industrial devices and sensors; s2, collecting original data from a data source in real time, and preprocessing the original data; s3, distributing part of calculation tasks to a plurality of edge nodes; s4, storing the data in a distributed database or cloud storage in a classified manner; s5, performing fragmentation processing on the data in the cache; s6, performing data synchronization and consistency verification; and S7, converting the data into a corresponding format. According to the invention, by implementing efficient data fragmentation processing, a dynamic storage strategy and secure data consistency verification, the processing speed and storage efficiency of the industrial internet data acquisition system and the security and integrity of data are remarkably improved, and the requirements of high efficiency and high reliability of modern industrial production are effectively supported.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial Internet data acquisition, and in particular to a data acquisition method and system based on an industrial Internet operating system. Background Art

[0002] With the advancement of the Industrial 4.0 revolution, the application of industrial Internet is becoming more and more extensive, becoming a key technology to promote the intelligent and information-based development of manufacturing industry. The industrial Internet operating system integrates various industrial equipment and sensors to realize real-time data collection, processing and analysis, support decision-making and operation optimization. However, with the increase in the number of devices and data types, the existing data collection system faces problems such as slow processing speed, complex data management, and difficulty in ensuring security and reliability. These problems limit the application of industrial Internet technology in efficient data processing and real-time response, affecting production efficiency and system stability.

[0003] The current industrial Internet data acquisition systems generally have problems with low data processing and storage efficiency, especially in terms of data security and consistency assurance. For example, data is easily tampered with and lost during transmission and storage, and there is a lack of effective data synchronization and consistency verification mechanisms, which makes it difficult to ensure data quality and reliability. In addition, with the rapid increase in data volume, traditional data acquisition systems often respond slowly when processing large-scale data and cannot meet the high real-time requirements in industrial production.

[0004] Therefore, there is an urgent need to develop a new data collection method and system to solve the problems of slow data processing speed and difficulty in ensuring data security and consistency. Summary of the invention

[0005] Based on the above objectives, the present invention provides a data collection method and system based on an industrial Internet operating system.

[0006] The data collection method based on the industrial Internet operating system includes the following steps:

[0007] S1: Based on the Industrial Internet operating system, it uses a preset intelligent recognition algorithm to automatically detect and identify the data sources of multiple industrial devices and sensors, including device type, data format, collection frequency and communication protocol;

[0008] S2: Collect raw data from the data source in real time and pre-process the raw data, including denoising, filtering, repairing and standardization;

[0009] S3: Perform edge computing on the industrial equipment side, conduct preliminary analysis and processing on the preprocessed data, distribute some computing tasks to multiple edge nodes, and reduce the burden on the central processing unit;

[0010] S4: Temporarily store the data processed by edge computing in the cache module of the industrial Internet operating system, and select appropriate storage strategies based on the importance and real-time needs of the data to classify and store the data in a distributed database or cloud storage;

[0011] S5: According to the preset time window and data type, the data in the cache is fragmented and a label is added to each data fragment;

[0012] S6: Through the multi-channel communication module of the industrial Internet operating system, the fragmented data is transmitted to the central data processing center for data synchronization and consistency verification;

[0013] S7: In the central data processing center, the received data is parsed and converted into the corresponding format according to different application requirements, and a standardized data interface is opened for third-party applications and analysis tools to call.

[0014] Optionally, the S1 specifically includes:

[0015] S11: Through the device management module in the industrial Internet operating system, an initial list of devices and sensors is established, which contains basic information of known devices and sensors, including device type, data format, collection frequency and communication protocol;

[0016] S12: Scan the equipment and sensors at the industrial site through the equipment management module to collect real-time information of the equipment and sensors, including equipment ID, model, location and current status;

[0017] S13: Match the collected real-time information with the initial list, use the support vector machine algorithm to perform pattern recognition, identify newly added devices and sensors, and update the initial list;

[0018] S14: For the matched and identified devices and sensors, the convolutional neural network algorithm is used to analyze their data formats and communication protocols, and the preset protocol parsing library is used to automatically configure the adaptive communication protocols and data formats;

[0019] S15: Dynamically adjust the data collection frequency according to the device type and data characteristics to maximize the accuracy and timeliness of data collection without affecting the normal operation of the device; assuming that the collection frequency is f, the adjustment formula is: Where, f is the acquisition frequency, f max is the maximum acquisition frequency, C is the preset constant, and V is the data change rate;

[0020] S16: Generate a complete configuration file of the device and sensor, including device type, data format, acquisition frequency, and communication protocol.

[0021] Optionally, the S2 specifically includes:

[0022] S21: Collect raw data from data sources in real time, using the configured data format, collection frequency, and communication protocol to obtain raw data generated by industrial equipment and sensors;

[0023] S22: performing denoising processing on the raw data collected in real time, using a denoising algorithm based on wavelet transform to remove random noise signals in the raw data;

[0024] S23: Using a Kalman filter to filter the denoised data, filter out high-frequency interference components in the data, and retain the effective signal part of the data;

[0025] S24: repairing the filtered data, and estimating and completing the missing values ​​in the data using the Lagrange interpolation method;

[0026] S25: Standardize the repaired data, specifically use the Z-score standardization algorithm to convert the data to the same scale range and eliminate the scale differences between different data sources.

[0027] Optionally, the S3 specifically includes:

[0028] S31: Deploy edge computing nodes on industrial equipment, including embedded computing devices or industrial controllers, each of which is configured with computing resources and storage capacity to perform edge computing tasks;

[0029] S32: The pre-processed data is transmitted to the edge computing node through the data transmission module in the industrial Internet operating system, and preliminary analysis and processing are performed on the edge computing node, including data aggregation, data filtering, and data feature extraction;

[0030] S33: using the resource scheduling algorithm of the edge computing node, according to the complexity of the computing task and the load of each edge node, determine the distribution of the computing task, allocate the tasks with high computing load to the edge nodes with high computing power, and allocate the tasks with low computing load to other edge nodes, so as to achieve load balancing;

[0031] S34: Execute the assigned computing tasks on the edge computing nodes, including data processing, model reasoning, and real-time response;

[0032] S35: The result data processed by the edge computing nodes are aggregated to the central data processing center through the industrial Internet operating system, and the result data are summarized.

[0033] Optionally, the S33 specifically includes:

[0034] S331: Divide the task into multiple subtasks according to the complexity of the computing task; the complexity C is measured by the computing time T and resource consumption R required by the task, and the formula is: C = T × R, where T is the computing time and R is the resource consumption;

[0035] S332: Obtain real-time load information of each edge node, including CPU usage U, memory occupancy M, and network bandwidth B; the load L calculation formula is: L = αU + βM + γB, where α, β, and γ are weight coefficients, reflecting the influence weight of each resource in the total load;

[0036] S333: Calculate the available resource amount A of each edge node, the formula is: A = R max -L, where R max is the maximum resource capacity of the node, and L is the current load of the node;

[0037] S334: According to the task complexity C and the available resource amount A of the edge node, the distribution of the computing task is determined by using a resource scheduling algorithm; the resource scheduling algorithm adopts a minimum resource utilization priority strategy based on load balancing, and the specific formula is: Among them, N is the selected edge node, i is the index of all candidate edge nodes, A i is the available resources of the ith edge node, It means taking the i value that minimizes the score value; and assigning the subtask to the selected edge node N.

[0038] Optionally, the S4 specifically includes:

[0039] S41: Evaluate the importance of data processed by edge computing, and set the data importance level I based on the source, type, purpose and impact of the data on the industrial production process; the data importance level is divided into three categories: high, medium and low, corresponding to importance level I h ,I m and I l ;

[0040] S42: Classify data according to real-time requirements and evaluate the real-time requirements R of the data. The real-time requirements are divided into high real-time, medium real-time and low real-time, corresponding to the real-time level R respectively. h , R m and R l ;

[0041] S43: Determine a data storage strategy based on the importance level I and the real-time requirement R of the data, wherein the data storage strategy includes real-time storage, periodic storage, and archive storage;

[0042] For data with high importance and high real-time h ,Rh , using real-time storage strategies to store data directly in high-performance distributed databases;

[0043] For data with high importance but medium real-time requirements h ,R m , or data of medium importance and high real-time nature m ,R h ,Using a periodic storage strategy, the data is stored in a distributed database according to a preset period;

[0044] For data of medium importance and medium real-time m ,R m , or data with high importance but low real-time nature h ,R l ,adopting a hybrid storage strategy, storing part of the data in real time in a distributed database and the other part periodically in the cloud storage;

[0045] For low-importance but high-time data l ,R h ,Using a low-latency storage strategy, the data is temporarily stored in the cache module;

[0046] For data of medium importance but low real-time nature m ,R l , or data of low importance and medium real-time nature l ,R m ,Adopt an archival storage strategy and store data directly in the cloud storage;

[0047] For data with low importance and low real-time performance, l ,R l ,A delayed storage strategy is adopted to temporarily store the data in the local storage of the edge node, and batch ,storage in the cloud storage when the data volume reaches a certain threshold.

[0048] Optionally, the S5 specifically includes:

[0049] S51: fragmenting the data in the cache according to a preset time window and data type; the time window is implemented by a predetermined fixed time interval or a dynamically adjusted time interval, and the data in each time window is regarded as a fragment;

[0050] S52: Divide the data in each time window to generate multiple data segments, each data segment includes a data set in the time window;

[0051] S53: adding a label to each data segment, including a timestamp, a device ID, a data type, and an abnormality mark;

[0052] S54: Store the labeled data segments in the cache module of the industrial Internet operating system.

[0053] Optionally, the S6 specifically includes:

[0054] S61: Select a transport protocol, including MQTT, HTTP, or CoAP, through the communication module in the Industrial Internet operating system, and transmit the fragmented data from the edge node to the central data processing center;

[0055] S62: During data transmission, the TLS protocol is used for data encryption, and the Gzip algorithm is used for data compression;

[0056] S63: After receiving the fragmented data, the central data processing center performs synchronization processing, specifically by comparing timestamps and performing hash verification to ensure that the timing and content of each data fragment are consistent;

[0057] S64: Perform consistency check on the received data fragments, specifically including the following steps:

[0058] S641: Calculate the hash value of each data fragment and compare it with the hash value transmitted by the edge node;

[0059] S642: Compare the timestamps of the data segments to check whether there is data loss or repeated transmission. If an abnormality is found, retransmit or merge the data.

[0060] S643: Using a consistency check algorithm based on blockchain technology, the hash value of the data fragment is stored in the blockchain network.

[0061] Optionally, the S7 specifically includes:

[0062] S71: After the central data processing center receives the data, the data parsing module is used to parse the data. The parsing process includes data format recognition and data content extraction. The data format recognition is used to match the received data format through the format library. The data content is used to extract the data content and convert it into structured data through the parser.

[0063] S72: According to different application requirements, the structured data is converted into a corresponding format; specifically, the following steps are included:

[0064] S721: Select corresponding data conversion rules according to application requirements; the data conversion rules are pre-defined in a conversion rule library, including mapping relationships between different data formats;

[0065] S722: using the selected conversion rule, convert the structured data into a target format, where the target format includes JSON, XML, and CSV;

[0066] S723: During the conversion process, data unit conversion and format adjustment are performed simultaneously;

[0067] S73: Open a standardized data interface for third-party applications and analysis tools to use, including the following steps:

[0068] S731: Deploy RESTful API services in the central data processing center to provide standardized data access interfaces; RESTful API services define interfaces based on resource paths and HTTP methods, including GET, POST, PUT, and DELETE;

[0069] S732: Define detailed API documentation for each data interface, including the interface's request format, response format, parameter description, and sample code;

[0070] S733: Use API gateway to perform security management and flow control on data interfaces, and use OAuth 2.0 for identity authentication and authorization;

[0071] S74: The parsed and converted data are stored in a database of a central data processing center, and provided to third-party applications and analysis tools in real time through a standardized data interface.

[0072] The data acquisition system based on the industrial Internet operating system is used to implement the above-mentioned data acquisition method based on the industrial Internet operating system, and includes the following modules:

[0073] Device management module: used to establish and maintain the initial list of industrial devices and sensors, including device type, data format, acquisition frequency and communication protocol, and automatically detect and identify newly added devices and sensors;

[0074] Data acquisition module: used to collect raw data from identified devices and sensors in real time, and pre-process the raw data, including denoising, filtering, repairing and standardization;

[0075] Edge computing module: performs preliminary analysis and processing on pre-processed data at the industrial equipment end, distributes computing tasks to multiple edge nodes, and reduces the burden on the central processing unit;

[0076] Cache module: used to temporarily store data processed by edge computing, select appropriate storage strategies based on the importance and real-time needs of the data, and classify and store the data in a distributed database or cloud storage;

[0077] Data fragmentation module: fragments the data in the cache according to the preset time window and data type, and adds tags to each data fragment, including timestamp, device ID, data type and abnormal mark;

[0078] Communication module: Transmits fragmented data from edge nodes to the central data processing center by selecting appropriate transmission protocols and utilizing encryption and compression techniques during transmission;

[0079] Data synchronization module: synchronizes the received data in the central data processing center;

[0080] Data verification module: performs consistency verification on the received data fragments, calculates the hash value of each data fragment, and stores the hash value using blockchain technology;

[0081] Data parsing module: parses the received data, including data format recognition and data content extraction, and converts the data into corresponding formats according to different application requirements;

[0082] Data interface module: Open standardized data interfaces for third-party applications and analysis tools to call, provide RESTful API services, define detailed API documents, and perform security management and traffic control through API gateways.

[0083] Beneficial effects of the present invention:

[0084] The present invention significantly improves the processing speed and storage efficiency of the data acquisition system by introducing advanced data fragmentation processing and dynamic storage strategies. By real-time data fragmentation and dynamically selecting storage locations based on the importance of the data and real-time needs, the present invention can ensure that while maintaining high data processing efficiency, large amounts of data can be effectively managed, reducing data processing delays and resource consumption. In addition, by accurately timestamping and marking anomalies for each data fragment, the present invention enhances data traceability and analysis accuracy, thereby improving the operational efficiency and reliability of the entire system.

[0085] The present invention greatly enhances the security and integrity of data by realizing consistency verification and secure transmission of data, and adopts blockchain technology to perform consistency verification of data, thereby ensuring the non-tamperability and traceability of data during storage and transmission. At the same time, by utilizing standardized data interfaces and security protocols, the present invention supports efficient access and secure calling of third-party applications and analysis tools, and provides a secure and open data environment for industrial systems. These features give the present invention significant advantages in improving data processing and security management of industrial Internet operating systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0087] Figure 1 A schematic diagram of a data collection method for an industrial Internet operating system according to an embodiment of the present invention;

[0088] Figure 2 Schematic diagram of the data acquisition system of the industrial Internet operating system of an embodiment of the present invention. DETAILED DESCRIPTION

[0089] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0090] It should be noted that the references to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0091] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0092] like Figure 1 As shown, the data collection method based on the industrial Internet operating system includes the following steps:

[0093] S1: Based on the Industrial Internet operating system, it uses a preset intelligent recognition algorithm to automatically detect and identify the data sources of multiple industrial devices and sensors, including device type, data format, collection frequency and communication protocol;

[0094] S2: Collect raw data from data sources in real time and pre-process the raw data, including denoising, filtering, repairing, and standardization, to improve data quality and consistency;

[0095] S3: Perform edge computing on the industrial equipment side, conduct preliminary analysis and processing on the preprocessed data, distribute some computing tasks to multiple edge nodes, reduce the burden on the central processing unit, and improve the real-time and efficiency of data processing;

[0096] S4: Temporarily store the data processed by edge computing in the cache module of the industrial Internet operating system, and select appropriate storage strategies based on the importance and real-time needs of the data to classify and store the data in a distributed database or cloud storage;

[0097] S5: According to the preset time window and data type, the data in the cache is fragmented and each data fragment is labeled, including timestamp, device ID, data type and abnormal mark;

[0098] S6: Through the multi-channel communication module of the industrial Internet operating system, the fragmented data is transmitted to the central data processing center for data synchronization and consistency verification to ensure data consistency and real-time update between various data nodes;

[0099] S7: In the central data processing center, the received data is parsed and converted into the corresponding format according to different application requirements. The standardized data interface is opened for third-party applications and analysis tools to call, so as to achieve efficient use of data and maximize its value.

[0100] S1 specifically includes:

[0101] S11: Through the device management module in the industrial Internet operating system, an initial list of devices and sensors is established, which contains basic information of known devices and sensors, including device type, data format, collection frequency and communication protocol;

[0102] S12: Scan the equipment and sensors at the industrial site through the equipment management module to collect real-time information of the equipment and sensors, including equipment ID, model, location and current status;

[0103] S13: Match the collected real-time information with the initial list, use the support vector machine (SVM) algorithm for pattern recognition, identify the newly added devices and sensors, and update the initial list; the pattern recognition formula is: Where f(x) is the classification function, x is the input vector, n is the number of support vectors, α i is the Lagrange multiplier, y i is the sample label, K(xi , x) is the kernel function, x i is the support vector, b is the bias term;

[0104] S14: For the matched and identified devices and sensors, the convolutional neural network algorithm is used to analyze their data formats and communication protocols, and the adaptive communication protocols and data formats are automatically configured through the preset protocol parsing library; the formula of the convolutional neural network is: Among them, h i,j,k is the convolution output, i and j are spatial position indexes, k is the channel index, σ is the activation function, and w m,n,k is the convolution kernel, x i+m,j+n is the input data, m and n are the spatial sizes of the convolution kernel, and b k is the bias term;

[0105] S15: Dynamically adjust the data collection frequency according to the device type and data characteristics to maximize the accuracy and timeliness of data collection without affecting the normal operation of the device; assuming that the collection frequency is f, the adjustment formula is: Where, f is the acquisition frequency, f max is the maximum acquisition frequency, C is the preset constant, and V is the data change rate;

[0106] S16: Generate a complete configuration file for devices and sensors, including device type, data format, acquisition frequency and communication protocol, and store the configuration file in the configuration management module of the industrial Internet operating system for use in subsequent steps; the above steps S11 to S16, through the combination of device management module, support vector machine (SVM), convolutional neural network (CNN) and adaptive algorithm, realize comprehensive and accurate identification and configuration of devices and sensors.

[0107] S2 specifically includes:

[0108] S21: Collect raw data from data sources in real time, using the configured data format, collection frequency, and communication protocol to obtain raw data generated by industrial equipment and sensors;

[0109] S22: De-noising the raw data collected in real time, using a denoising algorithm based on wavelet transform to remove random noise signals in the raw data to improve the purity and credibility of the data. The wavelet transform denoising algorithm includes three steps: wavelet decomposition, threshold processing and wavelet reconstruction;

[0110] S23: Using a Kalman filter to filter the denoised data, filter out high-frequency interference components in the data, retain the effective signal part of the data, and enhance the stability of the data. The Kalman filter uses state estimation and noise covariance matrix to perform filtering;

[0111] S24: repairing the filtered data, estimating and completing missing values ​​in the data using the Lagrange interpolation method to ensure the integrity and continuity of the data. The Lagrange interpolation method completes the missing data through polynomial interpolation of known data points;

[0112] S25: Standardize the repaired data, specifically use the Z-score standardization algorithm to convert the data to the same scale range, eliminate the scale differences between different data sources, and improve the consistency and comparability of the data. The Z-score standardization algorithm achieves data standardization by calculating the deviation of each data point from the mean and dividing it by the standard deviation; the above steps S21 to S25, through data acquisition, wavelet transform denoising, Kalman filtering, Lagrange interpolation and Z-score standardization, achieve high-quality acquisition and preprocessing of the original data, providing a reliable foundation for subsequent data analysis and processing.

[0113] S3 specifically includes:

[0114] S31: Deploy edge computing nodes on industrial equipment, including embedded computing devices or industrial controllers, each of which is configured with computing resources and storage capacity to perform edge computing tasks;

[0115] S32: The pre-processed data is transmitted to the edge computing node through the data transmission module in the industrial Internet operating system, and preliminary analysis and processing are performed on the edge computing node, including data aggregation, data filtering and data feature extraction; data aggregation is used to merge data from multiple data sources, data filtering is used to remove irrelevant or invalid data, and data feature extraction is used to extract meaningful feature values ​​from the original data;

[0116] S33: using the resource scheduling algorithm of the edge computing node, according to the complexity of the computing task and the load of each edge node, determine the distribution of the computing task, assign the tasks with high computing load to the edge nodes with high computing power, and assign the tasks with low computing load to other edge nodes, so as to achieve load balancing; the resource scheduling algorithm includes but is not limited to the minimum load priority algorithm and the task priority algorithm;

[0117] S34: Execute the assigned computing tasks on the edge computing nodes, including data processing, model reasoning, and real-time response. Data processing refers to further cleaning and conversion of data, model reasoning refers to predicting and classifying data using pre-trained machine learning models, and real-time response refers to generating control instructions based on the calculation results and feeding them back to industrial equipment.

[0118] S35: The result data processed by the edge computing nodes are aggregated to the central data processing center through the industrial Internet operating system, and the result data are summarized;

[0119] The above steps S31 to S35, by deploying edge computing nodes, preliminarily analyzing and processing data, using resource scheduling algorithms to distribute computing tasks, executing computing tasks and aggregating result data, achieve efficient preliminary analysis and processing of preprocessed data, thereby reducing the burden on the central processing unit.

[0120] S33 specifically includes:

[0121] S331: Divide the task into multiple subtasks according to the complexity of the computing task; the complexity C is measured by the computing time T and resource consumption R required by the task, and the formula is: C = T × R, where T is the computing time and R is the resource consumption;

[0122] S332: Obtain real-time load information of each edge node, including CPU usage U, memory occupancy M, and network bandwidth B; the load L calculation formula is: L = αU + βM + γB, where α, β, and γ are weight coefficients, reflecting the influence weight of each resource in the total load;

[0123] S333: Calculate the available resource amount A of each edge node, the formula is: A = R max -L, where R max is the maximum resource capacity of the node, and L is the current load of the node;

[0124] S334: According to the task complexity C and the available resource amount A of the edge node, the distribution of the computing task is determined by using a resource scheduling algorithm; the resource scheduling algorithm adopts a minimum resource utilization priority strategy based on load balancing, and the specific formula is: Among them, N is the selected edge node, i is the index of all candidate edge nodes, A i is the available resources of the ith edge node, Indicates taking the i value that minimizes the score value; and assigning subtasks to the selected edge node N, ensuring that each subtask is assigned to the node with the best resources, thereby ensuring efficient execution of the computing task; the above steps S331 to S334, by calculating the complexity of the computing tasks, real-time load information and the amount of available resources, adopt the minimum resource utilization priority strategy, realize the reasonable allocation and load balancing of the computing tasks, and ensure the efficient operation of the system.

[0125] Choosing an appropriate storage strategy in S4 specifically includes:

[0126] S41: Evaluate the importance of data processed by edge computing, and set the data importance level I based on the source, type, purpose and impact of the data on the industrial production process; the data importance level is divided into three categories: high, medium and low, corresponding to importance level I h ,Im and I l ;

[0127] S42: Classify data according to real-time requirements and evaluate the real-time requirements R of the data. The real-time requirements are divided into high real-time, medium real-time and low real-time, corresponding to the real-time level R respectively. h , R m and R l ;

[0128] S43: Determine a data storage strategy based on the importance level I and the real-time requirement R of the data. The data storage strategy includes real-time storage, periodic storage, and archive storage.

[0129] For data with high importance and high real-time h ,R h , adopts real-time storage strategy to store data directly in high-performance distributed database to ensure real-time access and processing of data;

[0130] For data with high importance but medium real-time requirements h ,R m , or data of medium importance and high real-time nature m ,R h ,Using a periodic storage strategy, the data is stored in a distributed database according to a preset period;

[0131] For data of medium importance and medium real-time m ,R m , or data with high importance but low real-time nature h ,R l ,adopting a hybrid storage strategy, storing part of the data in real time in a distributed database and the other part periodically in the cloud storage;

[0132] For low-importance but high-time data l ,R h , using a low-latency storage strategy to temporarily store data in the cache module and store it in batches in the distributed database during idle periods;

[0133] For data of medium importance but low real-time nature m ,R l , or data of low importance and medium real-time nature l ,R m , adopt an archiving storage strategy to store data directly in cloud storage to reduce storage costs;

[0134] For data with low importance and low real-time performance, l ,R l, adopt a delayed storage strategy to temporarily store the data in the local storage of the edge node, and store it in batches in the cloud storage after the data volume reaches a certain threshold; according to the determined storage strategy, the data is classified and stored in the distributed database or cloud storage through the storage management module of the industrial Internet operating system to ensure the efficiency and reliability of data storage; the above steps S41 to S43, through data importance assessment, real-time demand classification and the application of specific storage strategies, realize efficient storage and management of data, and ensure the security and availability of data storage.

[0135] S5 specifically includes:

[0136] S51: fragmenting the data in the cache according to a preset time window and data type; the time window is implemented by a predetermined fixed time interval or a dynamically adjusted time interval, and the data in each time window is regarded as a fragment;

[0137] S52: Divide the data in each time window to generate multiple data segments, each data segment includes a data set in the time window;

[0138] S53: adding a label to each data fragment, including a timestamp, a device ID, a data type, and an abnormality mark; specifically comprising the following steps:

[0139] S531: Add a timestamp to each data segment, where the timestamp represents the start time of the data segment, obtain the current time through the system clock and record it;

[0140] S532: adding a device ID to each data fragment, where the device ID represents the device or sensor that generates the data, and obtaining a unique identifier of the device or sensor through the configuration information of the data source;

[0141] S533: adding a data type tag to each data segment, where the data type tag represents the type of data, and the data type is determined by analyzing the configuration information of the data source or the data format, such as temperature data, pressure data, etc.;

[0142] S534: Perform anomaly detection on each data segment and add an anomaly mark. The anomaly detection algorithm adopts a statistical method, such as anomaly detection based on mean and standard deviation. If the data value in the data segment exceeds the normal range, an anomaly mark is added.

[0143] S54: Store the tagged data segments in the cache module of the industrial Internet operating system for subsequent data synchronization and processing; the above steps S51 to S54, through time window division, tag addition and anomaly detection, realize the systematic management of data segments, providing a reliable foundation for subsequent data processing and analysis.

[0144] S6 specifically includes:

[0145] S61: Select a transport protocol, including MQTT, HTTP, or CoAP, through the communication module in the Industrial Internet operating system, and transmit the fragmented data from the edge node to the central data processing center;

[0146] S62: During data transmission, the built-in encryption and compression functions of the transmission protocol are used to ensure data security and transmission efficiency, for example, the TLS protocol is used for data encryption and the Gzip algorithm is used for data compression;

[0147] S63: After receiving the fragmented data, the central data processing center performs synchronization processing, specifically by comparing timestamps and performing hash verification to ensure that the timing and content of each data fragment are consistent;

[0148] S64: Perform consistency check on the received data fragments, specifically including the following steps:

[0149] S641: Calculate the hash value of each data fragment and compare it with the hash value transmitted by the edge node to ensure that the data has not been tampered with during transmission;

[0150] S642: Compare the timestamps of the data segments to check whether there is data loss or repeated transmission. If an abnormality is found, retransmit or merge the data.

[0151] S643: Using the consistency check algorithm based on blockchain technology, the hash value of the data fragment is stored in the blockchain network to ensure that the data cannot be tampered with and is traceable; the above steps S61 to S64, by selecting appropriate transmission protocols, encrypting and compressing data, using timestamps and hash checks for data synchronization, and using blockchain technology for consistency check, achieve the security, consistency and reliability of data transmission, providing a solid foundation for subsequent data processing and analysis.

[0152] S7 specifically includes:

[0153] S71: After the central data processing center receives the data, the data parsing module is used to parse the data. The parsing process includes data format recognition and data content extraction. The data format recognition is used to match the received data format through the format library. The data content is used to extract the data content through the parser to convert the data content into structured data that can be processed internally.

[0154] S72: According to different application requirements, the structured data is converted into a corresponding format; specifically, the following steps are included:

[0155] S721: Select corresponding data conversion rules according to application requirements; the data conversion rules are pre-defined in a conversion rule library, including mapping relationships between different data formats;

[0156] S722: using the selected conversion rule, convert the structured data into a target format, where the target format includes common data formats such as JSON, XML, and CSV;

[0157] S723: During the conversion process, a preset formula is used to perform data unit conversion and format adjustment. Specifically, for the unit conversion of temperature data, the formula is used: Convert Fahrenheit to Celsius, where T C is the temperature in degrees Celsius, T F is the temperature in degrees Fahrenheit;

[0158] S73: Open a standardized data interface for third-party applications and analysis tools to use, including the following steps:

[0159] S731: Deploy RESTful API services in the central data processing center to provide standardized data access interfaces; RESTful API services define interfaces based on resource paths and HTTP methods, including GET, POST, PUT, and DELETE;

[0160] S732: Define detailed API documentation for each data interface, including the interface's request format, response format, parameter description, and sample code, to ensure that third-party applications and analysis tools can correctly call the interface;

[0161] S733: Use API gateway to perform security management and flow control on data interfaces, and use OAuth 2.0 for identity authentication and authorization to ensure the security and reliability of data access;

[0162] S74: Store the parsed and converted data in the database of the central data processing center, and provide it to third-party applications and analysis tools in real time through standardized data interfaces to support the needs of various application scenarios; the above steps S71 to S74, through data format identification, content extraction, conversion rule library, RESTful API service and security management, realize data parsing, format conversion and interface opening, providing efficient data support for various application scenarios.

[0163] like Figure 2 As shown, the data acquisition system based on the industrial Internet operating system is used to implement the above-mentioned data acquisition method based on the industrial Internet operating system, and includes the following modules:

[0164] Device management module: used to establish and maintain the initial list of industrial devices and sensors, including device type, data format, acquisition frequency and communication protocol, and automatically detect and identify newly added devices and sensors;

[0165] Data acquisition module: used to collect raw data from identified devices and sensors in real time, and pre-process the raw data, including denoising, filtering, repairing and standardization, to improve data quality and consistency;

[0166] Edge computing module: performs preliminary analysis and processing on pre-processed data at the industrial equipment end, distributes computing tasks to multiple edge nodes, reduces the burden on the central processing unit, and improves the real-time and efficiency of data processing;

[0167] Cache module: used to temporarily store data processed by edge computing, select appropriate storage strategies based on the importance and real-time needs of the data, and classify and store the data in a distributed database or cloud storage;

[0168] Data fragmentation module: fragments the data in the cache according to the preset time window and data type, and adds tags to each data fragment, including timestamp, device ID, data type and abnormal mark;

[0169] Communication module: By selecting an appropriate transmission protocol, the fragmented data is transmitted from the edge node to the central data processing center, and encryption and compression technology are used during the transmission process to ensure data security and transmission efficiency;

[0170] Data synchronization module: synchronizes the received data in the central data processing center, and ensures the consistency of the timing and content of each data fragment through timestamp comparison and hash verification;

[0171] Data verification module: performs consistency verification on the received data fragments, calculates the hash value of each data fragment, and uses blockchain technology to store the hash value to ensure that the data cannot be tampered with and is traceable;

[0172] Data parsing module: parses the received data, including data format recognition and data content extraction, and converts the data into corresponding formats according to different application requirements;

[0173] Data interface module: Open standardized data interfaces for third-party applications and analysis tools to call, provide RESTful API services, define detailed API documents, and perform security management and traffic control through API gateways.

[0174] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0175] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data collection method based on an industrial Internet operating system, characterized in that: The following steps are involved: S1: Based on the Industrial Internet operating system, it uses a preset intelligent recognition algorithm to automatically detect and identify the data sources of multiple industrial devices and sensors, including device type, data format, collection frequency and communication protocol; S2: Collect raw data from the data source in real time and pre-process the raw data, including denoising, filtering, repairing and standardization; S3: Perform edge computing on the industrial equipment side, conduct preliminary analysis and processing on the preprocessed data, distribute some computing tasks to multiple edge nodes, and reduce the burden on the central processing unit; S4: Temporarily store the data processed by edge computing in the cache module of the industrial Internet operating system, and select appropriate storage strategies based on the importance and real-time needs of the data to classify and store the data in a distributed database or cloud storage; S5: According to the preset time window and data type, the data in the cache is fragmented and a label is added to each data fragment; S6: Through the multi-channel communication module of the industrial Internet operating system, the fragmented data is transmitted to the central data processing center for data synchronization and consistency verification; S7: In the central data processing center, the received data is parsed and converted into the corresponding format according to different application requirements, and a standardized data interface is opened for third-party applications and analysis tools to call.

2. The data collection method based on the industrial Internet operating system according to claim 1 is characterized in that: The S1 specifically includes: S11: Through the device management module in the industrial Internet operating system, an initial list of devices and sensors is established, which contains basic information of known devices and sensors, including device type, data format, collection frequency and communication protocol; S12: Scan the equipment and sensors at the industrial site through the equipment management module to collect real-time information of the equipment and sensors, including equipment ID, model, location and current status; S13: matching the collected real-time information with the initial list, using the support vector machine algorithm for pattern recognition, identifying newly added devices and sensors, and updating the initial list; S14: For the matched and identified devices and sensors, the convolutional neural network algorithm is used to analyze their data formats and communication protocols, and the preset protocol parsing library is used to automatically configure the adaptive communication protocols and data formats; S15: Dynamically adjust the data collection frequency according to the device type and data characteristics to maximize the accuracy and timeliness of data collection without affecting the normal operation of the device; assuming that the collection frequency is f, the adjustment formula is: Where, f is the acquisition frequency, f max is the maximum acquisition frequency, C is the preset constant, and V is the data change rate; S16: Generate a complete configuration file of the device and sensor, including device type, data format, acquisition frequency, and communication protocol.

3. The data collection method based on the industrial Internet operating system according to claim 1 is characterized in that: The S2 specifically includes: S21: Collect raw data from data sources in real time, using the configured data format, collection frequency, and communication protocol to obtain raw data generated by industrial equipment and sensors; S22: performing denoising processing on the raw data collected in real time, using a denoising algorithm based on wavelet transform to remove random noise signals in the raw data; S23: Using a Kalman filter to filter the denoised data, filter out high-frequency interference components in the data, and retain the effective signal part of the data; S24: repairing the filtered data, and estimating and completing the missing values ​​in the data using the Lagrange interpolation method; S25: Standardize the repaired data, specifically use the Z-score standardization algorithm to convert the data to the same scale range and eliminate the scale differences between different data sources.

4. The data collection method based on the industrial Internet operating system according to claim 1 is characterized in that: The S3 specifically includes: S31: Deploy edge computing nodes on industrial equipment, including embedded computing devices or industrial controllers, each of which is configured with computing resources and storage capacity to perform edge computing tasks; S32: The pre-processed data is transmitted to the edge computing node through the data transmission module in the industrial Internet operating system, and preliminary analysis and processing are performed on the edge computing node, including data aggregation, data filtering, and data feature extraction; S33: using the resource scheduling algorithm of the edge computing node, according to the complexity of the computing task and the load of each edge node, determine the distribution of the computing task, allocate the tasks with high computing load to the edge nodes with high computing power, and allocate the tasks with low computing load to other edge nodes, so as to achieve load balancing; S34: Execute the assigned computing tasks on the edge computing nodes, including data processing, model reasoning, and real-time response; S35: The result data processed by the edge computing nodes are aggregated to the central data processing center through the industrial Internet operating system, and the result data are summarized.

5. The data collection method based on the industrial Internet operating system according to claim 4 is characterized in that: The S33 specifically includes: S331: Divide the task into multiple subtasks according to the complexity of the computing task; the complexity C is measured by the computing time T and resource consumption R required by the task, and the formula is: C = T × R, where T is the computing time and R is the resource consumption; S332: Obtain real-time load information of each edge node, including CPU usage U, memory occupancy M, and network bandwidth B; the load L calculation formula is: L = αU + βM + γB, where α, β, and γ are weight coefficients, reflecting the influence weight of each resource in the total load; S333: Calculate the available resource amount A of each edge node, the formula is: A = R max -L, where R max is the maximum resource capacity of the node, and L is the current load of the node; S334: According to the task complexity C and the available resource amount A of the edge node, the distribution of the computing task is determined by using a resource scheduling algorithm; the resource scheduling algorithm adopts a minimum resource utilization priority strategy based on load balancing, and the specific formula is: Among them, N is the selected edge node, i is the index of all candidate edge nodes, A i is the available resources of the ith edge node, It means taking the i value that minimizes the score value; and assigning the subtask to the selected edge node N.

6. The data collection method based on the industrial Internet operating system according to claim 1 is characterized in that: The S4 specifically includes: S41: Evaluate the importance of data processed by edge computing, and set the data importance level I based on the source, type, purpose and impact of the data on the industrial production process; the data importance level is divided into three categories: high, medium and low, corresponding to importance level I h ,I m and I l ; S42: Classify data according to real-time requirements and evaluate the real-time requirements R of the data. The real-time requirements are divided into high real-time, medium real-time and low real-time, corresponding to the real-time level R respectively. h , R m and R l ; S43: Determine a data storage strategy based on the importance level I and the real-time requirement R of the data, wherein the data storage strategy includes real-time storage, periodic storage, and archive storage; For data with high importance and high real-time h ,R h , using real-time storage strategies to store data directly in high-performance distributed databases; For data with high importance but medium real-time requirements h ,R m , or data of medium importance and high real-time nature m ,R h ,Using a periodic storage strategy, the data is stored in a distributed database according to a preset period; For data of medium importance and medium real-time m ,R m , or data with high importance but low real-time nature h ,R l ,adopting a hybrid storage strategy, storing part of the data in real time in a distributed database and the other part periodically in the cloud storage; For low-importance but high-time data l ,R h ,Using a low-latency storage strategy, the data is temporarily stored in the cache module; For data of medium importance but low real-time nature m ,R l , or data of low importance and medium real-time nature l ,R m ,Adopt an archival storage strategy and store data directly in the cloud storage; For data with low importance and low real-time performance, l ,R l ,A delayed storage strategy is adopted to temporarily store data in the local storage of the edge node, and then store it in batches in the cloud storage when the data volume reaches a certain threshold.

7. The data collection method based on the industrial Internet operating system according to claim 1 is characterized in that: The S5 specifically includes: S51: fragmenting the data in the cache according to a preset time window and data type; the time window is implemented by a predetermined fixed time interval or a dynamically adjusted time interval, and the data in each time window is regarded as a fragment; S52: Divide the data in each time window to generate multiple data segments, each data segment includes a data set in the time window; S53: adding a label to each data segment, including a timestamp, a device ID, a data type, and an abnormality mark; S54: Store the labeled data segments in a cache module of the industrial Internet operating system.

8. The data collection method based on the industrial Internet operating system according to claim 1 is characterized in that: The S6 specifically includes: S61: Through the communication module in the Industrial Internet operating system, select the transmission protocol, including MQTT, HTTP or CoAP, and transmit the fragmented data from the edge node to the central data processing center; S62: During data transmission, the TLS protocol is used for data encryption, and the Gzip algorithm is used for data compression; S63: After receiving the fragmented data, the central data processing center performs synchronization processing, specifically by comparing timestamps and performing hash verification to ensure that the timing and content of each data fragment are consistent; S64: Perform consistency check on the received data fragments, specifically including the following steps: S641: Calculate the hash value of each data fragment and compare it with the hash value transmitted by the edge node; S642: Compare the timestamps of the data segments to check whether there is data loss or repeated transmission. If an abnormality is found, retransmit or merge the data. S643: Using a consistency check algorithm based on blockchain technology, the hash value of the data fragment is stored in the blockchain network.

9. The data collection method based on the industrial Internet operating system according to claim 1 is characterized in that: The S7 specifically includes: S71: After the central data processing center receives the data, the data parsing module is used to parse the data. The parsing process includes data format recognition and data content extraction. The data format recognition is used to match the received data format through the format library. The data content is used to extract the data content and convert it into structured data through the parser. S72: According to different application requirements, the structured data is converted into a corresponding format; specifically, the following steps are included: S721: Select corresponding data conversion rules according to application requirements; the data conversion rules are pre-defined in a conversion rule library, including mapping relationships between different data formats; S722: using the selected conversion rule, convert the structured data into a target format, where the target format includes JSON, XML, and CSV; S723: During the conversion process, data unit conversion and format adjustment are performed simultaneously; S73: Open a standardized data interface for third-party applications and analysis tools to use, including the following steps: S731: Deploy RESTful API services in the central data processing center to provide standardized data access interfaces; RESTful API services define interfaces based on resource paths and HTTP methods, including GET, POST, PUT, and DELETE; S732: Define detailed API documentation for each data interface, including the interface's request format, response format, parameter description, and sample code; S733: Use API gateway to perform security management and flow control on data interfaces, and use OAuth 2.0 for identity authentication and authorization; S74: The parsed and converted data are stored in a database of a central data processing center, and provided to third-party applications and analysis tools in real time through a standardized data interface.

10. A data acquisition system based on an industrial Internet operating system, used to implement the data acquisition method based on an industrial Internet operating system as described in any one of claims 1 to 9, characterized in that: Includes the following modules: Device management module: used to establish and maintain the initial list of industrial devices and sensors, including device type, data format, acquisition frequency and communication protocol, and automatically detect and identify newly added devices and sensors; Data acquisition module: used to collect raw data from identified devices and sensors in real time, and pre-process the raw data, including denoising, filtering, repairing and standardization; Edge computing module: performs preliminary analysis and processing on pre-processed data at the industrial equipment end, distributes computing tasks to multiple edge nodes, and reduces the burden on the central processing unit; Cache module: used to temporarily store data processed by edge computing, select appropriate storage strategies based on the importance and real-time needs of the data, and classify and store the data in a distributed database or cloud storage; Data fragmentation module: fragments the data in the cache according to the preset time window and data type, and adds tags to each data fragment, including timestamp, device ID, data type and abnormal mark; Communication module: Transmits fragmented data from edge nodes to the central data processing center by selecting appropriate transmission protocols and utilizing encryption and compression techniques during transmission; Data synchronization module: synchronizes the received data in the central data processing center; Data verification module: performs consistency verification on the received data fragments, calculates the hash value of each data fragment, and stores the hash value using blockchain technology; Data parsing module: parses the received data, including data format recognition and data content extraction, and converts the data into corresponding formats according to different application requirements; Data interface module: Open standardized data interfaces for third-party applications and analysis tools to call, provide RESTful API services, define detailed API documents, and perform security management and traffic control through API gateways.

Images