A data processing method, system and storage medium based on industrial networking

Abstract

The present invention relates to the field of data processing technology, and in particular to a method, system and storage medium for data processing based on industrial networking. The method comprises the following steps: collecting on-site industrial data at an industrial site to obtain original industrial data; performing data stratification on the original industrial data to obtain edge layer industrial data and cloud layer industrial data; storing edge layer industrial data; performing industrial data splitting on cloud layer industrial data to obtain edge layer processing data and cloud layer processing data; performing complex event processing on edge layer processing data to obtain monitoring and analysis data; performing real-time monitoring of industrial networks at industrial sites to obtain real-time communication network status data; and sending cloud layer processing data and monitoring and analysis data to the cloud based on real-time communication network status data and storing them. The present invention can reduce the computing burden of cloud servers and save the use of computing resources and storage resources.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, system and storage medium based on industrial networking. Background Art

[0002] At industrial production sites, with the rapid development of industrial Internet technology, more and more sensors and equipment are being used in the production process, generating a large amount of industrial data. These data include but are not limited to parameter data such as temperature, pressure, humidity, vibration, and other parameter data during the production process, as well as information such as equipment status and production efficiency. Using these data, the production process can be monitored, optimized, and predicted, production efficiency and product quality can be improved, energy consumption and maintenance costs can be reduced, and intelligent industrial production management can be achieved.

[0003] However, current industrial Internet data processing faces some challenges and problems. Transmitting and storing large amounts of data requires a lot of computing and storage resources, which limits the efficiency and real-time nature of data processing. Existing industrial data processing methods usually upload various data to the cloud for processing and storage, but this approach will lead to a large waste of cloud server resources and increased costs.

[0004] First, uploading a large amount of data to the cloud consumes a lot of network bandwidth resources, increases data transmission delays, and causes network congestion and instability. Second, cloud servers need to consume a lot of computing resources and storage resources to process and store this data, and usually charge according to a fixed billing method, which means that even if the amount of data is not large, you still need to pay the corresponding fees, resulting in a waste of resources and increased costs. Summary of the invention

[0005] Based on this, it is necessary for the present invention to provide a data processing method, system and storage medium based on industrial networking to solve at least one of the above technical problems.

[0006] To achieve the above purpose, a method for processing data based on industrial networking includes the following steps:

[0007] Step S1: collecting on-site industrial data at the industrial site to obtain original industrial data;

[0008] Step S2: stratify the original industrial data to obtain edge layer industrial data and cloud layer industrial data; store the edge layer industrial data;

[0009] Step S3: splitting the cloud layer industrial data into industrial data to obtain edge layer processing data and cloud layer processing data;

[0010] Step S4: Perform complex event processing on the edge layer processing data to obtain monitoring and analysis data; perform real-time monitoring of the industrial network on the industrial site to obtain real-time communication network status data;

[0011] Step S5: Based on the real-time communication network status data, the cloud layer processing data and the monitoring and analysis data are sent to the cloud and stored.

[0012] The present invention can reduce the amount of data transmitted to the cloud by performing layered processing of industrial data on site, reduce the consumption of network bandwidth and data transmission delay, and thus improve the efficiency and real-time performance of data processing. By storing edge layer industrial data, the computing burden of the cloud server can be reduced, saving the use of computing resources and storage resources. In addition, by splitting the cloud layer industrial data, only the edge layer processing data and monitoring and analysis data are sent to the cloud for storage, avoiding the transmission and storage of a large amount of data that does not need to be processed, further saving resources. By performing complex event processing on the edge layer processing data, monitoring and analysis data can be obtained to achieve real-time monitoring and analysis of the production process. At the same time, through real-time monitoring of the industrial network, real-time communication network status data can be obtained to provide real-time evaluation of network performance and stability. By performing real-time processing and analysis on the monitoring and analysis data of the industrial site, abnormal conditions and potential problems in the production process can be discovered in time.

[0013] Preferably, the present invention further provides an industrial networking-based data processing system for executing the industrial networking-based data processing method as described above, the industrial networking-based data processing system comprising:

[0014] The data acquisition module is used to collect industrial data on the industrial site to obtain the original industrial data;

[0015] The data tiered storage module is used to tier the original industrial data to obtain edge layer industrial data and cloud layer industrial data; and store the edge layer industrial data;

[0016] The cloud data splitting module is used to split the cloud layer industrial data into industrial data to obtain edge layer processing data and cloud layer processing data;

[0017] The complex event processing module is used to perform complex event processing on edge layer processing data to obtain monitoring and analysis data; perform real-time monitoring of industrial networks at industrial sites to obtain real-time communication network status data;

[0018] The cloud storage module is used to send and store cloud layer processing data and monitoring analysis data to the cloud based on real-time communication network status data.

[0019] In the present invention, the data acquisition module is used to collect on-site industrial data on the industrial site to obtain the original industrial data. This can reduce the time and labor cost of manual collection and improve the efficiency and accuracy of data collection. The original industrial data is processed in layers by the data layered storage module to obtain edge layer industrial data and cloud layer industrial data, and the edge layer industrial data is stored. This can reduce the amount of data transmission to the cloud, reduce the consumption of network bandwidth and data transmission delay, thereby improving the efficiency and real-time performance of data processing. The cloud layer industrial data is split by the cloud data splitting module to obtain edge layer processing data and cloud layer processing data, and only the data to be processed is sent to the cloud for storage. This can avoid the transmission and storage of a large amount of data that does not need to be processed, optimize the utilization of resources, and reduce the waste of resources. The complex event processing module is used to process the edge layer processing data to obtain monitoring and analysis data. At the same time, the industrial network is monitored in real time on the industrial site to obtain real-time communication network status data. This can realize real-time monitoring and analysis of the production process and timely discover abnormal situations and potential problems. The cloud storage module sends cloud-layer processing data and monitoring and analysis data to the cloud and stores them, ensuring the security and reliability of the data. Cloud storage can provide backup and disaster recovery capabilities, prevent data loss and damage, and facilitate remote access and sharing of data. In summary, the present invention achieves high efficiency in data processing, optimizes resource utilization, improves real-time monitoring and analysis capabilities, and ensures data security and reliability, thereby effectively supporting the needs of industrial networking data processing.

[0020] Preferably, a computer-readable storage medium stores a data processing method based on industrial networking as described above which can be loaded and executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments thereof made with reference to the following drawings:

[0022] Figure 1 A schematic flow chart of the steps of a method for processing data based on industrial networking according to an embodiment is shown.

[0023] Figure 2 A detailed flow chart of step S24 of an embodiment is shown.

[0024] Figure 3 A detailed flow chart of step S248 of an embodiment is shown. DETAILED DESCRIPTION

[0025] The following is a clear and complete description of the technical method of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different network and / or processor methods and / or microcontroller methods.

[0027] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0028] To achieve this, please refer to Figures 1 to 3 The present invention provides a method for processing data based on industrial networking, comprising the following steps:

[0029] Step S1: collecting on-site industrial data at the industrial site to obtain original industrial data;

[0030] Step S2: stratify the original industrial data to obtain edge layer industrial data and cloud layer industrial data; store the edge layer industrial data;

[0031] Step S3: splitting the cloud layer industrial data into industrial data to obtain edge layer processing data and cloud layer processing data;

[0032] Step S4: Perform complex event processing on the edge layer processing data to obtain monitoring and analysis data; perform real-time monitoring of the industrial network on the industrial site to obtain real-time communication network status data;

[0033] Step S5: Based on the real-time communication network status data, the cloud layer processing data and the monitoring and analysis data are sent to the cloud and stored.

[0034] The present invention can reduce the amount of data transmitted to the cloud by performing layered processing of industrial data on site, reduce the consumption of network bandwidth and data transmission delay, and thus improve the efficiency and real-time performance of data processing. By storing edge layer industrial data, the computing burden of the cloud server can be reduced, saving the use of computing resources and storage resources. In addition, by splitting the cloud layer industrial data, only the edge layer processing data and monitoring and analysis data are sent to the cloud for storage, avoiding the transmission and storage of a large amount of data that does not need to be processed, further saving resources. By performing complex event processing on the edge layer processing data, monitoring and analysis data can be obtained to achieve real-time monitoring and analysis of the production process. At the same time, through real-time monitoring of the industrial network, real-time communication network status data can be obtained to provide real-time evaluation of network performance and stability. By performing real-time processing and analysis on the monitoring and analysis data of the industrial site, abnormal conditions and potential problems in the production process can be discovered in time.

[0035] In this embodiment, for example, at an industrial site, various sensors and acquisition devices are first used to collect on-site industrial data to collect raw industrial data. These data include but are not limited to production equipment status, environmental monitoring information, and production process data. Next, the raw industrial data is processed in layers, the data is divided into edge layer industrial data and cloud layer industrial data, and the edge layer industrial data is stored in the local edge computing device to achieve fast response and low-latency processing. Subsequently, the cloud layer industrial data is split into industrial data, and the data is further divided into data that needs to be processed at the edge layer (edge ​​layer processing data) and data that needs to be processed in the cloud (cloud layer processing data). Complex event processing is performed on the edge layer processing data, and the complex event processing (CEP) engine is used to analyze these data in real time, identify and process industrial events, and generate monitoring and analysis data. At the same time, the industrial network is monitored in real time at the industrial site, and real-time communication network status data is obtained to ensure the reliability and stability of network transmission. Finally, based on the real-time communication network status data, the cloud layer processing data and monitoring and analysis data are transmitted to the cloud for storage to ensure centralized management and long-term preservation of the data for subsequent analysis and decision support.

[0036] Preferably, step S2 comprises the following steps:

[0037] Step S21: Acquire a collection device metadata list, wherein the collection device metadata list includes a number of device metadata entries, and the entries include a device ID, a data type identifier, and a data security value;

[0038] Specifically, start and configure the ICS server, connect the ICS server to the database management system, monitor all connected acquisition devices through the SCADA system, identify and record the ID, data type identifier, and data security value of each device. Write and run scripts to extract device metadata from the SCADA system through API or database query and store it in the database. Extract the metadata of all devices from the database and summarize them to generate a device metadata list. The list should include the ID, data type identifier, and data security value of each device.

[0039] Step S22: performing preliminary data classification on the original industrial data according to the data type identifier based on the collection device metadata list to obtain classified industrial data, wherein the classified industrial data includes industrial production data, equipment status data, product quality data, environmental data, human resources data, supply chain data and energy consumption data;

[0040] Specifically, the raw industrial data is classified into industrial production data, equipment status data, product quality data, environmental data, human resources data, supply chain data, and energy consumption data categories according to the data type identifier. The corresponding equipment metadata is matched to each piece of raw data, its data type is determined, and it is classified into the corresponding data set. The classified data is stored in different database tables or files respectively to generate the corresponding classified industrial data sets. For example, all entries identified as "industrial production data" will be stored in a separate database table, entries identified as "equipment status data" will be stored in another table, and so on.

[0041] Step S23: Obtain historical classified industrial data; perform data access pattern analysis on each type of industrial data in the historical classified industrial data based on a preset statistical time window to obtain a data access statistical table, wherein the attributes of the data access statistical table include a preset statistical time window, a data type identifier, a number of local accesses, a frequency of local data changes, a number of remote accesses, and a frequency of remote data changes;

[0042] Specifically, various types of industrial data are extracted from historical data storage systems (such as SQL databases or data warehouses), and the extracted data are imported into data processing tools such as Pandas libraries. Preset statistical time windows are determined, such as daily, weekly or monthly, and access patterns of each type of industrial data are analyzed based on these time windows. The attributes analyzed include the number of local accesses, the frequency of local data changes, the number of remote accesses and the frequency of remote data changes. In each statistical time window, the above attributes are counted to generate a data access statistics table.

[0043] Step S24: stratifying the classified industrial data based on the data access statistics table to obtain edge layer industrial data and cloud layer industrial data;

[0044] Specifically, the stratification standard is set according to the data access frequency and the change frequency, and the classified industrial data is stratified according to the stratification standard to obtain edge layer industrial data and cloud layer industrial data. For the detailed stratification process, refer to the sub-steps of step S24.

[0045] Step S25: storing edge layer industrial data.

[0046] Specifically, please refer to the sub-steps of step S25 for the detailed process of this embodiment.

[0047] The present invention obtains classified industrial data by preliminarily classifying the original industrial data according to the metadata list of the acquisition device and the data type identifier. This can accurately classify the data according to different types. By analyzing the data access mode of the historical classified industrial data based on the preset statistical time window, a data access statistics table is obtained. By counting the attributes such as the number of local accesses, the frequency of local data changes, the number of remote accesses and the frequency of remote data changes, the access mode and change frequency of different types of data can be understood. This can optimize the data access strategy and improve the data access efficiency and response speed. By layering the classified industrial data according to the data access statistics table, edge layer industrial data and cloud layer industrial data are obtained. By reasonably allocating the data to the edge layer and the cloud layer according to the attributes such as the number of local accesses and the frequency of local data changes in the data type identifier and the data access statistics table, hierarchical storage and processing of the data can be realized. This can reduce the amount of data transmission to the cloud, reduce the delay of data processing, and improve the efficiency and real-time performance of data processing. By storing the edge layer industrial data, the storage burden of the cloud server can be reduced and the use of storage resources can be saved.

[0048] Preferably, step S24 comprises the following steps:

[0049] Step S241: traverse the classified industrial data based on the data access statistics table, subtract the number of local accesses from the number of remote accesses for each type of industrial data, and obtain the local-remote access difference;

[0050] Specifically, the number of local accesses and the number of remote accesses for each type of industrial data are extracted from the data access statistics table. The classified industrial data are traversed one by one using data processing tools. For each type of industrial data, the difference between the number of local accesses and the number of remote accesses is calculated to obtain the local-remote access difference. For example, assuming that the number of local accesses of a certain type of industrial data in a certain statistical time window is 120 times and the number of remote accesses is 30 times, then the local-remote access difference is 90.

[0051] Step S242: if the local-foreign access difference is a positive value, and the local-foreign access difference is greater than or equal to the preset access difference, the corresponding class industrial data is recorded as edge layer industrial data;

[0052] Specifically, a preset access difference value is set, which can be determined based on actual business needs and experience, such as 50 times. Then, each type of industrial data is traversed, and if its local-remote access difference is positive and greater than or equal to the preset access difference, this type of industrial data is marked as edge layer industrial data. This means that this type of data is frequently accessed locally and is more suitable for storage in the edge layer to reduce access latency. For example, if the local-remote access difference of a certain type of industrial data is 90 (greater than the preset 50 times), it is classified as edge layer industrial data.

[0053] Step S243: If the local-foreign access difference is a positive value, and the local-foreign access difference is less than the preset access difference, then proceed to step S246;

[0054] Specifically, for those industrial data whose local-remote access difference is positive but less than the preset access difference, they are further processed. The specific operation steps are: traverse these data and check whether their local-remote access difference meets the conditions for entering the next step (step S246). For example, if the preset access difference is 50 times, and the local-remote access difference of a certain type of industrial data is 30 times (less than 50 times), then this type of data does not meet the storage standard of the edge layer and needs further processing and judgment.

[0055] Step S244: if the local-remote access difference is a negative value, and the absolute value of the local-remote access difference is greater than or equal to the preset access difference, the corresponding class industrial data is recorded as cloud-layer industrial data;

[0056] Specifically, a preset access difference value is set, such as 50 times. Then, all classified industrial data are traversed to check whether their local-remote access difference is a negative value. If the absolute value of the local-remote access difference of a certain type of industrial data is greater than or equal to the preset access difference, the data is recorded as cloud-layer industrial data. This means that this type of data is frequently accessed remotely and is more suitable to be stored in the cloud layer to utilize the powerful computing and storage capabilities of the cloud. For example, if the local-remote access difference of a certain type of industrial data is -60 (its absolute value is 60, which is greater than the preset 50 times), it is classified as cloud-layer industrial data.

[0057] Step S245: if the local-foreign access difference is a negative value, and the absolute value of the local-foreign access difference is less than the preset access difference, then proceed to step S246;

[0058] Specifically, for those industrial data whose local-remote access difference is negative but whose absolute value is less than the preset access difference, further processing is required. The specific operation steps are: traverse these data and check whether their local-remote access difference meets the conditions for entering the next step (step S246). For example, if the preset access difference is 50 times, and the local-remote access difference of a certain type of industrial data is -30 (the absolute value is 30, which is less than 50 times), then this type of data does not meet the cloud layer storage standard and requires further processing and judgment.

[0059] Step S246: Compare and judge the corresponding local data change frequency with the remote data change frequency. If the local data change frequency is greater than or equal to the remote data change frequency, the corresponding quasi-industrial data is recorded as edge layer industrial data. If the local data change frequency is less than the remote data change frequency, the corresponding quasi-industrial data is recorded as cloud layer industrial data.

[0060] Specifically, use a data processing tool to traverse the industrial data marked in steps S243 and S245 one by one, and compare the local data change frequency with the remote data change frequency. If the local data change frequency of a certain type of industrial data is greater than or equal to the remote data change frequency, it is marked as edge layer industrial data. This is because data that changes frequently locally is more suitable for storage in the edge layer to reduce latency and improve real-time processing capabilities. Conversely, if the local data change frequency is less than the remote data change frequency, it is marked as cloud layer industrial data, because these data have a higher frequency of changes in remote locations and are more suitable for utilizing the powerful processing and storage capabilities of the cloud. For example, if the local change frequency of a certain type of industrial data is 8 times / day, and the remote change frequency is 5 times / day, it is classified as edge layer industrial data.

[0061] Step S247: when the classified industrial data is completely traversed, that is, the edge layer industrial data and the cloud layer industrial data are obtained, an empty set judgment is performed on the edge layer industrial data and the cloud layer industrial data to obtain empty set judgment result data;

[0062] Specifically, all classified data are traversed to check whether the sets of edge layer industrial data and cloud layer industrial data are empty. Use data processing tools to make empty set judgments on these two sets. If any set is found to be empty, the classification standard needs to be further adjusted. For example, if after the previous steps, the edge layer industrial data set is empty, but the cloud layer industrial data set is not empty, this means that the current classification standard is too strict and needs to be adjusted. The judgment result data will record the current classification status, including whether the set is non-empty or empty.

[0063] Step S248: Adjust the preset access difference according to the empty set judgment result data to obtain the adjusted access difference, and return the adjusted access difference to step S242 until both the edge layer industrial data and the cloud layer industrial data are not empty sets.

[0064] Specifically, if any set is empty, it means that the current classification standard fails to effectively distribute the data to the edge layer and the cloud layer. According to the empty set judgment result, the preset access difference value is appropriately reduced or increased to make the classification standard more relaxed or strict, until both the edge layer industrial data and the cloud layer industrial data are not empty sets. For example, if the edge layer industrial data set is empty, the access difference can be reduced from 50 times to 40 times, and the classified industrial data is traversed again, and the operations of steps S242 to S247 are repeated until both the edge layer industrial data and the cloud layer industrial data are not empty sets.

[0065] The present invention makes hierarchical judgment on classified industrial data according to the number of local accesses and the number of remote accesses in the data access statistics table, as well as the local data change frequency and the remote data change frequency. According to the comparison of the local-remote access difference and the data change frequency, the industrial data is accurately allocated to the edge layer and the cloud layer. This can reasonably store and process the data according to the access characteristics and change frequency of the data, thereby improving the accuracy of data stratification. By judging the local-remote access difference and the data change frequency, the qualified industrial data is allocated to the edge layer or the cloud layer for storage. The edge layer industrial data is stored locally, which reduces the amount of data transmission and the burden of cloud storage, and improves the efficiency and real-time performance of data processing. The cloud layer industrial data can be stored and analyzed using cloud computing resources, which facilitates remote access and sharing of data. The preset access difference is dynamically adjusted according to the empty set judgment result data. According to whether the edge layer industrial data and the cloud layer industrial data are empty sets, the size of the access difference is adjusted, and the hierarchical judgment is re-performed. This can dynamically adjust the data stratification strategy according to the actual situation, and further optimize the efficiency of data storage and transmission.

[0066] Preferably, step S248 includes the following steps:

[0067] Step S2481: if the empty set judgment result data shows that the edge layer industrial data is an empty set, the preset access difference is adjusted downward according to the preset threshold fine-tuning gradient to obtain the adjusted access difference;

[0068] Specifically, when the empty set judgment result shows that the edge layer industrial data is an empty set, it is necessary to adjust the preset access difference downward according to the preset threshold fine-tuning gradient. The specific operation steps are: set a threshold fine-tuning gradient, for example, 5 times. Use the data processing tool to read the current preset access difference value, and subtract the threshold fine-tuning gradient from it to obtain the new adjusted access difference. For example, the current preset access difference is 50 times, and the threshold fine-tuning gradient is 5 times, then the adjusted access difference is 45 times.

[0069] Step S2482: If the empty set judgment result data shows that the cloud layer industrial data is an empty set, the preset access difference is adjusted upward according to the preset threshold fine-tuning gradient to obtain the adjusted access difference;

[0070] Specifically, when the empty set judgment result shows that the cloud-layer industrial data is an empty set, it is necessary to adjust the preset access difference upward according to the preset threshold fine-tuning gradient. The specific operation steps are: set a threshold fine-tuning gradient, for example 5 times. Use a data processing tool to read the current preset access difference value, and add it to the threshold fine-tuning gradient to obtain a new adjusted access difference. For example, the current preset access difference is 50 times, and the threshold fine-tuning gradient is 5 times, then the adjusted access difference is 55 times.

[0071] Step S2483: Return the adjusted access difference to step S242 until both the edge layer industrial data and the cloud layer industrial data are not empty sets.

[0072] Specifically, the new adjusted access difference value obtained in step S2481 or S2482 is used as the new preset access difference, and the operations of steps S242 to S247 are performed again to reclassify and judge the industrial data.

[0073] The present invention adjusts the size of the access difference according to whether the edge layer industrial data and the cloud layer industrial data are empty sets, and re-performs the tiering judgment. If the edge layer industrial data is an empty set, it means that the idle capacity of the edge layer is too large, and the gradient can be fine-tuned to adjust the preset access difference downward. If the cloud layer industrial data is an empty set, it means that the idle capacity of the cloud is too large, and the gradient can be fine-tuned to adjust the preset access difference upward. This can dynamically adjust the data tiering strategy according to actual conditions, and further optimize the efficiency of data storage and transmission.

[0074] Preferably, step S25 comprises the following steps:

[0075] Step S251: Obtain historical edge layer access log data;

[0076] Specifically, historical edge layer access log data is obtained through the log management tool provided by the edge computing platform or by directly extracting data from log files. These log data record the access history of edge layer industrial data, including access time, visitor ID, access device ID, and access data type.

[0077] Step S252: Analyze the change pattern of edge layer industrial data according to historical edge layer access log data to obtain change frequency distribution data and change operation device ID data;

[0078] Specifically, the access records of edge layer industrial data are screened out from the historical edge layer access log data, and the change operation information, such as data modification and deletion, is extracted. Then, these change operations are counted and analyzed using data processing tools (such as the Pandas library in Python) to obtain the change frequency distribution data, that is, the change frequency distribution of each industrial data. At the same time, the change operation device ID data is extracted, that is, the unique identifier of the device that performs the change operation.

[0079] Step S253: extracting high-change data from the edge layer industrial data according to the change frequency distribution data to obtain high-change data, and removing the high-change data from the edge layer industrial data to obtain low-change data;

[0080] Specifically, according to a pre-set threshold (e.g., industrial data with a change frequency exceeding a certain number of times is identified as high-change data), industrial data with a change frequency higher than the threshold is screened out and marked as high-change data. Then, these high-change data are removed from the edge layer industrial data to obtain low-change data. Use data processing tools for automated processing, such as writing Python scripts, to screen and mark according to the set threshold, and then delete the high-change data from the data set.

[0081] Step S254: Acquire edge layer network topology data;

[0082] Specifically, the edge layer network topology data is obtained through network management tools (such as SNMP), APIs provided by the edge computing platform, or manual configuration. The edge layer network topology data describes the connection relationship and topology structure between various devices in the edge computing environment, including the physical connection between devices, network topology information, etc. For example, a network scanning tool (such as Nmap) can be used to scan the edge network to obtain the IP address, MAC address and network topology information of the device, and store these data in the database.

[0083] Step S255: Identify the device ID of the edge layer industrial data to obtain the current device ID data;

[0084] Specifically, for example, you can add a device ID field to the tag or metadata of industrial data, or record the source device ID of each data access request through the data access log. This will get the current device ID data.

[0085] Step S256: selecting a local storage location for the high-change data according to the current device ID data and the edge layer network topology data, and obtaining high-change data storage location data;

[0086] Specifically, according to the current device ID data, the corresponding device node is found in the edge layer network topology data, and the storage device information of the node, such as a flash storage device, is obtained.

[0087] Step S257: selecting an edge storage location for the low-change data according to the current device ID data and the edge layer network topology data, and obtaining storage location data for the low-change data;

[0088] Specifically, according to the current device ID data, the corresponding device node is found in the edge layer network topology data, and the storage device information of the node, such as a solid-state storage device, is obtained.

[0089] Step S258: transferring the high change data to the flash storage device corresponding to the high change data storage location data for storage, and transferring the low change data to the solid-state storage device corresponding to the low change data storage location data for storage.

[0090] Specifically, a network transmission protocol (such as SFTP, SCP) or a data transmission tool (such as rsync, FTP) is used to transfer high-change data from the current storage device to the flash storage device. A network transmission protocol (such as SFTP, SCP) or a data transmission tool (such as rsync, FTP) is used to transfer low-change data from the current storage device to the solid-state storage device.

[0091] The present invention can obtain the change frequency distribution data and the change operation device ID data of the edge layer industrial data by acquiring and analyzing the historical edge layer access log data. This can understand the change frequency of different data and the characteristics of the change operation device, and provide a basis for subsequent data processing and storage location selection. By extracting high-change data from the edge layer industrial data according to the change frequency distribution data. High-change data usually has a higher change frequency and needs to be stored and processed more frequently. After the high-change data is extracted, the subsequent storage location selection and transmission can be carried out in a targeted manner, thereby improving the efficiency of data processing. By acquiring the edge layer network topology data and the current device ID data, the storage location of high-change data and low-change data is selected according to the device ID and network topology information. According to the location of the current device and the network topology structure, the high-change data is stored in a local flash storage device to achieve fast storage and access. The low-change data can be stored on the edge storage device to reduce the burden of data transmission and cloud storage. The high-change data is transferred to the flash storage device corresponding to the high-change data storage location for storage, and the low-change data is transferred to the solid-state storage device corresponding to the low-change data storage location for storage. This can optimize data storage and transmission and improve the efficiency and real-time performance of data processing based on the frequency of data changes and the choice of storage location.

[0092] Preferably, step S3 comprises the following steps:

[0093] Step S31: Acquire key indicator requirements for the industrial site to obtain key indicator requirement data;

[0094] Specifically, communicate with the management and technical staff of the industrial site to understand the specific needs and key performance indicators (KPIs) of the site. For example, indicators in terms of production efficiency, equipment operating status, product quality, and energy consumption. Then, record these needs in detail through on-site surveys, questionnaires, or expert interviews. Use demand management tools (such as Jira, Trello) to organize and track these needs and generate key indicator demand data documents. This document should contain detailed information such as the definition, calculation method, required data type, and data source of each key indicator.

[0095] Step S32: Tracing back the data source of the key indicator requirement data to obtain the required data source data;

[0096] Specifically, analyze the key indicator demand data to determine the data type and specific data items required for each indicator. Then, find the storage location of these data items in the existing industrial data system. Data management tools (such as DataCatalogs) or data lineage analysis tools (such as Apache Atlas) can be used to trace back the data source. This can trace the source of each data item and understand the data generation process and storage path. Finally, record the detailed information of the required data source data, including but not limited to the name of the data source, storage location, and access method.

[0097] Step S33: extract source data from the cloud-layer industrial data according to the required data source data to obtain edge-layer processed data, and remove the edge-layer processed data from the cloud-layer industrial data to obtain cloud-layer processed data.

[0098] Specifically, data extraction tools (such as ETL tools) are used according to the required data source data. For example, Python scripts are used to extract the required data from the cloud layer industrial data through API or database query to obtain edge layer processing data. Then, the extracted edge layer processing data is stored on the edge computing device, and this data is removed from the original data set to obtain cloud layer processing data.

[0099] The present invention obtains the key indicator requirements of the industrial site. This can determine the important indicators that the industrial data processing system needs to pay attention to. The required data source is determined by tracing back the data source of the key indicator requirement data. This can trace the source and path of the data, understand the data generation and collection process, and provide guidance and support for subsequent data extraction and processing. By extracting the source data of the cloud layer industrial data according to the required data source data. This can extract data that meets the key indicator requirements from the cloud layer industrial data to form edge layer processing data. The edge layer processing data has high relevance and importance and can be used for real-time monitoring, control and decision-making applications. The cloud layer processing data is obtained by removing the edge layer processing data from the cloud layer industrial data. This can separate the cloud layer processing data from the edge layer processing data, and perform different processing and storage respectively. The edge layer processing data can be analyzed and responded to locally in real time, reducing the delay of data transmission and cloud computing. The cloud layer processing data can use cloud computing resources for large-scale analysis and mining to achieve more complex data processing and applications.

[0100] Preferably, step S4 comprises the following steps:

[0101] Step S41: constructing an event stream for edge layer processing data to obtain an industrial event data stream;

[0102] Specifically, for example, using Kafka, you can configure a data stream producer to publish edge layer processing data to Kafka topics and build an industrial event data stream. Specifically, you can publish data items one by one to the message queue in the event format, and each message represents an industrial event. The event data stream can be continuously generated from the edge layer processing data.

[0103] Step S42: using a complex event processing engine preset in the edge computing device to perform real-time complex event pattern matching on the industrial event data stream to obtain matching event data;

[0104] Specifically, deploy a CEP engine, such as Esper or Apache Flink CEP, on the edge computing device. Then, define event pattern rules that describe complex event patterns that need to be detected, such as equipment failures and abnormal temperature changes. Configure these rules into the CEP engine so that it can analyze industrial event data streams in real time and match qualifying events. For example, you can set a rule to generate matching event data when the reading of a sensor exceeds a predetermined threshold. The CEP engine processes the event data stream in real time according to these rules and outputs matching event data that meets the conditions.

[0105] Step S43: using the complex event processing engine to perform event response processing on the matching event data to obtain monitoring analysis data;

[0106] Specifically, define the response processing logic, which includes but is not limited to alarm triggering, data recording, and control instruction sending. Configure these response logics into the CEP engine so that it is automatically executed after matching the event pattern. For example, when an equipment failure event is detected, the system can be configured to immediately trigger an alarm, notify relevant personnel and record the failure information, or send a control instruction to stop the equipment from running.

[0107] Step S44: Perform real-time monitoring of the industrial network at the industrial site to obtain real-time communication network status data.

[0108] Specifically, deploy network monitoring tools such as Nagios, Zabbix or Wireshark on key network nodes at industrial sites. These tools can collect network communication status information in real time, including but not limited to data packet transmission information, queue length, transmission delay, bandwidth utilization, latency, and packet loss rate in the current network. Configure monitoring tools to report network status data periodically or in real time. For example, configure network monitoring tools to collect network data once a minute and trigger alarms when abnormalities are found.

[0109] The present invention constructs an event stream by processing data at the edge layer. This can convert data into the form of an event stream and organize the data in chronological order. By using a complex event processing engine preset in an edge computing device, real-time complex event pattern matching is performed on the industrial event data stream. This can detect and match complex events in the event data stream in real time according to predefined event patterns, and capture key events related to the monitoring target. By using a complex event processing engine, event response processing is performed on the matching event data. This can perform specific response actions such as generating alarms, triggering control operations, etc. according to the matched events. Through real-time event response processing, abnormal situations can be discovered in time, corresponding measures can be taken, and the safety and efficiency of industrial systems can be improved. By performing real-time monitoring of industrial networks at industrial sites, real-time communication network status data can be obtained. This can monitor the connection status, bandwidth utilization, and delay indicators of industrial networks, and timely discover network failures or congestion to ensure reliable transmission and real-time performance of data.

[0110] Preferably, step S5 comprises the following steps:

[0111] Step S51: performing queue retrieval on the real-time communication network status data to obtain the current transmission queue;

[0112] Specifically, real-time communication network status data, including but not limited to data packet transmission information, queue length, and transmission delay in the current network. Use a queue management system or software (such as RabbitMQ, Kafka, etc.) to search the transmission queue and extract the data packet queue being transmitted in the current network. By analyzing this data, the specific situation of the current transmission queue can be identified, including the transmission status and queue position of each data packet, thereby obtaining the information of the current transmission queue.

[0113] Step S52: performing alarm detection on the monitoring and analysis data to obtain alarm detection data;

[0114] Specifically, complex event processing engines (such as Esper, Apache Flink CEP) are used to process monitoring and analysis data, and alarm detection rules are set. These rules can determine whether to trigger an alarm based on predefined alarm conditions (such as high device temperature and abnormal traffic). Then, these rules are run to detect real-time monitoring and analysis data. If an event that meets the alarm conditions is detected, alarm detection data is generated and detailed information of these alarm events is recorded, including but not limited to the alarm type, trigger time, and related data packets.

[0115] Step S53: extracting labels from the current transmission queue to obtain a data packet label set, wherein the data packet label set includes a plurality of data packet labels;

[0116] Specifically, use a network traffic analysis tool (such as Wireshark) or a custom label extraction program to analyze and mark each data packet in the current transmission queue one by one. Then, summarize the extracted label information to form a data packet label set. The data packet label set should contain the label information of all data packets in the transmission queue.

[0117] Step S54: if there is an alarm tag in the data packet tag set, then detect whether the alarm data packet corresponding to the alarm tag is at the head of the current transmission queue; if the alarm data packet is not at the head of the queue, then adjust the alarm data packet to the head of the queue to obtain an adjusted transmission queue;

[0118] Specifically, scan the data packet label set to find out whether there is an alarm label. The alarm label corresponds to the alarm data packet detected in step S52. Then, check the position of the data packet corresponding to the alarm label in the current transmission queue. If the alarm data packet is not at the head of the queue, the queue order needs to be adjusted. Use the queue operation function of the queue management tool (such as RabbitMQ, Kafka) to adjust the alarm data packet to the head of the queue to ensure its priority transmission.

[0119] Step S55: if the alarm data packet is at the head of the queue, there is no need to adjust the sequence of the current transmission queue, and the current transmission queue is recorded as the adjustment transmission queue;

[0120] Specifically, check whether the head position of the current transmission queue is an alarm data packet. If the alarm data packet is already at the head of the queue, it is confirmed that the current transmission queue does not need further adjustment. The state of the current transmission queue is recorded as adjusting the transmission queue, and prepare for subsequent operations.

[0121] Step S56: Encapsulating the cloud layer processing data and the monitoring and analysis data into data packets respectively to obtain a cloud layer data packet and a monitoring data packet;

[0122] Specifically, a standard data encapsulation protocol (such as TCP / IP) is used to encapsulate cloud layer processing data and monitoring analysis data into packets, respectively, to generate cloud layer packets and monitoring packets. The encapsulation process includes adding necessary metadata to each packet, such as packet type, packet length, source identifier, target identifier, and alarm label.

[0123] Step S57: if the alarm detection data indicates that an alarm exists, the monitoring data packet is placed in the second position of the adjustment transmission queue, and the cloud layer data packet is placed in the tail of the adjustment transmission queue;

[0124] Specifically, check the alarm detection data. If the detection data shows an alarm event, the transmission queue needs to be prioritized. Insert the monitoring data packet into the second position of the adjusted transmission queue to ensure that it is immediately after the alarm data packet. This ensures that urgent monitoring data is transmitted quickly. Then, put the cloud layer data packet at the end of the adjusted transmission queue to ensure that normal data is transmitted after the urgent data is processed.

[0125] Step S58: If the alarm detection data shows no alarm, the monitoring data packet is placed in the second to last position of the adjustment transmission queue, and the cloud layer data packet is placed in the tail of the adjustment transmission queue.

[0126] Specifically, the alarm detection data is checked. If the detection data shows that there is no alarm event at present, the data packets are sorted according to the normal priority. The monitoring data packet is inserted into the second to last position of the adjustment transmission queue so that it is transmitted before all normal data packets but after the alarm data packet. Then, the cloud layer data packet is placed at the end of the adjustment transmission queue.

[0127] The present invention obtains the current transmission queue by performing queue retrieval on the real-time communication network status data. This can obtain the queue situation of the current data transmission in real time, and understand the queuing and transmission status of the data packet. By performing alarm detection on the monitoring and analysis data, the alarm detection data is obtained. This can detect whether there are abnormal conditions or alarm signals in the monitoring data according to the preset alarm rules and models, and timely discover and handle potential problems. By performing label extraction on the current transmission queue, a data packet label set is obtained. This can extract the label information of each data packet from the transmission queue, including the source, destination, priority, and alarm index of the data packet. By adjusting the alarm data packet, it is placed at the top of the transmission queue or kept in place. This can ensure that the alarm data packet is transmitted first, and the relevant personnel or system are notified in time to respond and process, thereby improving the abnormal detection and processing capabilities of the system. By performing data packet encapsulation and sequencing on the cloud layer processing data and the monitoring and analysis data. According to the alarm detection results and priorities, the monitoring data packet and the cloud layer data packet are placed in the appropriate position in the adjusted transmission queue according to the predetermined rules. This can optimize the transmission order of the data and ensure the real-time and priority transmission of key data.

[0128] Preferably, the present invention further provides an industrial networking-based data processing system for executing the industrial networking-based data processing method as described above, the industrial networking-based data processing system comprising:

[0129] The data acquisition module is used to collect industrial data on the industrial site to obtain the original industrial data;

[0130] The data tiered storage module is used to tier the original industrial data to obtain edge layer industrial data and cloud layer industrial data; and store the edge layer industrial data;

[0131] The cloud data splitting module is used to split the cloud layer industrial data into industrial data to obtain edge layer processing data and cloud layer processing data;

[0132] The complex event processing module is used to perform complex event processing on edge layer processing data to obtain monitoring and analysis data; perform real-time monitoring of industrial networks at industrial sites to obtain real-time communication network status data;

[0133] The cloud storage module is used to send and store cloud layer processing data and monitoring analysis data to the cloud based on real-time communication network status data.

[0134] In the present invention, the data acquisition module is used to collect on-site industrial data on the industrial site to obtain the original industrial data. This can reduce the time and labor cost of manual collection and improve the efficiency and accuracy of data collection. The original industrial data is processed in layers by the data layered storage module to obtain edge layer industrial data and cloud layer industrial data, and the edge layer industrial data is stored. This can reduce the amount of data transmission to the cloud, reduce the consumption of network bandwidth and data transmission delay, thereby improving the efficiency and real-time performance of data processing. The cloud layer industrial data is split by the cloud data splitting module to obtain edge layer processing data and cloud layer processing data, and only the data to be processed is sent to the cloud for storage. This can avoid the transmission and storage of a large amount of data that does not need to be processed, optimize the utilization of resources, and reduce the waste of resources. The complex event processing module is used to process the edge layer processing data to obtain monitoring and analysis data. At the same time, the industrial network is monitored in real time on the industrial site to obtain real-time communication network status data. This can realize real-time monitoring and analysis of the production process and timely discover abnormal situations and potential problems. The cloud storage module sends cloud-layer processing data and monitoring and analysis data to the cloud and stores them, ensuring the security and reliability of the data. Cloud storage can provide backup and disaster recovery capabilities, prevent data loss and damage, and facilitate remote access and sharing of data. In summary, the present invention achieves high efficiency in data processing, optimizes resource utilization, improves real-time monitoring and analysis capabilities, and ensures data security and reliability, thereby effectively supporting the needs of industrial networking data processing.

[0135] Preferably, a computer-readable storage medium stores a data processing method based on industrial networking as described above which can be loaded and executed by a processor.

[0136] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0137] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A data processing method based on industrial networking, characterized in that: The following steps are involved: Step S1: collecting on-site industrial data at the industrial site to obtain original industrial data; Step S2: stratify the original industrial data to obtain edge layer industrial data and cloud layer industrial data; The edge layer industrial data is stored; wherein step S2 includes the following steps: Step S21: Acquire a collection device metadata list, wherein the collection device metadata list includes a number of device metadata entries, and the entries include a device ID, a data type identifier, and a data security value; Step S22: performing preliminary data classification on the original industrial data according to the data type identifier based on the collection device metadata list to obtain classified industrial data, wherein the classified industrial data includes industrial production data, equipment status data, product quality data, environmental data, human resources data, supply chain data and energy consumption data; Step S23: Obtain historical classified industrial data; perform data access pattern analysis on each type of industrial data in the historical classified industrial data based on a preset statistical time window to obtain a data access statistical table, wherein the attributes of the data access statistical table include a preset statistical time window, a data type identifier, a number of local accesses, a frequency of local data changes, a number of remote accesses, and a frequency of remote data changes; Step S24: stratify the classified industrial data based on the data access statistics table to obtain edge layer industrial data and cloud layer industrial data; wherein step S24 includes the following steps: Step S241: traverse the classified industrial data based on the data access statistics table, subtract the number of local accesses from the number of remote accesses for each type of industrial data, and obtain the local-remote access difference; Step S242: if the local-foreign access difference is a positive value, and the local-foreign access difference is greater than or equal to the preset access difference, the corresponding class industrial data is recorded as edge layer industrial data; Step S243: If the local-foreign access difference is a positive value, and the local-foreign access difference is less than the preset access difference, then proceed to step S246; Step S244: if the local-remote access difference is a negative value, and the absolute value of the local-remote access difference is greater than or equal to the preset access difference, the corresponding class industrial data is recorded as cloud-layer industrial data; Step S245: if the local-foreign access difference is a negative value, and the absolute value of the local-foreign access difference is less than the preset access difference, then proceed to step S246; Step S246: Compare and judge the corresponding local data change frequency with the remote data change frequency. If the local data change frequency is greater than or equal to the remote data change frequency, the corresponding quasi-industrial data is recorded as edge layer industrial data. If the local data change frequency is less than the remote data change frequency, the corresponding quasi-industrial data is recorded as cloud layer industrial data. Step S247: when the classified industrial data is completely traversed, that is, the edge layer industrial data and the cloud layer industrial data are obtained, an empty set judgment is performed on the edge layer industrial data and the cloud layer industrial data to obtain empty set judgment result data; Step S248: adjusting the preset access difference according to the empty set judgment result data to obtain the adjusted access difference, and returning the adjusted access difference to step S242 until both the edge layer industrial data and the cloud layer industrial data are not empty sets; Step S25: storing edge layer industrial data; Step S3: splitting the cloud layer industrial data into industrial data to obtain edge layer processing data and cloud layer processing data; wherein step S3 includes the following steps: Step S31: Acquire key indicator requirements for the industrial site to obtain key indicator requirement data; Step S32: Tracing back the data source of the key indicator requirement data to obtain the required data source data; Step S33: extracting source data from the cloud layer industrial data according to the required data source data to obtain edge layer processing data, and removing the edge layer processing data from the cloud layer industrial data to obtain cloud layer processing data; Step S4: Perform complex event processing on the edge layer processing data to obtain monitoring and analysis data; perform real-time monitoring of the industrial network on the industrial site to obtain real-time communication network status data; wherein step S4 includes the following steps: Step S41: constructing an event stream for edge layer processing data to obtain an industrial event data stream; Step S42: using a complex event processing engine preset in the edge computing device to perform real-time complex event pattern matching on the industrial event data stream to obtain matching event data; Step S43: using the complex event processing engine to perform event response processing on the matching event data to obtain monitoring analysis data; Step S44: performing real-time monitoring of the industrial network at the industrial site to obtain real-time communication network status data; Step S5: sending the cloud layer processing data and monitoring analysis data to the cloud and storing them based on the real-time communication network status data; wherein step S5 includes the following steps: Step S51: performing queue retrieval on the real-time communication network status data to obtain the current transmission queue; Step S52: performing alarm detection on the monitoring and analysis data to obtain alarm detection data; Step S53: extracting labels from the current transmission queue to obtain a data packet label set, wherein the data packet label set includes a plurality of data packet labels; Step S54: if there is an alarm tag in the data packet tag set, then detect whether the alarm data packet corresponding to the alarm tag is at the head of the current transmission queue; if the alarm data packet is not at the head of the queue, then adjust the alarm data packet to the head of the queue to obtain an adjusted transmission queue; Step S55: if the alarm data packet is at the head of the queue, there is no need to adjust the sequence of the current transmission queue, and the current transmission queue is recorded as the adjustment transmission queue; Step S56: Encapsulating the cloud layer processing data and the monitoring and analysis data into data packets respectively to obtain a cloud layer data packet and a monitoring data packet; Step S57: if the alarm detection data indicates that an alarm exists, the monitoring data packet is placed in the second position of the adjustment transmission queue, and the cloud layer data packet is placed in the tail of the adjustment transmission queue; Step S58: If the alarm detection data shows no alarm, the monitoring data packet is placed in the second to last position of the adjustment transmission queue, and the cloud layer data packet is placed in the tail of the adjustment transmission queue.

2. The method for processing data based on industrial networking according to claim 1, characterized in that: Step S248 includes the following steps: Step S2481: if the empty set judgment result data shows that the edge layer industrial data is an empty set, the preset access difference is adjusted downward according to the preset threshold fine-tuning gradient to obtain the adjusted access difference; Step S2482: If the empty set judgment result data shows that the cloud layer industrial data is an empty set, the preset access difference is adjusted upward according to the preset threshold fine-tuning gradient to obtain the adjusted access difference; Step S2483: Return the adjusted access difference to step S242 until both the edge layer industrial data and the cloud layer industrial data are not empty sets.

3. The method for processing data based on industrial networking according to claim 1 is characterized in that: Step S25 includes the following steps: Step S251: Obtain historical edge layer access log data; Step S252: Analyze the change pattern of edge layer industrial data according to historical edge layer access log data to obtain change frequency distribution data and change operation device ID data; Step S253: extracting high-change data from the edge layer industrial data according to the change frequency distribution data to obtain high-change data, and removing the high-change data from the edge layer industrial data to obtain low-change data; Step S254: Acquire edge layer network topology data; Step S255: Identify the device ID of the edge layer industrial data to obtain the current device ID data; Step S256: selecting a local storage location for the high-change data according to the current device ID data and the edge layer network topology data, and obtaining high-change data storage location data; Step S257: selecting an edge storage location for the low-change data according to the current device ID data and the edge layer network topology data, and obtaining storage location data for the low-change data; Step S258: transferring the high change data to the flash storage device corresponding to the high change data storage location data for storage, and transferring the low change data to the solid-state storage device corresponding to the low change data storage location data for storage.

4. A data processing system based on industrial networking, characterized in that: Used to execute the industrial networking-based data processing method as claimed in claim 1, the industrial networking-based data processing system comprises: The data acquisition module is used to collect industrial data on the industrial site to obtain the original industrial data; The data tiered storage module is used to tier the original industrial data to obtain edge layer industrial data and cloud layer industrial data; and store the edge layer industrial data; The cloud data splitting module is used to split the cloud layer industrial data into industrial data to obtain edge layer processing data and cloud layer processing data; The complex event processing module is used to perform complex event processing on edge layer processing data to obtain monitoring and analysis data; perform real-time monitoring of industrial networks at industrial sites to obtain real-time communication network status data; The cloud storage module is used to send and store cloud layer processing data and monitoring analysis data to the cloud based on real-time communication network status data.

5. A computer-readable storage medium storing a program that can be loaded by a processor and executed by a method for processing data based on industrial networking as claimed in any one of claims 1 to 3.

Images