A data processing system for industrial network operation

Abstract

The present invention discloses a data processing system for industrial network operation, comprising a data acquisition module, a data processing module and a node management module, characterized in that: the data acquisition module is used to acquire business data in the industrial network and enter the data into the system; the data processing module is used to remove redundant and incomplete data in the data to ensure that the processed data is correct data; the node management module is used to adjust the allocation of processing nodes and adjust the load rate of the node processing data according to the processed data capacity; the data acquisition module, the data processing module and the node management module are electrically connected to each other; the data acquisition module comprises a monitoring module, a data collection module and a data entry module; the monitoring module is used to respectively monitor the data pulled from the system and the data processed by the processing node. The present invention has the characteristics of improving processing efficiency and improving resource utilization.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing system for industrial network operation. Background Art

[0002] With the rapid development of Internet technology, the number of factories using the industrial Internet has increased rapidly, generating a large amount of data that needs to be processed and analyzed. The data generated by the system needs to be processed in a timely manner. At present, most industrial Internets include two parts: business systems and analysis systems. The business system is mainly responsible for processing the business data of the enterprise, and the analysis system is mainly responsible for analyzing the relevant data of the enterprise and extracting the value of the data. However, due to the different data formats in different application scenarios, the system is prone to data loading errors or data loss during data processing. It is necessary to interrupt the data loading task and correct the erroneous data fragments. If there are multiple erroneous data, the correction node must be executed repeatedly each time, which may cause the data correction order to be disordered. When faced with a large amount of data that needs to be corrected, the correction node will be congested, resulting in low system processing efficiency. Moreover, when processing data interaction among multiple clients, the existing technology is that the server processes the data based on the received client access request, distributes the data to the corresponding client, and renders the data on the client. The rendering is based on the number of client requests. This will result in the same data being rendered multiple times in the same scenario, wasting a lot of computing resources and occupying a large number of data processing nodes. Therefore, it is very necessary to design a data processing system for industrial network operation that improves processing efficiency and resource utilization. Summary of the Invention

[0003] The object of the present invention is to provide a data processing system for industrial network operation to solve the problems raised in the above background technology.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions: a data processing system for industrial network operation, comprising a data acquisition module, a data processing module, and a node management module, characterized in that: the data acquisition module is used to collect business data in the industrial network and enter the data into the system; the data processing module is used to remove redundant and incomplete data from the data to ensure that the processed data is correct; the node management module is used to adjust the allocation of processing nodes and adjust the load rate of the node processing data according to the processed data capacity; the data acquisition module, the data processing module, and the node management module are electrically connected to each other;

[0005] The node cluster management module includes a node adjustment submodule, a node replacement submodule and a thread management submodule. The node allocation submodule is used to allocate data correction nodes, the node replacement submodule is used to replace correction nodes that do not meet the conditions, and the thread management submodule is used to manage data correction threads.

[0006] According to the above technical solution, the data acquisition module includes a monitoring module, a data collection module and a data entry module. The monitoring module is used to monitor the data pulled from the system and the data processed by the processing node respectively. The data collection module is used to periodically pull system data through the industrial network port. The data entry module is used for operators to enter external data into the system.

[0007] According to the above technical solution, the data processing module includes a data preprocessing module, which is used to obtain data and eliminate redundant or incomplete data.

[0008] According to the above technical solution, the node management module includes a performance adjustment module and a node cluster management module. The node monitoring module is used to monitor the load rate of the processing node, and the node cluster management module is used to manage the processing node cluster.

[0009] According to the above technical solution, the data processing module also includes a data review module and a data correction module. The data review module is used to review whether there are errors in the transmitted data, and the data correction module is used to complete the incomplete data and correct the erroneous data.

[0010] According to the above technical solution, the operation method of the data processing system mainly includes the following steps:

[0011] Step S1: Using the data collection module, periodically pull the operating data of the enterprise system through the enterprise system port; using the data entry module, enter the paper data, supplementary data and user access requests into the system; using the monitoring module, monitor and retrieve the data before and after processing;

[0012] Step S2: When the network port is connected to retrieve or input data, the system sends an electrical signal to trigger the data protection module to start and begin analyzing the data in the transmission line;

[0013] Step S3: When data inconsistency occurs, the data correction module is activated to analyze the specific location of the data error and replace the erroneous data;

[0014] Step S4: After the data is transmitted to the server, the system starts the data analysis module and begins to analyze the client's data request through the edge server;

[0015] Step S5: After anchoring the rendering data corresponding to the client request, the edge server builds a virtual transmission chain to connect the current client and the target client, and transmits the rendering data to the current client through the virtual transmission chain for response.

[0016] According to the above technical solution, step S2 further includes the following steps:

[0017] Step S21: Retrieve the target system data port, scan the data transmission protocol of the target system data port, identify the data type received by the transmission protocol, establish a transmission chain and connect to the data port according to the data type, and the system periodically pulls system data through the data port;

[0018] Step S22: Retrieve the data pulled through the data interface, scan and identify the pulled data, eliminate redundant data in the pulled data, mark the incomplete data, retrieve the marked data and compare it with the original data in the target system collected by the monitoring module. If the data similarity under any attribute is greater than the threshold, the original data with the similarity greater than the threshold is retrieved; otherwise, the current data is deleted;

[0019] Step S23: Retrieve the data in the target system collected by the monitoring module, and the system compares the data in the processing system with the data in the target system one by one. If there is a difference between the current data and the data in the target system, it means that there is an error in the current data, and the current data is marked as error data and the difference part is marked. If the data in the target system does not exist in the processing system, it means that the data in the processing system is missing, and the data is marked as missing data and the data location is marked.

[0020] According to the above technical solution, step S3 further includes the following steps:

[0021] Step S31: retrieve the operating status and data throughput speed of each processing node in the current node cluster, retrieve the amount of data to be corrected at the current marked position, and calculate the time required to correct the data according to the formula Where i = 1, 2, 3...n, T represents the time required for the current correction node to correct the data, M represents the amount of data that needs to be corrected in the current area, represents the average data throughput speed of the current correction node, α represents the influence coefficient of the data transmission delay of the current correction node, sorts the time in ascending order, and marks the priority of the correction node from top to bottom;

[0022] Step S32: retrieve the time for each correction node to correct the data, and compare it with the time for the system to process the data. If there is a correction node among the current correction nodes whose correction data time is less than the system processing time, mark all correction nodes whose time is less than the system processing time, and sort them in ascending order. Select the correction node with the shortest time for processing. Otherwise, control the correction node to perform multi-threaded data correction.

[0023] According to the above technical solution, step S32 further includes the following steps:

[0024] Step S321: retrieve the actual throughput and total throughput of the correction data of the current correction node, and calculate the efficiency of the current correction node through the formula Where S represents the load efficiency of the current correction node, w represents the actual throughput of the current correction node, and W represents the total throughput of the current correction node. If the load efficiency of the current node is less than the threshold, it means that the correction node has entered the fatigue period and the node that does not need to be corrected is called to replace the current node. Otherwise, the current node continues to correct data.

[0025] Step S322: When retrieving multi-threaded correction data, the system transfers the processing node to the standby database. The standby correction node divides the database according to the location of the data to be corrected and encapsulates it. The standby correction node identifies missing or erroneous data in the blocked data block, retrieves the original data collected by the monitoring module to correct the erroneous data, retrieves the current modification thread, identifies the correction position of the current multi-threads, and compares the correction positions. If the data correction positions overlap, the overlapping threads will be merged. Otherwise, the system continues to perform resource locking modifications.

[0026] According to the above technical solution, step S4 further includes the following steps:

[0027] Step S41: Retrieve the client's data request, scan the data request, identify the data request characteristics, and retrieve the rendering data stored by the current client's rendering node based on the data request characteristics. If the current client's rendering node does not have the rendering data for the current request, anchor the rendering data stored by the client rendering node near the current client. Otherwise, directly retrieve the rendering data of the rendering node in the current client for response.

[0028] Step S42: Retrieve the location of the client that retains the rendering data, and calculate the distance between the current client and the client that retains the rendering data using the formula Where (X1, Y1) represents the current client's location coordinates, and (X2, Y2) represents the location coordinates of the client that retains the rendering data. If the distance L is greater than the threshold set by the system, it means that the transmission time of the edge server's retained rendering data is greater than the local rendering time, and local rendering is automatically performed. Otherwise, the retained rendering data is retrieved through the transmission node, and the request is responded to based on the retained rendering data.

[0029] Compared with the prior art, the present invention has the following beneficial effects: by eliminating redundant data in the system, the present invention can reduce the impact of redundant data on processing results and improve result accuracy; by comparing data in the processing system with data in the target system, incomplete data can be completed; by comparing data in the processing system with data in the target system, the location of erroneous data can be quickly marked and then quickly corrected, further improving the speed of data processing; the above formula can quickly obtain the time for each node to process data in the current area, and further quickly allocate the optimal correction node; by calculating the load efficiency of the current correction node and replacing the backup node when the load efficiency is less than a threshold, the optimal correction speed can be maintained at all times, the correction time can be reduced, and the efficiency of the system can be greatly improved; by partitioning and encapsulating the database, the correction thread can accurately lock the location of the data to be corrected when locking the resource area where the data to be corrected is located, preventing other threads from being unable to modify the data after the current modification thread locks the resource, further improving the efficiency of data correction; by retrieving the rendering data retained by the current client or the current client rendering node to respond, the computing resources occupied by repeated rendering of data requests on the current client rendering node can be reduced, the time required for rendering can be reduced, and the response efficiency of the system can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0031] Figure 1 It is a schematic diagram of the system module composition of the present invention. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0033] See also Figure 1The present invention provides a technical solution: a data processing system for industrial network operation, comprising a data acquisition module, a data processing module, and a node management module, characterized in that: the data acquisition module is used to collect business data in the industrial network and enter the data into the system; the data processing module is used to remove redundant and incomplete data from the data to ensure that the processed data is correct; the node management module is used to adjust the allocation of processing nodes and adjust the load rate of the node processing data according to the processed data capacity; the data acquisition module, the data processing module, and the node management module are electrically connected to each other;

[0034] The node cluster management module includes a node adjustment submodule, a node replacement submodule and a thread management submodule. The node allocation submodule is used to allocate data correction nodes, the node replacement submodule is used to replace correction nodes that do not meet the conditions, and the thread management submodule is used to manage data correction threads.

[0035] The data acquisition module includes a monitoring module, a data collection module and a data entry module. The monitoring module is used to monitor the data pulled from the system and the data processed by the processing node respectively. The data collection module is used to periodically pull system data through the industrial network port. The data entry module is used for operators to enter external data into the system.

[0036] The data processing module includes a data preprocessing module, which is used to obtain data and eliminate redundant or incomplete data.

[0037] The node management module includes a performance adjustment module and a node cluster management module. The node monitoring module is used to monitor the load rate of the processing node, and the node cluster management module is used to manage the processing node cluster.

[0038] The data processing module also includes a data review module and a data correction module. The data review module is used to review whether there are errors in the transmitted data, and the data correction module is used to complete the incomplete data and correct the erroneous data.

[0039] The operation method of the data processing system mainly includes the following steps:

[0040] Step S1: Using the data collection module, periodically pull the operating data of the enterprise system through the enterprise system port; using the data entry module, enter the paper data, supplementary data and user access requests into the system; using the monitoring module, monitor and retrieve the data before and after processing;

[0041] Step S2: When the network port is connected to retrieve or input data, the system sends an electrical signal to trigger the data protection module to start and begin analyzing the data in the transmission line;

[0042] Step S3: When data inconsistency occurs, the data correction module is activated to analyze the specific location of the data error and replace the erroneous data;

[0043] Step S4: After the data is transmitted to the server, the system starts the data analysis module and begins to analyze the client's data request through the edge server;

[0044] Step S5: After anchoring the rendering data corresponding to the client request, the edge server builds a virtual transmission chain to connect the current client and the target client, and transmits the rendering data to the current client through the virtual transmission chain for response.

[0045] Step S2 further includes the following steps:

[0046] Step S21: Retrieve the target system data port, scan the data transmission protocol of the target system data port, identify the data type received by the transmission protocol, establish a transmission chain and connect to the data port according to the data type, and the system periodically pulls system data through the data port;

[0047] Step S22: Retrieve the data pulled through the data interface, the system scans and identifies the pulled data, eliminates redundant data in the pulled data, and marks the incomplete data, retrieves the marked data and compares it with the original data in the target system collected by the monitoring module. If the data similarity under any attribute is greater than the threshold, it means that the currently marked missing data exists in the original data, and the original data with similarity greater than the threshold is retrieved. Otherwise, the current data is deleted. By eliminating redundant data in the system, the impact of redundant data on the processing results can be reduced, and the accuracy of the results can be improved. By comparing the data in the processing system with the data in the target system, the incomplete data can be completed.

[0048] Step S23: Retrieve the data in the target system collected by the monitoring module, and the system compares the data in the processing system with the data in the target system one by one. If there is a difference between the current data and the data in the target system, it means that there is an error in the current data, and the current data is marked as error data and the difference part is marked. If the data in the target system does not exist in the processing system, it means that the data in the processing system is missing, and the data is marked as missing data and the data location is marked. By comparing the data in the processing system with the data in the target system, the location of the error data can be quickly marked, and then quickly corrected, further improving the speed of data processing.

[0049] Step S3 further includes the following steps:

[0050] Step S31: retrieve the operating status and data throughput speed of each processing node in the current node cluster, retrieve the amount of data to be corrected at the current marked position, and calculate the time required to correct the data according to the formula Where i = 1, 2, 3...n, T represents the time required for the current correction node to correct the data, M represents the amount of data that needs to be corrected in the current area, represents the average data throughput speed of the current correction node, α represents the influence coefficient of the data transmission delay of the current correction node, and the time is sorted in ascending order. The priority of the correction nodes is marked from top to bottom. The above formula can quickly obtain the time for each node to process the current area data, and further quickly allocate the best correction node;

[0051] Step S32: retrieve the time for each correction node to correct the data, and compare it with the time for the system to process the data. If there is a correction node among the current correction nodes whose correction time is less than the system processing time, it means that the current correction node can correct the data within the time for the system to process the data. Mark all correction nodes whose time is less than the system processing time and sort them in ascending order. Select the correction node with the shortest time for processing. Otherwise, it means that data congestion will occur in the process of the correction node correcting the data, and control the correction node to perform multi-threaded data correction.

[0052] Step S32 further includes the following steps:

[0053] Step S321: retrieve the actual throughput and total throughput of the correction data of the current correction node, and calculate the efficiency of the current correction node through the formula Where S represents the load efficiency of the current correction node, w represents the actual throughput of the current correction node, and W represents the total throughput of the current correction node. If the load efficiency of the current node is less than the threshold, it means that the correction node has entered the fatigue period and a non-correction node is called to replace the current node. Otherwise, the current node continues to correct the data. By calculating the load efficiency of the current correction node and replacing the standby node when the load efficiency is less than the threshold, the optimal correction speed can be maintained at all times, the correction time can be reduced, and the efficiency of the system can be greatly improved.

[0054] Step S322: When retrieving multi-threaded correction data, the system transfers the processing node to the standby database. The standby correction node divides the database according to the location of the data to be corrected and encapsulates it. The standby correction node identifies missing or erroneous data in the blocked data block, retrieves the original data collected by the monitoring module to correct the erroneous data, retrieves the current modification thread, identifies the correction position of the current multi-threads, and compares the correction positions. If the data correction positions overlap, it means that one of the threads will lock the resources during the modification process, and the other threads will wait for resources, causing thread congestion. The overlapping threads will be merged, otherwise the system will continue to perform resource lock modification. By dividing and encapsulating the database, the correction thread can accurately lock the position to be corrected when locking the resource area where the data to be corrected is located, preventing other threads from being unable to modify the data after the current modification thread locks the resources, thereby further improving the efficiency of data correction.

[0055] Step S4 further includes the following steps:

[0056] Step S41: Retrieve the client's data request, scan the data request, identify the data request characteristics, and retrieve the rendering data retained by the current client's rendering node based on the data request characteristics. If the current client's rendering node does not have the rendering data for the current request, anchor the rendering data retained by the client's rendering node near the current client. Otherwise, directly retrieve the rendering data retained by the rendering node in the current client for response. By retrieving the rendering data retained by the current client or the current client's rendering node for response, the computing resources occupied by repeated rendering of data requests at the current client's rendering node can be reduced, the rendering time required can be reduced, and the system response efficiency can be improved.

[0057] Step S42: Retrieve the location of the client that retains the rendering data, and calculate the distance between the current client and the client that retains the rendering data using the formula Where (X1, Y1) represents the current client's location coordinates, and (X2, Y2) represents the location coordinates of the client that retains the rendering data. If the distance L is greater than the threshold set by the system, it means that the transmission time of the edge server retaining the rendering data is longer than the local rendering time, and local rendering is automatically performed. Otherwise, the retained rendering data is retrieved through the transmission node, and the request is responded to based on the retained rendering data. By calculating the distance between the two client rendering nodes, the server with the shorter transmission distance and shorter rendering time that retains the rendering data can be quickly selected.

[0058] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0059] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A data processing system for industrial network operation, comprising a data acquisition module, a data processing module, and a node management module, characterized in that: The data acquisition module is used to collect business data in the industrial network and enter the data into the system. The data processing module is used to remove redundant and incomplete data from the data to ensure that the processed data is correct. The node management module is used to adjust the allocation of processing nodes and adjust the load rate of the node processing data according to the processed data capacity. The data acquisition module, data processing module and node management module are electrically connected to each other; The node management module includes a performance adjustment module and a node cluster management module. The node monitoring module is used to monitor the load rate of the processing node, and the node cluster management module is used to manage the processing node cluster; The node cluster management module includes a node adjustment submodule, a node replacement submodule and a thread management submodule. The node allocation submodule is used to allocate data correction nodes, the node replacement submodule is used to replace correction nodes that do not meet the conditions, and the thread management submodule is used to manage data correction threads. The data acquisition module includes a monitoring module, a data collection module and a data entry module. The monitoring module is used to monitor the data pulled from the system and the data processed by the processing node respectively. The data collection module is used to periodically pull system data through the industrial network port. The data entry module is used for operators to enter external data into the system; The data processing module also includes a data review module and a data correction module. The data review module is used to review whether there are errors in the transmitted data, and the data correction module is used to complete the incomplete data and correct the erroneous data. The operation method of the data processing system mainly includes the following steps: Step S1: Using the data collection module, periodically pull the operating data of the enterprise system through the enterprise system port; using the data entry module, enter the paper data, supplementary data and user access requests into the system; using the monitoring module, monitor and retrieve the data before and after processing; Step S2: When the network port is connected to retrieve or input data, the system sends an electrical signal to trigger the data protection module to start and begin analyzing the data in the transmission line; Step S3: When data inconsistency occurs, the data correction module is activated to analyze the specific location of the data error and replace the erroneous data; Step S4: After the data is transmitted to the server, the system starts the data analysis module and begins to analyze the client's data request through the edge server; Step S5: After anchoring the rendering data corresponding to the client request, the edge server builds a virtual transmission chain to connect the current client and the target client, and transmits the rendering data to the current client via the virtual transmission chain for response; The step S2 further comprises the following steps: Step S21: Retrieve the target system data port, scan the data transmission protocol of the target system data port, identify the data type received by the transmission protocol, establish a transmission chain and connect to the data port according to the data type, and the system periodically pulls system data through the data port; Step S22: Retrieve the data pulled through the data interface, scan and identify the pulled data, eliminate redundant data in the pulled data, mark the incomplete data, retrieve the marked data and compare it with the original data in the target system collected by the monitoring module. If the data similarity under any attribute is greater than the threshold, the original data with the similarity greater than the threshold is retrieved; otherwise, the current data is deleted; Step S23: Retrieve the data in the target system collected by the monitoring module, and compare the data in the processing system with the data in the target system one by one. If there is a difference between the current data and the data in the target system, it means that the current data is wrong, and the current data is marked as wrong data and the difference is marked. If the data in the target system does not exist in the processing system, it means that the current data in the processing system is missing, and the current data is marked as missing data and the data location is marked. The step S3 further comprises the following steps: Step S31: retrieve the operating status and data throughput speed of each processing node in the current node cluster, retrieve the amount of data to be corrected at the current marked position, and calculate the time required to correct the data according to the formula , where , Indicates the time required for the current correction node to correct the data. Indicates the amount of data that needs to be corrected in the current area. Indicates the average data throughput speed of the current correction node. Indicates the impact coefficient of the data transmission delay of the current correction node. Sort the time in ascending order and mark the priority of the correction node from top to bottom; Step S32: Retrieve the data correction time of each correction node and compare it with the system data processing time. If there is a correction node whose data correction time is less than the system data processing time among the current correction nodes, mark all the correction nodes whose time is less than the system processing time and sort them in ascending order. Select the correction node with the shortest time for processing. Otherwise, control the correction node to perform multi-threaded data correction. The step S32 further includes the following steps: Step S321: retrieve the actual throughput and total throughput of the correction data of the current correction node, and calculate the efficiency of the current correction node through the formula , where Indicates the load efficiency of the current correction node, w indicates the actual throughput of the current correction node, Indicates the total throughput of the current correction node. If the load efficiency of the current node is less than the threshold, it means that the correction node has entered the fatigue period and a node that does not need correction is called to replace the current node. Otherwise, the current node continues to correct data. Step S322: When retrieving multi-threaded correction data, the system transfers the processing node to the standby database. The standby correction node divides the database according to the location of the data to be corrected and encapsulates it. The standby correction node identifies missing or erroneous data in the blocked data block, retrieves the original data collected by the monitoring module to correct the erroneous data, retrieves the current modification thread, identifies the correction position of the current multi-threads, and compares the correction positions. If the data correction positions overlap, the overlapping threads will be merged. Otherwise, the system continues to perform resource locking modifications.

2. A data processing system for industrial network operation according to claim 1, characterized in that: The data processing module includes a data pre-processing module, and the data pre-processing module is used to obtain data and eliminate redundant or incomplete data.

3. A data processing system for industrial network operation according to claim 2, characterized in that: The step S4 further comprises the following steps: Step S41: Retrieve the client's data request, scan the data request, identify the data request characteristics, and retrieve the rendering data stored by the current client's rendering node based on the data request characteristics. If the current client's rendering node does not have the rendering data for the current request, anchor the rendering data stored by the client rendering node near the current client. Otherwise, directly retrieve the rendering data of the rendering node in the current client for response. Step S42: Retrieve the location of the client that retains the rendering data, and calculate the distance between the current client and the client that retains the rendering data using the formula , where Indicates the current client's location coordinates. Indicates that the location coordinates of the rendering data client are retained. If the distance If it is greater than the threshold set by the system, it means that the transmission time of the edge server retaining the rendering data is greater than the local rendering time, and local rendering is automatically performed. Otherwise, the retained rendering data is retrieved through the transmission node, and the request is responded to according to the retained rendering data.

Images