A data processing method and system for an industrial big data platform

Abstract

The application discloses a data processing method and system for an industrial big data platform, relates to data processing technology, and is used for improving the problem of important data processing delay caused by the lengthening of overall processing time when multiple data are processed simultaneously, and comprises the following steps: recognizing the data source type in the industrial big data platform, importing a data classification database to obtain corresponding data partitions, detecting the transmission path of each data partition to calculate an average delay coefficient, combining a load index to evaluate the processing priority of each data partition by using an analytic hierarchy process, judging whether distributed collaborative processing is needed for a high-load partition, collecting the working frequency and network throughput of a data node, calculating a collaborative score based on a nonlinear regression algorithm, and determining the overall performance state of the platform.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a data processing method and system for an industrial big data platform. Background Art

[0002] Data processing technology refers to the methods and technologies for transforming, analyzing, and managing collected data. Data management technology is applied to industrial big data platforms to analyze the execution of various data and make more efficient decisions.

[0003] The prior art has the following deficiencies:

[0004] When there is a lot of data to be processed on the industrial big data platform, multiple data will be processed at the same time. In this case, the overall data processing time will be prolonged, causing some important data to not be processed in time, thereby increasing the data processing risk in the big data platform. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a data processing method and system for an industrial big data platform. After partitioning the data, the load conditions and priority processing levels of each zone are calculated, and high-load partitions are screened out to determine whether distributed collaborative processing is performed to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A data processing method for an industrial big data platform comprises the following steps:

[0008] Step S1: Identify the data source in the industrial big data platform, determine the data source type, and import the data source type into the data classification database to obtain multiple data partitions corresponding to the data source;

[0009] Step S2: Detect the data transmission paths of different data partitions, obtain the transmission delay of each data partition and calculate the average delay coefficient, set the data sampling range, collect the load index of each data partition based on the data sampling range, and use the hierarchical analysis method to calculate the processing priority of each data partition based on the average delay coefficient and load index;

[0010] Step S3: Filter high-load partitions based on the processing priority of each data partition and determine whether to perform distributed collaborative processing on the industrial big data platform. When entering distributed collaborative processing, collect the operating frequency and network throughput of the data nodes in the high-load partitions;

[0011] Step S4: The working frequency of the data node in the high-load partition and the network throughput are analyzed by a nonlinear regression algorithm to obtain a collaborative score, and the overall performance state of the industrial big data platform is determined according to the collaborative score.

[0012] In a preferred embodiment, in step S1, when a new access or updated data source is detected, the structured meta-information of the data source is extracted, and the structured meta-information is combined into an identification feature vector;

[0013] The identification feature vector is calculated with the Euclidean distance of each identification feature vector in the preset data source type library to obtain a similarity value.

[0014] In a preferred embodiment, in step S1, if the similarity value is less than or equal to the preset similarity threshold, it is determined that the matching is successful, and the corresponding data source type identifier is output;

[0015] If the similarity value is greater than the preset similarity threshold, it is determined that the matching fails, the identification feature vector is added to the preset data source type library as a feature vector of a new data source type, a new data source type entry is generated, and a unique data source type identifier is assigned;

[0016] The data source type identifier is input into the data classification database to retrieve one or more data partitions corresponding to the data source.

[0017] In a preferred embodiment, in step S1, if the similarity value is less than or equal to the preset similarity threshold, it is determined that the matching is successful, and the corresponding data source type identifier is output;

[0018] If the similarity value is greater than the preset similarity threshold, it is determined that the matching fails, the identification feature vector is added to the preset data source type library as a feature vector of a new data source type, a new data source type entry is generated, and a unique data source type identifier is assigned;

[0019] The data source type identifier is input into the data classification database to retrieve one or more data partitions corresponding to the data source.

[0020] In a preferred embodiment, in step S1, if the similarity value is less than or equal to the preset similarity threshold, it is determined that the matching is successful, and the corresponding data source type identifier is output;

[0021] If the similarity value is greater than the preset similarity threshold, it is determined that the matching fails, the identification feature vector is added to the preset data source type library as a feature vector of a new data source type, a new data source type entry is generated, and a unique data source type identifier is assigned;

[0022] The data source type identifier is input into the data classification database to retrieve one or more data partitions corresponding to the data source.

[0023] In a preferred embodiment, in step S2, a data flow routing path of each data partition is detected, where the data flow routing path is one or more data nodes that the data passes through during transmission from the source node to the target node;

[0024] Divide the preset collection time into multiple sampling moments, send a probe packet to the data flow routing path at each sampling moment, and record the round-trip time from sending to receiving the probe packet as the delay sampling value of the current sampling moment;

[0025] The delayed sampling values ​​at each sampling moment are combined to form a delayed sampling sequence of the corresponding data partition.

[0026] In a preferred embodiment, in step S2, a delay sampling threshold is set by the MAD algorithm. If the delay sampling value is greater than the delay sampling threshold, it is marked as an abnormal delay value; otherwise, it is marked as a normal delay value.

[0027] Only the normal delay values ​​are retained and merged into a new delay sampling sequence;

[0028] Calculate the average value of the new delayed sample sequence as the average delay coefficient;

[0029] Based on the preset sampling time, the sampling time is divided into sections to collect the load indicators of each data node in the data partition, including CPU usage, memory usage, and disk speed;

[0030] Calculate the average load index of each data node in the data partition as the comprehensive load index of the data partition;

[0031] Based on the average delay coefficient and comprehensive load indicators, the hierarchical analysis method is used to calculate the priority score of data partitions.

[0032] In a preferred embodiment, in step S3, the average of the priority scores of each data partition is used as the priority threshold. If the priority score is greater than the priority threshold, the corresponding data partition is marked as a high-load partition; and the high-load partitions are merged into a high-load partition set.

[0033] Preset load indicator threshold set, including CPU utilization threshold, memory usage threshold, and disk speed threshold;

[0034] If any comprehensive load indicator is greater than the corresponding threshold, distributed collaborative processing is performed; otherwise, distributed collaborative processing is not performed.

[0035] In a preferred embodiment, in step S3, after determining to perform distributed collaborative processing, the total number of processing tasks completed by the data nodes in each high-load partition, the data sending rate, and the receiving rate are collected within a preset collection period;

[0036] The working frequency is the total number of tasks completed by the data node in unit time;

[0037] Network throughput is the data sending and receiving rate of a data node per unit time;

[0038] The working frequency and network throughput of each data node are averaged and used as the total working frequency and total network throughput of the load partition.

[0039] In a preferred embodiment, in step S4, a preliminary collaboration score is calculated by weighting in combination with the total operating frequency and the total network throughput;

[0040] The preliminary collaboration score comprehensively considers the total working frequency and the total network throughput to obtain the predicted collaboration score of each high-load partition by constructing a polynomial regression model.

[0041] In a preferred embodiment, in step S4, a first scoring threshold and a second scoring threshold are preset, and the first scoring threshold is greater than the second scoring threshold, and the predicted collaborative score is divided into states:

[0042] If the predicted collaboration score is greater than the first scoring threshold, the overall performance status of the industrial big data platform is judged to be in an efficient operation state;

[0043] If the predicted collaboration score is less than or equal to the first scoring threshold and greater than the second scoring threshold, the overall performance status of the industrial big data platform is judged to be in a normal operating state;

[0044] If the predicted collaboration score is less than or equal to the second score threshold, the overall performance status of the industrial big data platform is judged to be an inefficient operating state.

[0045] A data processing system for an industrial big data platform, used to implement the above-mentioned data processing method for an industrial big data platform, including a data source identification and partitioning module, a priority evaluation module, a collaborative scheduling module, and a performance evaluation module;

[0046] The data source identification and partitioning module is used to identify the data source in the industrial big data platform, determine the type of data source, and obtain the corresponding data partition;

[0047] The priority evaluation module is used to detect the data transmission path of the data partition, collect delay and load indicators, and calculate the processing priority based on the hierarchical analysis method;

[0048] The collaborative scheduling module is used to screen high-load partitions, determine whether to perform distributed collaborative processing, and collect the working frequency and network throughput of the data nodes in the high-load partitions;

[0049] The performance evaluation module uses a nonlinear regression algorithm to analyze the collaborative score based on the working frequency and network throughput of the data nodes, and evaluates the overall performance status of the industrial big data platform accordingly.

[0050] The technical effects and advantages of the data processing method and system for industrial big data platform of the present invention are as follows:

[0051] The present invention identifies the data source type in the industrial big data platform, imports the data classification database to obtain the corresponding data partition, detects the transmission path of each data partition to calculate the average delay coefficient, and uses the hierarchical analysis method in combination with the load index to evaluate the processing priority of each data partition. The calculated processing priority is used to determine the category of each zone for candidate judgment and provide a data basis for screening partitions. For high-load partitions, it determines whether distributed collaborative processing is required, collects the working frequency and network throughput of data nodes, calculates the collaborative score based on the nonlinear regression algorithm, and determines the overall performance status of the platform, thereby improving the response efficiency and resource scheduling capabilities of the industrial big data platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of a data processing method for an industrial big data platform according to the present invention.

[0053] Figure 2 This is a flow chart of a data processing system for an industrial big data platform according to the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] The present invention identifies the data source type in the industrial big data platform, imports the data classification database to obtain the corresponding data partition, detects the transmission path of each data partition to calculate the average delay coefficient, and uses the hierarchical analysis method in combination with the load index to evaluate the processing priority of each data partition. For high-load partitions, it determines whether distributed collaborative processing is required, collects the working frequency and network throughput of the data nodes, calculates the collaborative score based on the nonlinear regression algorithm, and determines the overall performance status of the platform, thereby improving the response efficiency and resource scheduling capabilities of the industrial big data platform.

[0056] Embodiment 1, a data processing method and system for an industrial big data platform, as shown, comprising the following steps: Figure 1

[0057] Step S1: identifying the data source in the industrial big data platform, determining the data source type and importing the data source type into the data classification database to obtain a plurality of data partitions corresponding to the data source;

[0058] Step S2: detecting the data transmission path of different data partitions, obtaining the transmission delay of each data partition and calculating the average delay coefficient, setting the data sampling range, collecting the load index of each data partition according to the data sampling range, and comprehensively calculating the processing priority of each data partition using the analytic hierarchy process according to the average delay coefficient and the load index;

[0059] Step S3: filtering the high-load partition according to the processing priority of each data partition and judging whether to perform distributed collaborative processing on the industrial big data platform, when entering the distributed collaborative processing, collecting the working frequency and network throughput of the data nodes in the high-load partition;

[0060] Step S4: analyzing the collaborative score of the working frequency and network throughput of the data nodes in the high-load partition using a nonlinear regression algorithm, and determining the overall performance state of the industrial big data platform according to the collaborative score.

[0061] The specific implementation is as follows:

[0062] In step S1, when the system detects that a new data source is accessed or the information of an existing data source is updated, the structured meta-information of the data source is extracted, including the communication protocol type, data format, data upload frequency and data identification code, which are combined into a recognition feature vector;

[0063] The recognition feature vector is transmitted to the preset data source type library to match the data source type, and the Euclidean distance between the recognition feature vector and the recognition feature vector in the preset data source type library is calculated as a similarity value;

[0064] If the similarity value is less than or equal to the preset similarity threshold, it is judged that the matching is successful, and the corresponding data source type identifier is output;

[0065] If the similarity value is greater than the preset similarity threshold, it is judged that the matching fails, the recognition feature vector is taken as the recognition feature vector of a new data source type, added to the preset data source type library, a new data source type entry is generated, and a unique data source type identifier is assigned;

[0066] The data source type identifier is input into the data classification database to retrieve one or more data partitions corresponding to the data source, and the data partition includes partition identifier, sampling frequency, and structured information of data flow routing path.​

[0067] By identifying the data source type and obtaining the corresponding data partition, orderly hierarchical management of platform data can be achieved, providing a structural basis for partition scheduling and priority setting, and ensuring that subsequent operations are performed on clear data partition dimensions.

[0068] It should be noted that the data classification database is a database system used to record and manage structured information such as the identification type, partition structure metadata, scheduling priority, etc. of various data sources in the industrial big data platform; data partition refers to an independent data processing unit formed by logical division according to the data source type in the industrial big data platform. Each data partition corresponds to a set of data streams with similar structure or purpose; the preset data source type library refers to a set of data structures pre-built and continuously updated by professionals, which is used to store feature vectors of different data source types and their corresponding type identification information; the method for setting the preset similarity threshold is not unique, and the specific setting is performed by professionals, which will not be elaborated here.

[0069] In step S2, the data transmission path of each data partition is detected. The data flow routing path is one or more data nodes that the data in the data partition passes through during the process of being transmitted from the source node to the target node;

[0070] The sampling time is divided into sampling moments within the preset collection time. At each sampling moment, a probe packet is sent to the data flow routing path to detect the transmission performance. The complete round-trip time from sending to receiving is recorded as the delay sampling value of the current sampling moment. The delay sampling values ​​between each sampling moment are combined into a delay sampling sequence recorded as: ,in, is the delayed sampling value obtained by the j-th sampling, where j takes values ​​of 1, 2, 3, etc. is a delayed sampling sequence; the delayed sampling sequence reflects the data transmission delay characteristics within the preset acquisition time;

[0071] Use the MAD algorithm to set the delay sampling threshold and identify abnormal delay values ​​in the delay sampling sequence. The specific steps are as follows:

[0072] Calculate the median: Get the median of the delayed sampling sequence and record it as d;

[0073] Calculate the absolute deviation: Calculate the absolute deviation of each delayed sampling value from the median, and take the absolute value as the absolute deviation, recorded as ;

[0074] Calculate the median absolute deviation: Find the median of all absolute deviations to get the median absolute deviation: ,in, is the median absolute deviation;

[0075] Set the threshold: The sum of the median and the absolute deviation of the median in the delayed sampling sequence is used as the delayed sampling threshold: ,in, is the delayed sampling threshold;

[0076] If the delay sampling value is greater than the delay sampling threshold, it is marked as an abnormal delay value, otherwise it is marked as a normal delay value;

[0077] Only normal delay values ​​are retained to avoid interference of extreme cases on the overall results, and normal delay values ​​are merged into a new delay sampling sequence;

[0078] Calculate the average value of the new delay sampling sequence as the average delay coefficient, which reflects the comprehensive characteristics of the data partition transmission status;

[0079] The smaller the average delay coefficient, the higher the transmission efficiency of the corresponding data flow routing path, the lower the transmission delay, and the lower its processing priority. The larger the average delay coefficient, the lower the transmission efficiency of the corresponding data flow routing path, the higher the transmission delay, and the higher its processing priority.

[0080] Load indicators are performance parameters of the data partition's operating status and resource usage. Based on the preset sampling time, the sampling moments are divided and the load indicators of each data node in the data partition are collected, including CPU usage, memory usage, and disk speed.

[0081] The CPU usage rate is calculated by reading the CPU usage statistics of the data node and calculating the proportion of CPU resources occupied within the preset sampling time range;

[0082] Memory usage is used to monitor the memory usage of data nodes and obtain the ratio of used memory to total memory;

[0083] The disk rate is the number of bytes read and written and the number of operations on the disk during the statistical collection time;

[0084] The load index of each data node in the data partition is averaged and used as the comprehensive load index of the corresponding data partition, reflecting the overall resource usage status of the data partition during the sampling time;

[0085] The larger the comprehensive load index, the higher the resource utilization rate of the data partition, and computing and network resources need to be allocated first, and its priority is higher; conversely, the lower the comprehensive load index, the more sufficient the data partition resources are, and the lower the processing priority.

[0086] Based on the average delay coefficient and comprehensive load indicators, the hierarchical analysis method is used to comprehensively calculate the processing priority of data partitions, and the hierarchical analysis model is constructed as follows:

[0087] Input data: Evaluation indicators are used as input data, including average delay coefficient, CPU usage, memory usage, and disk rate;

[0088] Constructing a judgment matrix: Compare the evaluation indicators of the criterion layer pairwise. Relevant technical personnel assign different scale values ​​according to the importance of the evaluation indicators to form a judgment matrix: ,in, is the importance of the i-th evaluation index relative to the j-th evaluation index, is a judgment matrix; the conditions satisfied by the judgment matrix are: ;

[0089] For example, you can set the importance of scale values ​​as follows:

[0090] Scale value Index i vs. index j 1 Equally important 3 Slightly important 5 Obviously important 7 Strongly important 9 Extremely important 2,4,6,8 The median value of the above judgment

[0091] For example, the judgment matrix constructed according to the importance of evaluation indicators is as follows:

[0092] Average delay factor CPU usage Memory usage Disk speed Average delay factor 1 1 / 3 1 / 5 1 / 7 CPU usage 3 1 1 / 2 1 / 3 Memory usage 5 2 1 1 / 2 Disk speed 7 3 2 1

[0093] It should be explained that the setting of the scale value is adjusted according to the actual situation. The importance of the evaluation index can be determined by analyzing the correlation between the changes in the evaluation index and the system performance results based on historical operation data. The specific setting is done by professionals and will not be elaborated here.

[0094] Calculate the weight vector: Calculate the sum of the elements in each column of the judgment matrix: , where n is the number of elements in the corresponding column, is the column sum of the j-th column in the judgment matrix;

[0095] Divide each element in the judgment matrix by the column sum of the column to obtain the normalized element: ,in, is the normalized matrix element of the i-th evaluation index relative to the j-th evaluation index;

[0096] The normalized matrix elements are matched one by one with the elements at the corresponding positions in the judgment matrix to form a normalized matrix, and the average value of each row in the normalized matrix is ​​calculated as the weight value of the corresponding evaluation index;

[0097] Output priority score: Calculate the priority score by the weight value and the normalized value of the corresponding evaluation index: ,in, is the normalized value of the i-th evaluation index in the k-th data partition, is the weight value of the corresponding evaluation index, Score the priority of the kth data partition;

[0098] It should be noted that the analytic hierarchy process is a mathematical method used for multi-criteria decision analysis. It constructs a hierarchical structure of the problem and compares the elements between each layer to determine the relative importance of each factor in the overall decision.

[0099] In step S3, the average of the priority scores of each data partition is used as the priority threshold. If the priority score is greater than the priority threshold, the corresponding data partition is marked as a high-load partition; and the high-load partitions are merged into a high-load partition set.

[0100] A set of load indicator thresholds is preset, including CPU utilization threshold, memory usage threshold, and disk speed threshold. The comprehensive load indicators of the high-load partition are retrieved. If any comprehensive load indicator is greater than the corresponding threshold, distributed collaborative processing is performed; otherwise, distributed collaborative processing is not performed.

[0101] After determining to perform distributed collaborative processing, the total number of processing tasks completed by the data nodes in each high-load partition, the data sending rate, and the receiving rate are collected within a preset collection period;

[0102] The working frequency of a data node is the total number of tasks completed per unit time: ,in, is the total number of processing tasks completed within the sampling period, T is the acquisition period, is the operating frequency;

[0103] The network throughput is the data sending and receiving rate per unit time. The network throughput is: ,in, is the total amount of data received during the collection period, is the total amount of data sent during the collection period, is the network throughput;

[0104] The average working frequency of each data node is recorded as the total working frequency of the high-load partition , the average network throughput of each data node is recorded as the total network throughput of the high-load partition .

[0105] By screening high-load partitions based on priority scores and determining whether to execute distributed collaborative processing through priority thresholds, dynamic identification and regulation of resource pressure states can be achieved. The working frequency and network throughput of data nodes in high-load partitions are further collected to provide basic data support for subsequent collaborative scoring and platform performance evaluation.

[0106] It should be noted that the preset load indicator threshold set is used to judge the comprehensive load indicator of the data partition. The historical release of each comprehensive load indicator can be counted, and its mean and standard deviation can be calculated as the corresponding threshold. The setting method is not unique and will not be elaborated here.

[0107] In step S4, the preliminary cooperation score is calculated by combining the total operating frequency and the total network throughput: ,in, is the total network throughput, is the total operating frequency, a and b are preset coefficients, N is the total number of high-load partitions in the high-load partition set, is the preliminary collaboration score for the i-th high-load partition;

[0108] It should be explained that the preset coefficient is to balance the relative importance of task processing capability and network performance in the collaborative capability evaluation, and is used to construct the preliminary collaborative score of the high-load partition. The specific setting is done by professionals and will not be elaborated here; the preliminary collaborative score of each high-load partition is input into the polynomial regression model as a fitting target to further learn the nonlinear mapping relationship between task processing capability and network performance under different operating conditions, thereby realizing the refined modeling of collaborative capability and dynamic evaluation of performance status.

[0109] The total operating frequency and total network throughput are combined to construct a polynomial regression model to analyze and predict the collaborative score, reflecting the overall effectiveness of the collaborative operation between data nodes. The polynomial regression model is constructed as follows:

[0110] Data preparation: Normalize the total operating frequency and total network throughput and merge the feature vectors. The feature vectors of each high-load partition are used as input variables: ,in, is the normalized value of the total operating frequency of the i-th high-load partition, is the normalized value of the total network throughput of the i-th high-load partition, is the preliminary collaboration score for the i-th high-load partition;

[0111] Construct a polynomial regression function: ,in, is the regression coefficient, The predicted collaboration score for the i-th high-load partition;

[0112] Construct the regression matrix as :

[0113] ;

[0114] Combine the predicted co-scores for the high-load partitions corresponding to each row in the regression matrix into a target vector: ,in, is a target vector;

[0115] The least square method is used to solve the regression coefficient: the regression coefficient vector is obtained by the normal equation: wherein, is the transpose of the regression matrix, is the regression coefficient vector;

[0116] The predicted synergy score is obtained by substituting the regression coefficient vector into the polynomial regression function.

[0117] The first score threshold and the second score threshold are preset, and the first score threshold is greater than the second score threshold. The predicted synergy score is divided into states:

[0118] If the predicted synergy score is greater than the first score threshold, it is judged that the overall performance state of the industrial big data platform is a high-efficiency running state.

[0119] If the predicted synergy score is less than or equal to the first score threshold and greater than the second score threshold, it is judged that the overall performance state of the industrial big data platform is a normal running state.

[0120] If the predicted synergy score is less than or equal to the second score threshold, it is judged that the overall performance state of the industrial big data platform is a low-efficiency running state.

[0121] When it is judged to be a high-efficiency running state, it means that the synergy ability between the data nodes in the high-load partition is good, the matching degree of the working frequency and the network throughput is high, and the resource scheduling mechanism is stably running. When it is judged to be a normal running state, it means that some data partitions in the high-load partition have lower synergy efficiency. When it is judged to be a low-efficiency running state, it means that the synergy efficiency of the data nodes in the high-load partition decreases, which affects the overall processing capacity of the industrial big data platform.

[0122] The working frequency and the network throughput of each data node in the high-load partition are collected, a synergy score model is established by using a nonlinear regression algorithm, the overall synergy efficiency index of the partition is calculated, and the platform performance state is divided in combination with the preset score threshold, so as to realize the dynamic discrimination of the running state of the industrial big data platform.

[0123] It should be noted that the polynomial regression model is a statistical learning method for fitting the nonlinear relationship between the input variable and the output variable based on the polynomial function form.

[0124] Embodiment 2, based on a data processing system for an industrial big data platform, as shown in Figure 2 The data processing system for the industrial big data platform includes a data source identification partition module, a priority evaluation module, a synergy scheduling module, and a performance evaluation module.

[0125] The data source identification and partitioning module is used to identify the data source in the industrial big data platform, determine the type of data source, and obtain the corresponding data partition;

[0126] The priority evaluation module is used to detect the data transmission path of the data partition, collect delay and load indicators, and calculate the processing priority based on the hierarchical analysis method;

[0127] The collaborative scheduling module is used to screen high-load partitions, determine whether to perform distributed collaborative processing, and collect the working frequency and network throughput of the data nodes in the high-load partitions;

[0128] The performance evaluation module uses a nonlinear regression algorithm to analyze the collaborative score based on the working frequency and network throughput of the data nodes, and evaluates the overall performance status of the industrial big data platform accordingly.

[0129] The above embodiments may be implemented in whole or in part through software, hardware, firmware or any other combination. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0130] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application of the technical solution and the invention constraints. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0132] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

[0133] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing method for an industrial big data platform, characterized in that: The following steps are involved: Step S1: Identify the data source in the industrial big data platform, determine the data source type, and import the data source type into the data classification database to obtain multiple data partitions corresponding to the data source; Step S2: Detect the data transmission paths of different data partitions, obtain the transmission delay of each data partition and calculate the average delay coefficient, set the data sampling range, collect the load index of each data partition based on the data sampling range, and use the hierarchical analysis method to calculate the processing priority of each data partition based on the average delay coefficient and load index; Step S3: Filter high-load partitions based on the processing priority of each data partition and determine whether to perform distributed collaborative processing on the industrial big data platform. When entering distributed collaborative processing, collect the operating frequency and network throughput of the data nodes in the high-load partitions; Step S4: The working frequency of the data nodes in the high-load partition and the network throughput are analyzed using a nonlinear regression algorithm to obtain a collaborative score, and the overall performance status of the industrial big data platform is determined based on the collaborative score.

2. A data processing method for an industrial big data platform according to claim 1, characterized in that: In step S1, when a newly connected or updated data source is detected, the structured meta-information of the data source is extracted and the structured meta-information is combined into an identification feature vector; The Euclidean distance between the recognition feature vector and each recognition feature vector in the preset data source type library is calculated to obtain a similarity value.

3. The data processing method for an industrial big data platform according to claim 2, characterized in that: In step S1, if the similarity value is less than or equal to the preset similarity threshold, the match is determined to be successful and the corresponding data source type identifier is output; If the similarity value is greater than the preset similarity threshold, the match is judged to have failed, and the identification feature vector is used as the feature vector of the new data source type, added to the preset data source type library, and a new data source type entry is generated, and a unique data source type identifier is assigned; The data source type identifier is input into the data classification database, and one or more data partitions corresponding to the data source are retrieved.

4. The data processing method for an industrial big data platform according to claim 3, characterized in that: In step S2, the data flow routing path of each data partition is detected. The data flow routing path is one or more data nodes that the data passes through during the process of being transmitted from the source node to the target node. Divide the preset collection time into multiple sampling moments, send a probe packet to the data flow routing path at each sampling moment, and record the round-trip time from sending to receiving the probe packet as the delay sampling value of the current sampling moment; The delayed sampling values ​​at each sampling moment are combined to form a delayed sampling sequence of the corresponding data partition.

5. The data processing method for an industrial big data platform according to claim 4, characterized in that: In step S2, a delay sampling threshold is set by the MAD algorithm. If the delay sampling value is greater than the delay sampling threshold, it is marked as an abnormal delay value; otherwise, it is marked as a normal delay value. Only the normal delay values ​​are retained and merged into a new delay sampling sequence; Calculate the average value of the new delayed sample sequence as the average delay coefficient; Based on the preset sampling time, the sampling time is divided into sections to collect the load indicators of each data node in the data partition, including CPU usage, memory usage, and disk speed; Calculate the average load index of each data node in the data partition as the comprehensive load index of the data partition; Based on the average delay coefficient and comprehensive load indicators, the hierarchical analysis method is used to calculate the priority score of data partitions.

6. The data processing method for an industrial big data platform according to claim 5, characterized in that: In step S3, the average of the priority scores of each data partition is used as the priority threshold. If the priority score is greater than the priority threshold, the corresponding data partition is marked as a high-load partition; and the high-load partitions are merged into a high-load partition set. Preset load indicator threshold set, including CPU utilization threshold, memory usage threshold, and disk speed threshold; If any comprehensive load indicator is greater than the corresponding threshold, distributed collaborative processing is performed; Otherwise, no distributed collaborative processing is performed.

7. The data processing method for an industrial big data platform according to claim 5, characterized in that: In step S3, after determining to perform distributed collaborative processing, the total number of processing tasks completed by the data nodes in each high-load partition, the data sending rate and the receiving rate are collected within a preset collection period; The working frequency is the total number of tasks completed by the data node in unit time; Network throughput is the data sending and receiving rate of a data node per unit time; The working frequency and network throughput of each data node are averaged and used as the total working frequency and total network throughput of the load partition.

8. The data processing method for an industrial big data platform according to claim 7, characterized in that: In step S4, a preliminary cooperation score is calculated by weighting the total operating frequency and the total network throughput; The preliminary collaboration score comprehensively considers the total working frequency and the total network throughput to obtain the predicted collaboration score of each high-load partition by constructing a polynomial regression model.

9. The data processing method for an industrial big data platform according to claim 8, characterized in that: In step S4, a first scoring threshold and a second scoring threshold are preset, and the first scoring threshold is greater than the second scoring threshold, and the predicted collaborative score is divided into states: If the predicted collaboration score is greater than the first scoring threshold, the overall performance status of the industrial big data platform is judged to be in an efficient operation state; If the predicted collaboration score is less than or equal to the first scoring threshold and greater than the second scoring threshold, the overall performance status of the industrial big data platform is judged to be in a normal operating state; If the predicted collaboration score is less than or equal to the second score threshold, the overall performance status of the industrial big data platform is judged to be an inefficient operating state.

10. A data processing system for an industrial big data platform, based on a data processing method for an industrial big data platform according to any one of claims 1 to 9, characterized in that: It includes data source identification and partitioning module, priority assessment module, collaborative scheduling module and performance evaluation module; The data source identification and partitioning module is used to identify the data source in the industrial big data platform, determine the type of data source, and obtain the corresponding data partition; The priority evaluation module is used to detect the data transmission path of the data partition, collect delay and load indicators, and calculate the processing priority based on the hierarchical analysis method; The collaborative scheduling module is used to screen high-load partitions, determine whether to perform distributed collaborative processing, and collect the working frequency and network throughput of the data nodes in the high-load partitions; The performance evaluation module uses a nonlinear regression algorithm to analyze the collaborative score based on the working frequency and network throughput of the data nodes, and evaluates the overall performance status of the industrial big data platform accordingly.

Images