A Real-Time Testing Method and System for Industrial Algorithms Based on Data Playback

Abstract

This invention belongs to the field of algorithm testing and discloses a method and system for real-time testing of industrial algorithms based on data playback. The method includes: collecting field data and preprocessing it to obtain a simulated dataset; running a user algorithm using the simulated dataset, obtaining the algorithm's running results in real time, and calculating the algorithm score; while maintaining real-time algorithm operation and score calculation, collecting field data again and preprocessing it, filtering the preprocessed data based on cluster analysis and information entropy calculation to obtain an updated dataset, and using the updated dataset to hot-update the simulated dataset; judging the hot-updated simulated dataset, if a stopping condition is met, the latest calculated algorithm score is used as the final algorithm score; if the condition is not met, hot-update is performed again until the stopping condition is met, and the latest calculated user algorithm score is used as the final algorithm score.

Description

Technical Field

[0001] This invention belongs to the field of algorithm testing technology, and in particular relates to a real-time testing method and system for industrial algorithms based on data playback. Background Technology

[0002] With the continuous development of industrial technology, the intelligence level of algorithms in industrial fields such as intelligent manufacturing, chemical production, metallurgy, transportation and logistics, and construction and infrastructure is becoming increasingly higher. Testing and verification of these algorithms are becoming increasingly important. However, in traditional benchmark testing, algorithm verification can only be conducted offline, making real-time monitoring and data analysis difficult. This hinders the adaptation to changing industrial needs, and over time, equipment performance may degrade, and fault types may change, leading to errors in the algorithm test results. To address this challenge, there is an urgent need for a real-time testing method and system for industrial algorithms. This system should be able to effectively and accurately verify algorithms while they are running in real time, comprehensively evaluate their overall performance in a real-time environment, and provide effective support for industrial intelligence. Summary of the Invention

[0003] The technical problem to be solved by this invention is that traditional industrial algorithm testing lacks real-time performance and that real-time testing of industrial algorithms cannot guarantee the rationality of scoring and the authenticity and diversity of the dataset. To overcome the shortcomings and defects mentioned in the background technology, this invention provides a method and system for real-time testing of industrial algorithms based on data playback.

[0004] To address the aforementioned technical problems, this invention proposes a real-time testing method for industrial algorithms based on data playback, comprising the following steps:

[0005] S1: Collect on-site datasets to obtain raw datasets, preprocess the raw datasets to obtain simulated datasets;

[0006] S2: Run the user-uploaded algorithm under test using a simulated dataset, obtain the running results of the user-uploaded algorithm under test in real time, evaluate the user-uploaded algorithm under test, and calculate the algorithm score of the user-uploaded algorithm under test in real time.

[0007] S3: While keeping S2 running in real time, collect field data for a specific time period to obtain a new raw dataset. Preprocess the new raw dataset, and filter the preprocessed new raw dataset based on cluster analysis and information entropy calculation to obtain an updated dataset. Insert the updated dataset into the simulation dataset to perform a hot update of the simulation dataset.

[0008] S4: Judge the simulated dataset after hot update. If the set stopping condition is met, the algorithm score obtained from the last calculation is used as the final algorithm score. If the condition is not met, repeat S3 until the simulated dataset meets the stopping condition and the user algorithm score obtained from the last calculation is used as the final algorithm score.

[0009] Preferably, the evaluation of the user-uploaded algorithm and the real-time calculation of the algorithm score for the user-uploaded algorithm include:

[0010] The scores and weights of the performance metrics, complexity metrics, and stability metrics of the algorithms uploaded by users are calculated.

[0011] Calculate the algorithm score for the algorithm uploaded by the user. 现场录波回放 :

[0012] Score 现场录波回放 =w performance Score performance +w complexity Score complexity +w runtime Score runtime (1);

[0013] Among them, w performance w complexity w runtime The weights for performance metrics, complexity metrics, and stability metrics, respectively. performance Score complexity Score runtime The scores are for performance metrics, complexity metrics, and stability metrics, respectively.

[0014] Preferably, the performance index includes at least one performance parameter;

[0015] Performance score performance The calculation formula is as follows:

[0016]

[0017] Where, p i Let w represent the i-th performance parameter. pi This represents the weight of the i-th performance parameter, where n is the number of performance parameters.

[0018] Preferably, the complexity index includes:

[0019] Time complexity, space complexity;

[0020] Complexity score complexity The calculation formula is as follows:

[0021] Score complexity =w space Score space +w time Score time (3);

[0022]

[0023] Among them, w space w time Score represents the weights corresponding to space and time complexity. space Score time These are the space complexity score and the time complexity score, respectively, where T represents the time complexity of the current algorithm. max S represents the maximum allowed time complexity, and S represents the space complexity of the current algorithm. max This represents the maximum allowed space complexity.

[0024] Preferably, the stability index includes:

[0025] Algorithm stable running time;

[0026] Stability score runtime The calculation is as follows:

[0027]

[0028] Where β is the adjustment parameter for the fractional decay rate.

[0029] Preferably, step S3, which involves filtering the preprocessed new original dataset based on cluster analysis and information entropy calculation to obtain an updated dataset, includes:

[0030] S31: Perform cluster analysis on the preprocessed new original dataset to obtain the cluster centroids:

[0031] K initial cluster analysis centers are randomly selected from the new preprocessed original dataset;

[0032] Data points are assigned, with each data point assigned to a cluster formed by its nearest centroid based on distance.

[0033] The cluster analysis centroids are updated by recalculating the mean of all data points in each cluster. The mean of all data points in a cluster is calculated as shown in equation (6):

[0034]

[0035] Where, μ i C is the center point of the i-th cluster.i x represents all data points in the i-th cluster. j These are data points;

[0036] Repeat the process of assigning data points and updating cluster center points until the cluster center points no longer change or the specified number of iterations is reached;

[0037] S32: Calculate the information entropy and information gain of the preprocessed new original dataset. The information entropy is calculated as follows:

[0038]

[0039] Where H(D) is the information entropy of dataset D, p i It is the probability of category i;

[0040] Information gain is calculated as follows:

[0041] IG(D,x j )=H(D)-H(D\{x j}) (8);

[0042] Among them, IG(D,x) j ) is sample x j Information gain of dataset D, H(D\{x j}) is the removal of sample x j Conditional entropy of the subsequent dataset p i′ It is the probability that it is category i';

[0043] Based on the calculated information gain, find the sample points with larger information gain in different categories of data;

[0044] S33: Compare the cluster centers and sample points with high information gain obtained from S32 and S33 with the simulated dataset, calculate the vector similarity between the samples, delete existing working conditions or samples with high vector similarity, and use the remaining samples to form the updated dataset.

[0045] Based on a general inventive concept, the present invention also provides a real-time testing system for industrial algorithms based on data playback, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] This invention utilizes real-time data playback for algorithm verification and evaluation, simulating the algorithm model's operation in an industrial environment. This allows users to verify the algorithm in real time, simultaneously collecting real-time data and monitoring equipment performance changes. It extracts rich and representative typical data based on clustering analysis and information gain, and hot-updates the simulated dataset used in the algorithm verification process, ensuring the dataset reflects the latest field conditions and algorithm performance changes. Furthermore, it constructs evaluation indicators such as algorithm performance and complexity, comprehensively evaluating algorithms from different users in terms of accuracy, stability, and complexity. This achieves real-time monitoring and analysis of the algorithm while ensuring the fairness and effectiveness of algorithm evaluation. The real-time testing method for fault diagnosis algorithms based on data playback provided by this invention can be widely applied in industrial fields such as intelligent manufacturing, chemical production, metallurgy, transportation and logistics, and construction and infrastructure, for testing and evaluating algorithms for soft measurement, health monitoring, fault diagnosis, and process optimization. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a preferred embodiment of the real-time testing method for industrial algorithms based on data playback of the present invention;

[0050] Figure 2 This is a diagram comparing the test results of two testing methods: traditional benchmark testing and real-time testing of industrial algorithms based on data playback.

[0051] Figure 3 These are data charts of a simulated dataset that have not undergone hot data updates;

[0052] Figure 4 It is a data chart of a simulated dataset after a hot data update. Detailed Implementation

[0053] To facilitate understanding of the present invention, the present invention will be described more fully and in detail below with reference to the accompanying drawings and preferred embodiments, but the scope of protection of the present invention is not limited to the following specific embodiments.

[0054] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by those skilled in the art. The technical terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the invention.

[0055] Unless otherwise specified, all raw materials, reagents, instruments and equipment used in this invention can be purchased from the market or prepared by existing methods.

[0056] Example:

[0057] like Figure 1 As shown, this embodiment provides a real-time testing method for industrial algorithms based on data playback, including the following steps:

[0058] S1: Collect on-site data to obtain the raw dataset, preprocess the raw dataset to obtain the simulated dataset;

[0059] S2: Run the user-uploaded algorithm under test using a simulated dataset, obtain the running results of the user-uploaded algorithm under test in real time, evaluate the user-uploaded algorithm under test, and calculate the algorithm score of the user-uploaded algorithm under test in real time.

[0060] S3: While keeping S2 running in real time, collect field data for a specific time period again to obtain a new raw dataset. Preprocess the new raw dataset. The preprocessing methods include: timestamp alignment, data filtering, filtering the preprocessed new raw dataset based on cluster analysis and information entropy calculation to obtain an updated dataset. Insert the updated dataset into the simulation dataset to perform hot updates on the simulation dataset.

[0061] S4: Judge the simulated dataset after hot update. If the set stopping condition is met, the algorithm score obtained from the last calculation is used as the final algorithm score. If the condition is not met, repeat S3 until the simulated dataset meets the stopping condition and the user algorithm score obtained from the last calculation is used as the final algorithm score.

[0062] In this embodiment, the algorithm uploaded by the user is a fault detection algorithm for a central air conditioning system; the field data includes: chiller cooling capacity, chiller load rate, evaporator approach temperature, condenser approach temperature, chilled water outlet temperature, cooling water inlet temperature, condensing pressure, and the fault category corresponding to each data point; the preprocessing method includes: timestamp alignment and data selection. By preprocessing the collected field data, the data are made to correspond in time, while invalid data is filtered out.

[0063] The aforementioned real-time industrial algorithm testing method based on data playback verifies and evaluates algorithms in real time through on-site data playback, simulating the algorithm model's operation in an industrial environment. This allows users to verify the algorithm in real time. Furthermore, by hot-updating the simulated dataset, the dataset is updated without affecting the algorithm's real-time operation, ensuring that the simulated dataset reflects the latest on-site conditions and algorithm performance changes. This guarantees the diversity and realism of the simulated dataset, making algorithm verification more reliable. Compared to traditional benchmarking methods, this method offers advantages such as fair and accurate evaluation results, reliable verification process, high data authenticity, and good real-time and adaptive algorithm verification.

[0064] In a preferred embodiment of this practice, the evaluation of the user-uploaded algorithm and the real-time calculation of the algorithm score for the user-uploaded algorithm include:

[0065] The performance, complexity, and stability metrics of the user-uploaded algorithms are scored and weighted accordingly; an algorithm score (Score) is calculated for each algorithm uploaded for testing. 现场录波回放 :

[0066] Score 现场录波回放 =w performance Score performance +w complexity Score complexity +w runtime Score runtime (1);

[0067] Among them, w performance w complexity w runtime The weights for performance metrics, complexity metrics, and stability metrics, respectively. performance Score complexity Score runtime The scores are for performance metrics, complexity metrics, and stability metrics, respectively.

[0068] It is worth noting that, unlike traditional benchmark evaluation methods, this invention constructs performance indicators, complexity indicators, and stability indicators to comprehensively evaluate user algorithms from different aspects, ensuring the fairness and effectiveness of algorithm scoring. Furthermore, through dynamically adjustable evaluation indicator weights, it can adapt to different operating environments and user needs.

[0069] In a preferred embodiment of this invention, the performance indicators include at least one performance parameter;

[0070] Performance score performance The calculation formula is as follows:

[0071]

[0072] Where, p i Let w represent the i-th performance parameter. pi This represents the weight of the i-th performance parameter, where n is the number of performance parameters.

[0073] In this embodiment, four performance parameters were selected for the fault diagnosis algorithm of the central air conditioning system: accuracy, fault detection rate, fault isolation accuracy, and false alarm rate. These performance parameters were selected as representative performance parameters based on the characteristics of the user-uploaded algorithm "Fault Diagnosis Algorithm for Central Air Conditioning System" in this embodiment. Therefore, in this embodiment, the performance index score is... performance The calculation formula is as follows:

[0074] Score performance =w accuracy ·accuracy+w FDR ·FDR+w FIP ·FIP+w FAR ·FAR (3);

[0075] Among them, accuracy, FDR, FIP, and FAR are accuracy, fault detection rate, fault isolation accuracy, and false alarm rate, respectively. accuracy w FDR w FIP w FAR The weights are respectively for accuracy, fault detection rate, fault isolation accuracy, and false alarm rate;

[0076] The calculation methods for accuracy, fault detection rate (FDR), fault isolation accuracy (FIP), and false alarm rate (FAR) are as follows:

[0077]

[0078] Wherein, TP represents the number of samples that are actually faulty and are predicted to be faulty, TN represents the number of samples that are actually not faulty and are predicted to be not faulty, FP represents the number of samples that are actually not faulty but are predicted to be faulty, and FN represents the number of samples that are actually faulty but are predicted to be not faulty.

[0079] As a preferred embodiment of this invention, the complexity index includes:

[0080] Time complexity, space complexity;

[0081] Complexity score complexity The calculation formula is as follows:

[0082] Scorecomplexity =w space Score space +w time Score time (8);

[0083]

[0084] Among them, w space w time Score represents the weights corresponding to space and time complexity. space Score time These are the space complexity score and the time complexity score, respectively, where T represents the time complexity of the current algorithm. max S represents the maximum allowed time complexity, and S represents the space complexity of the current algorithm. max This represents the maximum allowed space complexity.

[0085] In a preferred embodiment of this invention, the stability indicators include:

[0086] Algorithm stable running time;

[0087] Stability score runtime The calculation is as follows:

[0088]

[0089] Where β is the adjustment parameter for the fractional decay rate.

[0090] In a preferred embodiment of this invention, the step of filtering the preprocessed new original dataset based on cluster analysis and information entropy calculation to obtain an updated dataset includes:

[0091] S31: Perform cluster analysis on the preprocessed new original dataset to obtain the cluster centroids:

[0092] K initial cluster analysis centers are randomly selected from the new preprocessed original dataset;

[0093] Data points are assigned, with each data point assigned to a cluster formed by its nearest centroid based on distance.

[0094] The cluster analysis centroids are updated by recalculating the mean of all data points in each cluster. The mean of all data points in a cluster is calculated as shown in equation (11):

[0095]

[0096] Where, μ i C is the center point of the i-th cluster. ix represents all data points in the i-th cluster. j These are data points;

[0097] Repeat the process of assigning data points and updating cluster center points until the cluster center points no longer change or the specified number of iterations is reached;

[0098] S32: Calculate the information entropy and information gain of the preprocessed new original dataset. The information entropy is calculated as follows:

[0099]

[0100] Where H(D) is the information entropy of dataset D, p i It is the probability of category i;

[0101] Information gain is calculated as follows:

[0102] IG(D,x j )=H(D)-H(D\{x j}) (13);

[0103] Among them, IG(D,x) j ) is sample x j Information gain of dataset D, H(D\{x j}) is the removal of sample x j Conditional entropy of the subsequent dataset p i′ It is the probability that it is category i';

[0104] Based on the calculated information gain, find the sample points with larger information gain in different categories of data;

[0105] In this embodiment, the field data includes: chiller cooling capacity, chiller load rate, evaporator approach temperature, condenser approach temperature, chilled water outlet temperature, cooling water inlet temperature, condensing pressure, and the fault category corresponding to each data point; therefore, finding the sample point with greater information gain in different categories of data in the above steps means finding the sample point with greater information gain in different fault categories of data.

[0106] S33: Compare the cluster centers obtained from S32 and S33, and the sample points with high information gain in the data of different fault categories, with the simulated dataset, calculate the vector similarity between the samples, delete existing working conditions or samples with high vector similarity, and use the remaining samples to form the updated dataset.

[0107] In this step, this embodiment uses cosine similarity to calculate the vector similarity between samples. The cosine similarity algorithm has the advantages of simple calculation, normalized measurement, and wide applicability. The formula for calculating cosine similarity is as follows:

[0108]

[0109] Where x is the simulated dataset that has not been hot-updated, and x' is the new data selected above, namely the centroids of the cluster analysis and the sample points with greater information gain in different categories of data.

[0110] It is worth noting that the data filtering and hot update method based on cluster analysis and information gain provided by this invention can supplement the simulated dataset with representative new sample data without affecting the real-time operation of the algorithm. This makes the algorithm data closer to the real running situation, thereby ensuring the diversity and authenticity of the simulated dataset used for data playback. Figure 3 , Figure 4 As shown, the distribution of the simulated dataset before and after the hot update has changed significantly. These changes represent the changes in the field operation. Through hot updates, the simulated dataset becomes closer to the real situation, thereby improving the diversity, representativeness, and authenticity of the verification data, and ensuring the accuracy and fairness of the algorithm scoring.

[0111] This application also provides a real-time algorithm testing system based on data playback, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described method. This real-time industrial algorithm testing system based on data playback can implement various embodiments of the above-described real-time industrial algorithm testing method based on data playback and achieve the same beneficial effects; further details are omitted here.

[0112] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A real-time testing method for industrial algorithms based on data playback, characterized in that, include: S1: Collect on-site data to obtain the raw dataset, preprocess the raw dataset to obtain the simulated dataset; S2: Run the user-uploaded algorithm under test using a simulated dataset, obtain the running results of the user-uploaded algorithm under test in real time, evaluate the user-uploaded algorithm under test, and calculate the algorithm score of the user-uploaded algorithm under test in real time. S3: While keeping S2 running in real time, collect field data for a specific time period to obtain a new original dataset. Preprocess the new original dataset, filter the preprocessed new original dataset based on cluster analysis and information entropy calculation to obtain an updated dataset, and insert the updated dataset into the simulation dataset to perform hot updates on the simulation dataset. S4: Judge the simulated dataset after hot update. If the set stopping condition is met, the algorithm score obtained from the last calculation is taken as the final algorithm score. If the condition is not met, repeat S3 until the simulated dataset meets the stopping condition and the user algorithm score obtained from the last calculation is taken as the final algorithm score.

2. The real-time testing method for industrial algorithms based on data playback as described in claim 1, characterized in that, The evaluation of user-uploaded algorithms and the real-time calculation of algorithm scores for these algorithms include: The scores and weights of the performance metrics, complexity metrics, and stability metrics of the algorithms uploaded by users are calculated. Calculate the algorithm score for the algorithm uploaded by the user. 现场录波回放 : Score 现场录波回放 =w performance ·Score performance +w complexity ·Score complexity +w runtime ·Score runtime (1); Among them, w performance w complexity w runtime The weights for performance metrics, complexity metrics, and stability metrics, respectively. performance Score complexity Score runtime The scores are for performance metrics, complexity metrics, and stability metrics, respectively.

3. The real-time testing method for industrial algorithms based on data playback as described in claim 2, characterized in that, The performance indicators include at least one performance parameter; Performance score performance The calculation formula is as follows: Where, p i Let w represent the i-th performance parameter. pi This represents the weight of the i-th performance parameter, where n is the number of performance parameters.

4. The real-time testing method for industrial algorithms based on data playback as described in claim 2, characterized in that, The complexity metrics include: Time complexity, space complexity; Complexity score complexity The calculation formula is as follows: Score complexity =w space ·Score space +w time ·Score time (3); Among them, w space w time Score represents the weights corresponding to space and time complexity. space Score time These are the space complexity score and the time complexity score, respectively, where T represents the time complexity of the current algorithm. max S represents the maximum allowed time complexity, and S represents the space complexity of the current algorithm. max This represents the maximum allowed space complexity.

5. The real-time testing method for industrial algorithms based on data playback as described in claim 2, characterized in that, The stability indicators include: Algorithm stable running time; Stability score runtime The calculation is as follows: Where β is the adjustment parameter for the fractional decay rate.

6. The real-time testing method for industrial algorithms based on data playback as described in claim 1, characterized in that, S3 describes the process of filtering the preprocessed original dataset using cluster analysis and information entropy calculation to obtain an updated dataset, including: S31: Perform cluster analysis on the preprocessed new original dataset to obtain the cluster centroids: K initial cluster analysis centers are randomly selected from the new preprocessed original dataset; Data points are assigned, with each data point assigned to a cluster formed by its nearest centroid based on distance. The cluster analysis centroids are updated by recalculating the mean of all data points in each cluster. The mean of all data points in a cluster is calculated as shown in equation (6): Where, μ i C is the center point of the i-th cluster. i x represents all data points in the i-th cluster. j These are data points; Repeat the process of assigning data points and updating cluster center points until the cluster center points no longer change or the specified number of iterations is reached; S32: Calculate the information entropy and information gain of the preprocessed new original dataset. The information entropy is calculated as follows: Where H(D) is the information entropy of dataset D, p i It is the probability of category i; Information gain is calculated as follows: IG(D,x j )=H(D)-H(D\{x j }) (9); Among them, IG(D,x) j ) is sample x j Information gain of dataset D, H(D\{x j }) is the removal of sample x j Conditional entropy of the subsequent dataset p i′ It is the probability that it is category i'; Based on the calculated information gain, find the sample points with larger information gain in different categories of data; S33: Compare the cluster centers and sample points with high information gain obtained from S32 and S33 with the simulated dataset, calculate the vector similarity between the samples, delete existing working conditions or samples with high vector similarity, and use the remaining samples to form the updated dataset.

7. A real-time testing system for industrial algorithms based on data playback, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

Images