An automatic evaluation method for the accuracy of defect identification software

Abstract

This invention relates to the field of data processing technology, specifically to an automatic accuracy evaluation method for defect identification software. The method includes: collecting abnormal data from sensors; obtaining abnormal data segments from the abnormal data; acquiring the fluctuation degree of the abnormal data; determining the sensitivity of the defect identification software to each fault type based on the fluctuation degree; acquiring the similarity between different abnormal data; clustering all abnormal data based on the similarity to obtain several clusters of abnormal data; determining the fault type of each cluster based on the clusters and fault types; and obtaining the final accuracy of the defect identification software based on the sensitivity of the fault type, the fault type of each cluster, and the cluster itself. This invention solves the problem of accuracy distortion caused by using historical sample data as a reference for fault identification accuracy in defect identification software, thus improving the authenticity and accuracy of the accuracy evaluation of defect identification software.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to an automatic method for evaluating the accuracy of defect identification software. Background Technology

[0002] In the field of software engineering, defect detection software is widely used to detect and locate defects, errors, or potential problems in software code. This software helps developers identify and fix potential defects in advance by analyzing source code, execution paths, and static analysis, thereby improving software quality and maintainability.

[0003] Current performance evaluation of defect identification software primarily relies on manually analyzed and labeled data, leading to subjectivity and inconsistency in the evaluation process. Furthermore, as software scale and complexity increase, more automated and precise evaluation methods are needed to ensure the accuracy and reliability of defect identification software. Typically, after fault analysis and identification of any type of sensor data, the results are analyzed and calculated using a confusion matrix to obtain the accuracy of the identification. However, sensor fault types are diverse, and judging accuracy solely based on the ratio of correctly identified defects to the total sample size is easily affected by data fluctuations, resulting in inaccurate accuracy assessments of defect identification software. Summary of the Invention

[0004] To address the above problems, this invention provides an automatic method for evaluating the accuracy of defect identification software.

[0005] The present invention provides an automatic accuracy evaluation method for defect identification software, which adopts the following technical solution:

[0006] One embodiment of the present invention provides an automatic method for evaluating the accuracy of defect identification software, the method comprising the following steps:

[0007] Collect abnormal data from the sensor, classify the abnormal data to obtain several fault types of the sensor and several abnormal data for each fault type;

[0008] Obtain the abnormal data segment for each abnormal data point; based on the amplitude of the data points in the abnormal data segment and the mean envelope of the abnormal data segment, obtain the fluctuation degree of each abnormal data point.

[0009] The sensitivity of the defect identification software to each fault type is obtained based on the fault type and the degree of fluctuation of different abnormal data of the same fault type; the similarity between different abnormal data is obtained based on different abnormal data and the degree of fluctuation of different abnormal data; all abnormal data are clustered based on the similarity to obtain several clusters of abnormal data; the fault type of each cluster is obtained based on the clusters and the abnormal data of fault type; the probability of error generated by the fault type of each cluster is obtained based on the sensitivity of fault type, the fault type of each cluster and the cluster.

[0010] The final accuracy of the defect identification software is obtained by determining the probability of error arising from the fault type of each cluster.

[0011] Furthermore, the specific steps for obtaining the abnormal data segment for each abnormal data based on the abnormal data are as follows:

[0012] Any abnormal data of any fault type is recorded as the target abnormal data; abnormal data points in the target abnormal data are obtained through defect identification software and recorded as target abnormal data points; the data segment range determined by the TH1 data points to the left of the target abnormal data point is taken as an abnormal data segment of the target abnormal data; TH1 is a preset first value.

[0013] Furthermore, the specific steps for obtaining the fluctuation degree of each abnormal data point based on the amplitude of the data points in the abnormal data segment and the mean envelope of the abnormal data segment are as follows:

[0014] Any abnormal data of any fault type is denoted as the target abnormal data, and the abnormal data segment of the target abnormal data is denoted as the target abnormal data segment; the mean envelope of the target abnormal data segment is obtained and denoted as the target mean envelope.

[0015] The amplitude of the linearly normalized data point t in the target abnormal data segment is denoted as the first amplitude. The product of the absolute value of the difference between the slope of the corresponding point of the t-th data point in the target mean envelope and the slope of the corresponding point of the (t+1)-th data point in the target mean envelope, and the first amplitude, is denoted as the first parameter of the t-th data point in the target abnormal data segment. The average value of the first parameters of all data points in the target abnormal data segment is taken as the fluctuation degree of the target abnormal data segment.

[0016] The degree of fluctuation of the target abnormal data segment is used as the degree of fluctuation of the target abnormal data.

[0017] Furthermore, the specific steps involved in determining the sensitivity of the defect identification software to each fault type based on the fault type and the degree of fluctuation of different abnormal data of the same fault type are as follows:

[0018] Let any one type of fault be denoted as the target fault type;

[0019] The ratio of the number of abnormal data of the target fault type to the number of abnormal data of all fault types is denoted as the first ratio. The absolute value of the difference in the fluctuation degree of any two different abnormal data of the target fault type is denoted as a second parameter of the target fault type. The product of the inverse proportional value of the sum of all the second parameters of the target fault type and the first ratio is used as the sensitivity of the defect identification software to the target fault type.

[0020] Furthermore, the specific steps for obtaining the similarity between different abnormal data based on different abnormal data and the degree of fluctuation of different abnormal data are as follows:

[0021] The product of the inverse proportional value of the absolute value of the difference between the volatility of the a-th outlier and the volatility of the b-th outlier, and the Pearson correlation coefficient between the a-th and b-th outliers, is used as the similarity factor between the a-th and b-th outliers.

[0022] Obtain the similarity factor among all different anomalous data, perform linear normalization on the similarity factor among all different anomalous data, and use the result as the similarity between different anomalous data.

[0023] Furthermore, the specific steps involved in clustering all abnormal data based on similarity to obtain several clusters of abnormal data are as follows:

[0024] K-means clustering is performed on all outlier data, and the distance metric is the difference between the similarity between outlier data and 1, resulting in several clusters of outlier data.

[0025] Furthermore, the specific steps for obtaining the fault type of each cluster based on the abnormal data of the cluster and fault type are as follows:

[0026]

[0027] In the formula, This represents the average fluctuation level of all outliers in the k-th cluster. E3 is the average fluctuation of all abnormal data for the s-th fault type. s,w W represents the fluctuation level of the w-th abnormal data of the s-th fault type. s f is the number of abnormal data of the s-th fault type. k,s Let γ be the similarity between the k-th cluster and the s-th fault type, and let γ be a hyperparameter to avoid a denominator of 0.

[0028] Obtain the similarity between the k-th cluster and each fault type, and take the fault type corresponding to the maximum similarity as the fault type of the k-th cluster.

[0029] Furthermore, the specific steps for determining the probability of error generation for each fault type based on the sensitivity of the fault type, the fault type of each cluster, and the cluster itself are as follows:

[0030]

[0031] In the formula, TN k TN1 represents the number of fault types corresponding to abnormal data within the k-th cluster that differ from the fault types of the k-th cluster itself. k Let f1 be the number of anomalous data within the k-th cluster; let f1 be the fault type of the k-th cluster, denoted as the first fault type. k P' represents the similarity between the k-th cluster and the first fault type. k T is a possible factor, representing the error type for the k-th cluster. k The sensitivity corresponding to the fault type of the k-th cluster;

[0032] The probability of error arising from the fault type of each cluster is obtained based on possible factors.

[0033] Furthermore, the specific steps involved in obtaining the probability of error arising from the fault type of each cluster based on possible factors are as follows:

[0034] Obtain the possible factors that cause errors for each cluster's fault type, perform linear normalization on the possible factors that cause errors for all clusters' fault types, and use the result as the probability of each cluster's fault type causing errors.

[0035] Furthermore, the specific steps for obtaining the final accuracy of the defect identification software based on the probability of error generation according to the fault type of each cluster are as follows:

[0036] The accuracy of the defect identification software is obtained based on the confusion matrix, and is denoted as the initial accuracy of the defect identification software.

[0037] The final accuracy of the defect identification software is obtained by multiplying the difference between 1 and the average probability of errors caused by all clusters of fault types, and the product of the difference and the initial accuracy of the defect identification software.

[0038] The beneficial effects of the technical solution of this invention are as follows: This invention collects abnormal data from sensors, classifies the abnormal data to obtain several fault types of the sensors and several abnormal data points for each fault type, obtains abnormal data segments for each abnormal data point, facilitating subsequent accuracy evaluation. By analyzing the amplitude of data points within the abnormal data segments and the mean envelope of the abnormal data segments, the fluctuation degree of each abnormal data point is obtained. This value is used as the change characteristic value of the abnormal data corresponding to a specific fault, avoiding the problem of unclear data change regularity when analyzing the changes of abnormal data corresponding to the same fault in subsequent analyses due to single data value changes. This improves the authenticity of the accuracy evaluation. Then, the defect identification software obtains the data for each... This operation assesses the sensitivity of different fault types, avoiding misjudgments caused by small differences in fault data when obtaining accuracy, thus improving the authenticity of accuracy assessment. All abnormal data are clustered to obtain the fault type for each cluster. Finally, based on the sensitivity of fault types, the fault type of each cluster, and the cluster itself, the probability of error arising from the fault type in each cluster is obtained. The final accuracy of the defect identification software is then derived based on the probability of error arising from the fault type in each cluster. This solves the problem of accuracy distortion caused by using historical sample data as a reference for fault identification accuracy, improving the authenticity and accuracy of the defect identification software's accuracy assessment. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart illustrating the steps of an automatic accuracy evaluation method for defect identification software provided in one embodiment of the present invention;

[0041] Figure 2 This is a feature flowchart illustrating the process of obtaining the final accuracy of defect identification software from abnormal data, as provided in one embodiment of the present invention. Detailed Implementation

[0042] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of an automatic accuracy evaluation method for defect identification software proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0044] The following description, in conjunction with the accompanying drawings, details a specific scheme for an automatic accuracy evaluation method for defect identification software provided by the present invention.

[0045] Please see Figure 1 and Figure 2 This document illustrates a flowchart of the steps in an automatic accuracy evaluation method for defect identification software provided by an embodiment of the present invention, and a feature flowchart for obtaining the final accuracy of defect identification software through abnormal data. The method includes the following steps:

[0046] Step S001: Collect abnormal data from the sensor, classify the abnormal data to obtain several fault types of the sensor and several abnormal data for each fault type.

[0047] Specifically, several abnormal data points are obtained from any type of sensor. These abnormal data points are then classified through manual quality inspection to obtain several fault types of the sensor and abnormal data for each fault type. It should be noted that each fault type corresponds to several abnormal data points, each abnormal data point contains several data points, and each data point corresponds to an amplitude and a time (moment).

[0048] At this point, we have obtained several fault types of the sensor and abnormal data for each fault type.

[0049] Step S002: Obtain the abnormal data segment for each abnormal data point based on the abnormal data; obtain the fluctuation degree of each abnormal data point based on the amplitude of the data points in the abnormal data segment and the mean envelope of the abnormal data segment.

[0050] It should be noted that, due to the variety of fault types and the differences in sensor data performance corresponding to different faults, the accuracy of software defect identification depends on the combination of data change characteristics to determine the fault type and the historical fault data samples to estimate the accuracy of the current determination. The greater the difference in the change characteristics of sensor data corresponding to the same fault, the less the defect identification software is affected by sensor data changes in the identification of the same type of fault, and the higher its accuracy. Therefore, the first step is to determine the degree of fluctuation among the various types of data obtained based on the fault type.

[0051] It should be noted that, since sensor data may exhibit some early warning signs when abnormal fluctuations occur, in order to more accurately represent the characteristics of data changes, for any abnormal data in any fault dataset, the data point corresponding to the fault is located, and then a local data segment is obtained for fluctuation analysis.

[0052] Specifically, the abnormal data segment for each abnormal data point is obtained, as follows:

[0053] Any abnormal data of any fault type is denoted as target abnormal data. Abnormal data points in the target abnormal data are obtained through defect identification software and denoted as target abnormal data points. It should be noted that an abnormal data contains only one abnormal data point. The data segment range determined by the TH1 data points to the left of the target abnormal data point is taken as an abnormal data segment of the target abnormal data. TH1 is a preset first value, which is 10 in this embodiment. It should be noted that obtaining abnormal data points in the target abnormal data through defect identification software is an existing method and will not be described in detail in this embodiment. The TH1 data points to the left of the target abnormal data point are adjacent TH1 data points. For example, if the target abnormal data point is the 100th data point, then the TH1 data points to the left are the 90th to the 99th data points. If the abnormal data point is to the left of the abnormal data, the abnormal data segment will exceed the range of the abnormal data when obtaining the abnormal data segment. Therefore, the abnormal data segment is formed by the actual existing abnormal data points.

[0054] Furthermore, based on the amplitude of the data points in the outlier data segment and the mean envelope of the outlier data segment, the degree of fluctuation of each outlier data point is obtained, as follows:

[0055] The abnormal data segment of the target abnormal data is denoted as the target abnormal data segment; the mean envelope of the target abnormal data segment is obtained and denoted as the target mean envelope; it should be noted that obtaining the mean envelope of the target abnormal data segment is an existing method, and will not be described in detail in this embodiment.

[0056]

[0057] In the formula, n is the number of data points in the target abnormal data segment, and q t Let q be the magnitude of the t-th data point in the target abnormal data segment. min q represents the minimum amplitude of data points in the target abnormal data segment. max A represents the maximum amplitude of data points in the target abnormal data segment. t Let be the slope of the t-th data point in the target outlier segment corresponding to the point in the target mean envelope; it should be noted that obtaining the slope of the midpoint of the target mean envelope is an existing method, which will not be described again in this embodiment; A t+1 Let || be the slope of the (t+1)th data point of the target outlier segment corresponding to the point in the target mean envelope, || be the absolute value, E be the degree of fluctuation of the target outlier segment, and α be a hyperparameter to avoid a denominator of 0. In this embodiment, α = 1 is used for description.

[0058] The degree of fluctuation of the target abnormal data segment is used as the degree of fluctuation of the target abnormal data.

[0059] It should be noted that, This represents the normalized result of the amplitude of the t-th data point. A larger value indicates a higher prominence of the target outlier segment and a greater degree of representation of its data characteristics; |A t -A t+1 | represents the slope difference between corresponding points on the target mean envelope of two adjacent data points in the target outlier data segment. This difference reflects the degree of fluctuation in the data baseline; a larger value indicates a higher degree of representation of data characteristics within the target outlier data segment. Traversing all data points within the target outlier data segment yields... The larger this value, the more obvious the fluctuation characteristics of the target abnormal data segment, and the greater the fluctuation of the sensor's abnormal data. By defining the fluctuation degree of the abnormal data segment by the data changes and baseline changes within the abnormal data segment of the sensor's abnormal data, this value is used as the change characteristic value of the sensor data corresponding to a certain fault. This avoids the lack of obvious data change regularity when analyzing the data changes corresponding to the same fault in subsequent analyses due to the single analysis of data value changes, and improves the authenticity of the accuracy assessment.

[0060] This gives us the degree of fluctuation for each abnormal data point.

[0061] Step S003: Based on the fault type and the fluctuation degree of different abnormal data of the same fault type, obtain the sensitivity of the defect identification software to each fault type; cluster all abnormal data to obtain several clusters of abnormal data; obtain the fault type of each cluster; based on the sensitivity of the fault type, the fault type of each cluster and the cluster, obtain the probability of error generated by the fault type of each cluster.

[0062] It should be noted that since fault classification is done through human evaluation, and the fluctuation level of each abnormal data is obtained according to the above steps, for defect identification software, the greater the difference in fluctuation characteristics of various data for any type of fault, the lower the sensitivity to identify that type of fault, indicating that the fault has more obvious characteristics. Conversely, if the difference in fluctuation characteristics is small, the sensitivity to that type is higher. At the same time, the higher the frequency of any type of fault, the greater the possibility of error when using software to identify anomalies. Therefore, it is necessary to first obtain the software's sensitivity to the current fault type.

[0063] It should be noted that since the types of faults that occur in any sensor are diverse, the total number of sensor faults in the collected historical data is counted, and the occurrence frequency of any particular sensor fault is also obtained, thereby obtaining the probability of occurrence of the current fault type. Then, according to the above logic, for any sensor fault data, the greater the difference in the fluctuation of the data, the lower the sensitivity of the defect identification software to that type of fault. Therefore, by calculating the difference of multiple abnormal data corresponding to any fault, and combining it with the above fault occurrence probability, the sensitivity of the current defect identification software to the current type of fault is determined.

[0064] Specifically, the sensitivity of the defect identification software to each fault type is determined based on the fault type and the degree of fluctuation of different abnormal data of the same fault type, as follows:

[0065]

[0066] In the formula, N j M represents the number of abnormal data points for the target fault type, M represents the number of abnormal data points for all fault types, and E1 represents the number of abnormal data points for all fault types. i E1 represents the fluctuation level of the i-th abnormal data of the target fault type. j Let || be the fluctuation level of the j-th abnormal data of the target fault type, || be the absolute value, T be the sensitivity of the defect identification software to the target fault type, and β be the hyperparameter to avoid a denominator of 0. In this embodiment, β = 1 is used for description.

[0067] It should be noted that, This indicates the frequency of occurrence of the target fault type. A higher value indicates a higher probability of the target fault type occurring, and also a greater likelihood that the defect identification software will produce errors in fault identification; |E1 i -E1 j| represents the difference in fluctuation between any two abnormal data points of the target fault type. A larger value indicates a greater difference. In this case, iterating through all abnormal data and obtaining the difference in fluctuation of all abnormal data points of the target fault type yields... The larger this value, the greater the difference in abnormal data for the target fault type, and the lower the sensitivity of the defect identification software to the target fault type. For a target fault type, if the fluctuation of its abnormal data varies greatly and its frequency of occurrence is high, the probability of error is also greater. Therefore, frequency is used to weight the fluctuation difference to obtain T. The larger this value, the greater the sensitivity of the defect identification software to the target fault type. By calculating the occurrence frequency of any fault type and combining it with the fluctuation difference of abnormal data, the sensitivity of the defect identification software to the occurrence of the fault type is determined. This operation avoids fault misjudgment caused by small differences in abnormal data when obtaining the identification accuracy in the later stage, and improves the authenticity of the accuracy assessment.

[0068] It should be noted that if the defect identification software has low sensitivity to the current fault, the adaptive identification accuracy of the fault type will be lower when comparing new abnormal data with the database. Theoretically, the smaller the fluctuation of the abnormal data, the greater the probability that it represents the same fault. However, there are also cases where the fluctuation of the abnormal data corresponding to the same fault is large. At the same time, when using the confusion matrix to calculate the accuracy of fault identification, it only indicates the correct or incorrect performance of the software in historical sample data. It cannot accurately represent the accuracy of fault identification at present and can only be used as a reference for the accuracy base. Therefore, it is necessary to calculate the probability of misjudgment in the fault identification process by combining the differences between abnormal data with the sensitivity calculated above.

[0069] Specifically, based on different types of outlier data and their varying degrees of fluctuation, the similarity between different outlier data is calculated as follows:

[0070]

[0071] In the formula, E2 a E2 represents the degree of fluctuation of the a-th outlier. b Let c represent the fluctuation level of the b-th outlier, and c be a preset hyperparameter to prevent the denominator from being zero. In this embodiment, c = 0.1 is used for description. a,b Let S' be the Pearson correlation coefficient between the a-th outlier and the b-th outlier, and S' be the similarity factor between the a-th and b-th outlier.

[0072] Obtain the similarity factor among all different anomalous data, perform linear normalization on the similarity factor among all different anomalous data, and use the result as the similarity between different anomalous data.

[0073] It should be noted that PC a,b The value ranges from [-1, 1]. The closer the value is to 0, the weaker the correlation; closer to 1 indicates a positive correlation; and -1 indicates a negative correlation. For the correlation between outliers, the closer to a positive correlation, the higher the similarity. Therefore, the larger this value, the higher the correlation between the a-th outlier and the b-th outlier. |E2 a -E2 b | represents the difference in the degree of fluctuation between any two outlier data points. The smaller this value, the more similar the data are; therefore, it is represented by a fraction. This yields the similarity judgment formula. The higher the value, the higher the similarity.

[0074] Furthermore, all abnormal data are clustered based on similarity to obtain several clusters of abnormal data, as follows:

[0075] K-means clustering is performed on all outlier data. The distance metric is the difference between the similarity between outlier data and 1, resulting in several clusters of outlier data. It should be noted that this embodiment uses the elbow method to obtain the K value of K-means clustering, and the specific method is an existing method.

[0076] Furthermore, based on the anomaly data of the cluster and the fault type, the fault type of each cluster is obtained, as follows:

[0077]

[0078] In the formula, This represents the average fluctuation level of all outliers in the k-th cluster. E3 is the average fluctuation of all abnormal data for the s-th fault type. s,w W represents the fluctuation level of the w-th abnormal data of the s-th fault type. s f is the number of abnormal data of the s-th fault type. k,s Let γ be the similarity between the k-th cluster and the s-th fault type, and let γ be a hyperparameter to avoid a denominator of 0. In this embodiment, γ = 1 is used for description.

[0079] Obtain the similarity between the k-th cluster and each fault type, and take the fault type corresponding to the maximum similarity as the fault type of the k-th cluster.

[0080] It should be noted that, This indicates the degree of matching between the fault type corresponding to the k-th cluster and the s-th fault type. The smaller the value, the higher the match. This represents the average difference between each abnormal data point of the s-th fault type and the average of the fluctuations of all abnormal data points in the k-th cluster. The smaller this value, the higher the matching degree between the s-th fault type and the fault type corresponding to the k-th cluster. Therefore, the reciprocal of the product of the two is taken, and the fault type corresponding to the highest similarity is taken as the fault type of the k-th cluster.

[0081] Furthermore, based on the sensitivity of the fault type, the fault type of each cluster, and the cluster itself, the probability of error generation for each fault type in each cluster is obtained, as follows:

[0082]

[0083] In the formula, TN k TN1 represents the number of fault types corresponding to abnormal data within the k-th cluster that differ from the fault types of the k-th cluster itself. k Let f1 be the number of anomalous data within the k-th cluster; let f1 be the fault type of the k-th cluster, denoted as the first fault type. k P' represents the similarity between the k-th cluster and the first fault type. k T is a possible factor, representing the error type for the k-th cluster. k This represents the sensitivity corresponding to the fault type of the k-th cluster.

[0084] Obtain the possible factors that cause errors for each cluster's fault type, perform linear normalization on the possible factors that cause errors for all clusters' fault types, and use the result as the probability of each cluster's fault type causing errors.

[0085] It should be noted that T k A larger value indicates a greater likelihood of error. This represents the proportion of other fault types within the k-th cluster, i.e., fault types other than those in the k-th cluster. A larger value indicates a greater likelihood of confusion caused by the outlier data, and consequently, a greater possibility of error. f1 k A higher frequency indicates greater reliability in the data comparison process. Furthermore, the frequency itself is obtained through clustering results; therefore, this value can be used to... We perform weighting; therefore, we use the Euclidean norm to transform the above characteristics into a single dimension, thus obtaining... The values ​​are then normalized to facilitate subsequent calculations, thereby obtaining the probability of errors in fault type for each cluster. The larger the value, the lower the accuracy should be in fault identification.

[0086] It should be noted that by clustering abnormal data based on the differences in fluctuation characteristics of different abnormal data, and obtaining the corresponding fault type based on the clustering results, and finally judging the possibility of error in identifying and obtaining the accuracy of this type of fault based on the difference between the data characteristics of the fault type and the data characteristics within the current cluster, this operation avoids the distortion of accuracy caused by simply indicating the correct or incorrect performance of the software in historical sample data, which cannot accurately represent the current accuracy of fault identification as a reference, thus improving the authenticity of the accuracy assessment.

[0087] This gives us the probability that each cluster's fault type will produce an error.

[0088] Step S004: Obtain the final accuracy of the defect identification software based on the probability of errors arising from the fault type of each cluster.

[0089] It should be noted that when using the confusion matrix to obtain the accuracy, it is usually necessary to obtain true positives (where the software correctly identifies the defect), true negatives (where the software correctly identifies areas without defects), false positives (where the software incorrectly marks areas as defective), and false negatives (where the software incorrectly marks areas as without defects). Then, the accuracy is obtained according to the calculation formula. At this point, the probability of error is obtained according to the above steps. After the identification software classifies the abnormal data of a certain sensor, this value is used to weight the accuracy, thereby completing the correction of the accuracy evaluation of the defect identification software.

[0090] Specifically, the final accuracy of the defect identification software is obtained based on the probability of error arising from the fault type of each cluster, as follows:

[0091] The accuracy of the defect identification software is obtained from the confusion matrix and denoted as the initial accuracy of the defect identification software. It should be noted that obtaining the accuracy of the defect identification software from the confusion matrix is ​​an existing method, and will not be described in detail in this embodiment.

[0092]

[0093] In the formula, Acc represents the initial accuracy of the defect identification software; Acc' represents the average probability of errors occurring across all clusters of fault types, and Acc' is the final accuracy of the defect identification software.

[0094] It should be noted that the greater the likelihood of error in fault type, the lower the accuracy should be in fault identification. Therefore, an inverse proportional processing is performed to correct the initial accuracy and obtain the final accuracy of the defect identification software.

[0095] At this point, the initial accuracy of the defect identification software is corrected based on the probability of errors arising from the fault types of the clusters, thus completing the automatic evaluation of the accuracy of the defect identification software.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatically evaluating the accuracy of defect identification software, characterized in that, The method includes the following steps: Collect abnormal data from the sensor, classify the abnormal data to obtain several fault types of the sensor and several abnormal data for each fault type; Obtain the abnormal data segment for each abnormal data point; based on the amplitude of the data points in the abnormal data segment and the mean envelope of the abnormal data segment, obtain the fluctuation degree of each abnormal data point. The sensitivity of the defect identification software to each fault type is determined based on the fault type and the degree of fluctuation of different abnormal data of the same fault type. The specific steps include the following: Let any one type of fault be denoted as the target fault type; The ratio of the number of abnormal data of the target fault type to the number of abnormal data of all fault types is recorded as the first ratio. The absolute value of the difference between the fluctuation of any two different abnormal data of the target fault type is recorded as a second parameter of the target fault type. The product of the inverse proportional value of the sum of all the second parameters of the target fault type and the first ratio is used as the sensitivity of the defect identification software to the target fault type. Based on different abnormal data and their fluctuation levels, the similarity between different abnormal data is obtained; based on the similarity, all abnormal data are clustered to obtain several clusters of abnormal data; based on the clusters and fault types of abnormal data, the fault type of each cluster is obtained; based on the sensitivity of the fault type, the fault type of each cluster, and the cluster, the probability of error generated by the fault type of each cluster is obtained. The final accuracy of the defect identification software is obtained by determining the probability of error arising from the fault type of each cluster.

2. The automatic accuracy evaluation method for defect identification software according to claim 1, characterized in that, The specific steps for obtaining the abnormal data segment for each abnormal data based on the abnormal data are as follows: Any abnormal data of any fault type is recorded as the target abnormal data; abnormal data points in the target abnormal data are obtained through defect identification software and recorded as target abnormal data points. To the left of the target abnormal data point The range of data segments determined by each data point is used as an abnormal data segment of the target abnormal data. This is a preset first value.

3. The method for automatically evaluating the accuracy of defect identification software according to claim 1, characterized in that, The specific steps for obtaining the fluctuation degree of each abnormal data point based on the amplitude of the data points in the abnormal data segment and the mean envelope of the abnormal data segment are as follows: Any abnormal data of any fault type is denoted as the target abnormal data, and the abnormal data segment of the target abnormal data is denoted as the target abnormal data segment; the mean envelope of the target abnormal data segment is obtained and denoted as the target mean envelope. The first segment of the target abnormal data segment The amplitude after linear normalization of the nth data point is denoted as the first amplitude. The nth amplitude of the target abnormal data segment is then used. The slope of the corresponding point of the target mean envelope for the data point and the slope of the target outlier segment The product of the absolute value of the difference in slope between the corresponding points of the n data points in the target mean envelope and the first amplitude is denoted as the nth data point in the target outlier segment. The first parameter of each data point is the average of the first parameters of all data points in the target abnormal data segment, which is taken as the degree of fluctuation of the target abnormal data segment. The degree of fluctuation of the target abnormal data segment is used as the degree of fluctuation of the target abnormal data.

4. The method for automatically evaluating the accuracy of defect identification software according to claim 1, characterized in that, The specific steps for obtaining the similarity between different abnormal data based on different abnormal data and the degree of fluctuation of different abnormal data are as follows: The first The degree of fluctuation of the first abnormal data and the first The inverse proportional value of the absolute value of the difference in the degree of fluctuation of the first outlier data, and the value of the second outlier data. The first abnormal data and the first The product of the Pearson correlation coefficients of the nth outlier data is used as the product of the nth outlier data. The first abnormal data and the first Similarity factors between anomalous data; Obtain the similarity factor among all different anomalous data, perform linear normalization on the similarity factor among all different anomalous data, and use the result as the similarity between different anomalous data.

5. The method for automatically evaluating the accuracy of defect identification software according to claim 1, characterized in that, The specific steps involved in clustering all abnormal data based on similarity to obtain several clusters of abnormal data are as follows: K-means clustering is performed on all outlier data, and the distance metric is the difference between the similarity between outlier data and 1, resulting in several clusters of outlier data.

6. The method for automatically evaluating the accuracy of defect identification software according to claim 1, characterized in that, The specific steps for obtaining the fault type for each cluster based on the anomaly data of the cluster and fault type are as follows: In the formula, For the first The average degree of fluctuation of all outliers in each cluster. For the first The average fluctuation of all abnormal data for each fault type. For the first The first type of fault The degree of fluctuation in the abnormal data For the first The number of abnormal data for each type of fault. For the first The first cluster and the first Similarity of various fault types To avoid hyperparameters with a denominator of 0; Get the The similarity between each cluster and each fault type is used to determine the fault type corresponding to the highest similarity score. Fault types for each cluster.

7. The automatic accuracy evaluation method for defect identification software according to claim 1, characterized in that, The specific steps for determining the probability of error generation for each fault type based on the sensitivity of the fault type, the fault type of each cluster, and the cluster itself are as follows: In the formula, For the first The fault type corresponding to the abnormal data within each cluster is the same as the first The number of different fault types in each cluster For the first The number of outliers within each cluster; the number of outliers within the first cluster; The fault types of each cluster are denoted as the first fault type. For the first The similarity between each cluster and the first fault type; For the first The possible factors that cause errors in each cluster of fault types For the first The sensitivity corresponding to the fault types of each cluster; The probability of error arising from the fault type of each cluster is obtained based on possible factors.

8. The method for automatically evaluating the accuracy of defect identification software according to claim 7, characterized in that, The specific steps involved in obtaining the probability of error arising from the fault type of each cluster based on possible factors are as follows: Obtain the possible factors that cause errors for each cluster's fault type, perform linear normalization on the possible factors that cause errors for all clusters' fault types, and use the result as the probability of each cluster's fault type causing errors.

9. The automatic accuracy evaluation method for defect identification software according to claim 1, characterized in that, The specific steps involved in obtaining the final accuracy of the defect identification software based on the probability of errors arising from the fault type of each cluster are as follows: The accuracy of the defect identification software is obtained based on the confusion matrix, and is denoted as the initial accuracy of the defect identification software. The final accuracy of the defect identification software is obtained by multiplying the difference between 1 and the average probability of errors caused by all clusters of fault types, and the product of the difference and the initial accuracy of the defect identification software.

Images