Method for automatically evaluating accuracy of defect identification software

Abstract

The invention relates to the technical field of data processing, in particular to a defect recognition software accuracy automatic evaluation method, which comprises the following steps: acquiring abnormal data of a sensor, and acquiring an abnormal data segment of the abnormal data according to the abnormal data; acquiring the fluctuation degree of the abnormal data; obtaining the sensitivity degree of the defect identification software to each fault type according to the fluctuation degree; obtaining the similarity between different abnormal data; clustering all the abnormal data according to the similarity to obtain a plurality of class clusters of the abnormal data; obtaining the fault type of the class cluster according to the abnormal data of the class cluster and the fault type; and according to the sensitivity degree of the fault type, the fault type of the class cluster and the class cluster, obtaining the final accuracy of the defect identification software. According to the method, the problem of accuracy distortion caused by the fact that the defect identification software obtains the fault identification accuracy as a reference based on the historical sample data is solved, and the authenticity and accuracy of the accuracy evaluation of the defect identification software are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for automatically evaluating the accuracy of defect recognition software. Background Art

[0002] In software engineering, defect detection software is widely used to detect and locate defects, errors, or potential issues in software code. By analyzing source code, execution paths, and performing static analysis, these software helps developers identify and fix potential defects in advance, thereby improving software quality and maintainability.

[0003] Current performance evaluation of defect identification software primarily relies on manual analysis and annotated data, leading to subjectivity and inconsistency in the evaluation process. Furthermore, as software scale and complexity increase, more automated and precise evaluation methods are needed to ensure the accuracy and reliability of defect identification software. Typically, after performing fault analysis and identification on any type of sensor data, the results are analyzed and calculated using a confusion matrix to determine the accuracy of the identification. However, sensor fault data types vary greatly, and simply evaluating the accuracy based on the number of correctly identified defects and their proportion to the total sample size is susceptible to inherent data fluctuations, leading to inaccurate evaluations of defect identification software accuracy. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a method for automatically evaluating the accuracy of defect recognition software.

[0005] The present invention provides a method for automatically evaluating the accuracy of defect recognition software using the following technical solutions:

[0006] An embodiment of the present invention provides a method for automatically evaluating the accuracy of defect recognition software, the method comprising the following steps:

[0007] Collect abnormal data of the sensor, classify the abnormal data to obtain several fault types of the sensor and several abnormal data of each fault type;

[0008] Obtaining an abnormal data segment of each abnormal data according to the abnormal data; obtaining the degree of fluctuation of each abnormal data according to the amplitude of the data point in the abnormal data segment and the mean envelope of the abnormal data segment;

[0009] The sensitivity of the defect recognition software to each fault type is determined based on the fault type and the degree of fluctuation of different abnormal data of the same fault type. The similarity between different abnormal data and the degree of fluctuation of different abnormal data is determined. All abnormal data are clustered based on the similarity to obtain several clusters of abnormal data. The fault type of each cluster is determined based on the cluster and the abnormal data of the fault type. The possibility of error caused by the fault type of each cluster is determined based on the sensitivity of the fault type, the fault type of each cluster and the cluster.

[0010] The final accuracy of the defect identification software is obtained based on the possibility of error generated by the fault type of each cluster.

[0011] Furthermore, the step of obtaining the abnormal data segment of each abnormal data according to the abnormal data includes the following specific steps:

[0012] Any abnormal data of any fault type is recorded as target abnormal data; abnormal data points in the target abnormal data are obtained through defect identification software, and the abnormal data points in the target abnormal data are recorded as target abnormal data points; the data segment range determined by TH1 data points to the left of the target abnormal data point is used as an abnormal data segment of the target abnormal data; TH1 is a preset first value.

[0013] Furthermore, the step of obtaining the fluctuation degree of each abnormal data according to the amplitude of the data point in the abnormal data segment and the mean envelope of the abnormal data segment includes the following specific steps:

[0014] Record any abnormal data of any fault type as target abnormal data, record the abnormal data segment of the target abnormal data as target abnormal data segment; obtain the mean envelope of the target abnormal data segment, record it as target mean envelope;

[0015] The linearly normalized amplitude of the t-th data point in the target abnormal data segment is recorded as the first amplitude. The product of the absolute value of the difference between the slope of the corresponding point of the t-th data point in the target abnormal data segment in the target mean envelope and the slope of the corresponding point of the t+1-th data point in the target abnormal data segment in the target mean envelope, and the first amplitude, is recorded as the first parameter of the t-th data point in the target abnormal data segment. The average value of the first parameters of all data points in the target abnormal data segment is taken as the fluctuation degree of the target abnormal data segment.

[0016] The fluctuation degree of the target abnormal data segment is used as the fluctuation degree of the target abnormal data.

[0017] Furthermore, the sensitivity of the defect identification software to each fault type is obtained based on the fault type and the degree of fluctuation of different abnormal data of the same fault type, including the following specific steps:

[0018] Any fault type is recorded as the target fault type;

[0019] The ratio of the number of abnormal data of the target fault type to the number of abnormal data of all fault types is recorded as the first ratio, the absolute value of the difference in the degree of fluctuation between any two different abnormal data of the target fault type is recorded as a second parameter of the target fault type, and the inverse proportional value of the accumulated value of all the second parameters of the target fault type multiplied by the first ratio is used as the sensitivity of the defect identification software to the target fault type.

[0020] Furthermore, the similarity between different abnormal data is obtained based on different abnormal data and the degree of fluctuation of different abnormal data, including the following specific steps:

[0021] The inverse proportional value of the absolute value of the difference between the fluctuation degree of the a-th abnormal data and the fluctuation degree of the b-th abnormal data, and the product of the Pearson correlation coefficient of the a-th abnormal data and the b-th abnormal data are used as the similarity factor between the a-th abnormal data and the b-th abnormal data;

[0022] Obtain the similarity factors between all different abnormal data, perform linear normalization on the similarity factors between all different abnormal data, and use the obtained results as the similarity between different abnormal data.

[0023] Furthermore, clustering all abnormal data according to similarity to obtain several clusters of abnormal data includes the following specific steps:

[0024] K-means clustering is performed on all abnormal data, and the distance metric is the difference between 1 and the similarity between abnormal data, to obtain several clusters of abnormal data.

[0025] Furthermore, the method of obtaining the fault type of each cluster based on the abnormal data of the cluster and the fault type includes the following specific steps:

[0026]

[0027] Where, is the average value of the fluctuation degree of all abnormal data in the k-th cluster, is the average fluctuation degree of all abnormal data of the sth fault type, E3 s,w is the fluctuation degree of the wth abnormal data of the sth fault type, W s is the number of abnormal data of the sth fault type, f k,s is the similarity between the kth cluster and the sth fault type, and γ is a hyperparameter to avoid the denominator being 0;

[0028] Obtain the similarity between the kth cluster and each fault type, and take the fault type corresponding to the maximum similarity as the fault type of the kth cluster.

[0029] Furthermore, the method of obtaining the possibility of the fault type of each cluster causing an error based on the sensitivity of the fault type, the fault type of each cluster, and the cluster includes the following specific steps:

[0030]

[0031] Where, TN k is the number of fault types corresponding to the abnormal data in the kth cluster that are different from the fault types of the kth cluster, TN1 k is the number of abnormal data in the kth cluster; the fault type of the kth cluster is recorded as the first fault type, f1 k is the similarity between the kth cluster and the first fault type; P' k is the possible factor causing error in the fault type of the kth cluster, T k is the sensitivity corresponding to the fault type of the kth cluster;

[0032] The probability of error caused by the fault type of each cluster is obtained based on the possible factors.

[0033] Furthermore, the method of obtaining the possibility of the error generated by the fault type of each cluster based on the possible factors includes the following specific steps:

[0034] Obtain the possible factors of the error caused by the fault type of each cluster, perform linear normalization on the possible factors of the error caused by the fault type of all clusters, and use the obtained results as the possibility of the error caused by the fault type of each cluster.

[0035] Furthermore, the final accuracy of the defect identification software is obtained based on the probability of error generated by the fault type of each cluster, including the following specific steps:

[0036] Obtain the accuracy of the defect recognition software based on the confusion matrix, which is recorded as the initial accuracy of the defect recognition software;

[0037] The final accuracy of the defect recognition software is obtained by subtracting the difference between 1 and the average probability of the fault types of all clusters generating errors and multiplying it by the initial accuracy of the defect recognition software.

[0038] The beneficial effects of the technical solution of the present invention are as follows: the present invention collects abnormal data from the sensor, classifies the abnormal data to obtain several fault types of the sensor and several abnormal data of each fault type, obtains the abnormal data segment of each abnormal data through the abnormal data, and facilitates the subsequent accuracy evaluation; obtains the fluctuation degree of each abnormal data through the amplitude of the data point in the abnormal data segment and the mean envelope of the abnormal data segment, and uses this value as the change characteristic value of the abnormal data corresponding to the current fault, avoiding the problem of unclear data change regularity in subsequent analysis of the abnormal data corresponding to the same fault due to the change of a single analysis data value, thereby improving the authenticity of the accuracy evaluation, and then obtains the defect recognition software for each The sensitivity of the fault type is determined. This operation avoids the subsequent misjudgment of faults caused by the small difference in fault data when obtaining the recognition accuracy, improves the authenticity of the accuracy evaluation, clusters all abnormal data, and obtains the fault type of each cluster. Finally, according to the sensitivity of the fault type, the fault type of each cluster and the cluster, the possibility of the fault type of each cluster causing errors is obtained, and then the final accuracy of the defect recognition software is obtained based on the possibility of the fault type of each cluster causing errors. This solves the problem of accuracy distortion caused by the defect recognition software using historical sample data as a reference to obtain fault recognition accuracy, and improves the authenticity and accuracy of the accuracy evaluation of the defect recognition software. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flowchart of a method for automatically evaluating the accuracy of defect recognition software provided by one embodiment of the present invention;

[0041] Figure 2 A characteristic flow chart of obtaining the final accuracy of defect recognition software through abnormal data provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0042] To further illustrate the technical means and effectiveness of the present invention in achieving its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of a method for automatically evaluating the accuracy of defect recognition software proposed in accordance with the present invention. In the following description, references to different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0043] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0044] The following describes in detail a specific scheme of a method for automatically evaluating the accuracy of defect recognition software provided by the present invention with reference to the accompanying drawings.

[0045] See also Figure 1 and Figure 2 , which shows a flowchart of the steps of a method for automatically evaluating the accuracy of defect recognition software provided by an embodiment of the present invention and a characteristic flowchart of obtaining the final accuracy of the defect recognition software through abnormal data. The method includes the following steps:

[0046] Step S001: Collect abnormal data of the sensor, classify the abnormal data to obtain several fault types of the sensor and several abnormal data of each fault type.

[0047] Specifically, several abnormal data of any sensor are obtained, and the abnormal data are classified through manual quality inspection to obtain several fault types of the sensor and abnormal data of each fault type; it should be noted that each fault type corresponds to several abnormal data, each abnormal data contains several data points, and each data point corresponds to an amplitude and time (moment).

[0048] At this point, several fault types of the sensor and abnormal data for each fault type are obtained.

[0049] Step S002: Obtain an abnormal data segment of each abnormal data according to the abnormal data; obtain the fluctuation degree of each abnormal data according to the amplitude of the data point in the abnormal data segment and the mean envelope of the abnormal data segment.

[0050] It should be noted that due to the variety of fault types and the differences in the sensor data performance corresponding to different faults; and for the accuracy of software defect identification, the software will determine the fault type based on the data change characteristics, and estimate the accuracy of this determination based on historical fault data samples of various types; at this time, the greater the difference in the change characteristics of the sensor data corresponding to the same fault, the less the defect identification software is affected by the change of sensor data for the same type of fault, and the higher its accuracy, so first judge the degree of fluctuation between the various types of data obtained according to the fault type.

[0051] It should be noted that, since sensor data may have some precursory change characteristics when abnormal fluctuations occur, in order to more accurately represent the change characteristics of the data, for any abnormal data in any fault data set, the data point corresponding to the fault is located, and then the local data segment is obtained for fluctuation analysis.

[0052] Specifically, the abnormal data segment of each abnormal data is obtained according to the abnormal data, as follows:

[0053] Any abnormal data of any fault type is recorded as target abnormal data; abnormal data points in the target abnormal data are obtained through defect recognition software, and the abnormal data points in the target abnormal data are recorded as target abnormal data points; it should be noted that one abnormal data only contains one abnormal data point; the data segment range determined by TH1 data points to the left of the target abnormal data point is used as an abnormal data segment of the target abnormal data; TH1 is a preset first value, and TH1=10 in this embodiment; it should be noted that obtaining abnormal data points in the target abnormal data through defect recognition software is an existing method, and this embodiment will not be repeated. The TH1 data points to the left of the target abnormal data point are adjacent TH1 data points. For example, if the target abnormal data point is the 100th data point, the TH1 data points on the left are the 90th data point to the 99th data point. If the abnormal data point is on the left of the abnormal data, then when obtaining the abnormal data segment, the abnormal data segment will exceed the range of the abnormal data, and the abnormal data segment is constituted by the actual abnormal data points.

[0054] Furthermore, based on the amplitude of the data points in the abnormal data segment and the mean envelope of the abnormal data segment, the fluctuation degree of each abnormal data is obtained, as follows:

[0055] The abnormal data segment of the target abnormal data is recorded as the target abnormal data segment; the mean envelope of the target abnormal data segment is obtained, which is recorded as the target mean envelope; it should be noted that obtaining the mean envelope of the target abnormal data segment is an existing method and will not be repeated in this embodiment.

[0056]

[0057] Where n is the number of data points in the target abnormal data segment, q t is the amplitude of the tth data point in the target abnormal data segment, q min is the minimum amplitude of the data points in the target abnormal data segment, q max is the maximum amplitude of the data points in the target abnormal data segment, A t is the slope of the corresponding point of the t-th data point of the target abnormal data segment in the target mean envelope; it should be noted that obtaining the slope of the midpoint of the target mean envelope is an existing method and will not be repeated in this embodiment; A t+1 is the slope of the corresponding point of the t+1th data point of the target abnormal data segment in the target mean envelope, || is the absolute value, E is the fluctuation degree of the target abnormal data segment, and α is a hyperparameter to avoid the denominator being 0. This embodiment is described as α=1.

[0058] The fluctuation degree of the target abnormal data segment is used as the fluctuation degree of the target abnormal data.

[0059] It should be noted that Represents the normalized result of the amplitude of the t-th data point. The larger the value, the more prominent the target abnormal data segment is and the higher the degree of expression of the data characteristics of the target abnormal data segment is. t -A t+1 | represents the slope difference between the corresponding points of the target mean envelope of two adjacent data points in the target abnormal data segment. This difference value represents the degree of fluctuation of the data value baseline. The larger the value, the higher the degree of performance of the data features in the target abnormal data segment. Traverse all data points in the target abnormal data segment and obtain The larger the value is, the more obvious the fluctuation characteristics of the target abnormal data segment are, and it also indicates that the fluctuation degree of the abnormal data of the sensor is greater. By defining the fluctuation degree of the abnormal data segment based on the data changes and baseline changes within the abnormal data segment of the abnormal data of the sensor, and using this value as the change characteristic value of the sensor data corresponding to a certain fault, it avoids the lack of obvious regularity of data changes in subsequent analyses of data changes corresponding to the same fault due to changes in a single analysis of data value, thereby improving the authenticity of the accuracy assessment.

[0060] At this point, the fluctuation degree of each abnormal data is obtained.

[0061] Step S003: Obtain the sensitivity of the defect recognition software to each fault type based on the fault type and the degree of fluctuation of different abnormal data of the same fault type; cluster all abnormal data to obtain several clusters of abnormal data; obtain the fault type of each cluster; and obtain the possibility of error in the fault type of each cluster based on the sensitivity of the fault type, the fault type of each cluster, and the cluster.

[0062] It should be noted that, since the fault classification is carried out through human participation and the fluctuation degree of each abnormal data is obtained according to the above steps, for the defect identification software, the greater the difference in the fluctuation characteristics of various types of data in any fault, the lower the sensitivity to the identification of this type of fault, indicating that the fault caused by the current fault has more obvious characteristics; on the contrary, if the difference in the fluctuation characteristics is smaller, it means that the sensitivity to this type is higher; at the same time, for any fault, the higher the frequency of its occurrence, the greater the possibility of errors in this type of fault when using software for abnormal identification, so it is necessary to first obtain the sensitivity of the software to the current fault type.

[0063] It should be noted that, since the types of failures that may occur in any sensor are diverse, the total number of sensor failures in the collected historical data is counted, and the number of occurrences of any sensor failure is obtained at the same time, thereby obtaining the probability of occurrence of the current fault type; then, according to the above logic, for any sensor failure data, the greater the difference in data fluctuation performance, the lower the sensitivity of the defect recognition software to this type of failure. Therefore, by calculating the difference between multiple abnormal data corresponding to any failure, combined with the above-mentioned failure occurrence probability, the sensitivity of the current defect recognition software to the current type of failure is judged.

[0064] Specifically, the sensitivity of the defect recognition software to each fault type is obtained based on the fault type and the degree of fluctuation of different abnormal data of the same fault type, as follows:

[0065]

[0066] Where N j is the number of abnormal data of the target fault type, M is the number of abnormal data of all fault types, E1 i is the fluctuation degree of the i-th abnormal data of the target fault type, E1 j is the fluctuation degree of the j-th abnormal data of the target fault type, || is the absolute value, T is the sensitivity of the defect recognition software to the target fault type, and β is a hyperparameter to avoid the denominator being 0. This embodiment is described as β=1.

[0067] It should be noted that Indicates the frequency of occurrence of the target fault type. The larger the value, the higher the probability of the target fault type occurring. It also indicates that the defect identification software is more likely to make errors in fault identification. i -E1 j| represents the difference in the degree of fluctuation between any two abnormal data of the target fault type. The larger the value, the greater the difference. At this time, all abnormal data are traversed to obtain the difference in the degree of fluctuation of all abnormal data of the target fault type. The larger the value, the greater the difference in abnormal data of the target fault type, and the lower the sensitivity of the defect recognition software to the target fault type. For the target fault type, if the fluctuation degree of its abnormal data is large and its occurrence frequency is high, the probability of error is also greater. Therefore, the fluctuation degree difference is weighted by frequency to obtain T. The larger the value, the greater the sensitivity of the defect recognition software to the target fault type. By calculating the occurrence frequency of any fault type and combining the fluctuation degree difference of abnormal data, the sensitivity of the defect recognition software to the fault type is judged when the fault type occurs. This operation avoids the subsequent misjudgment of faults caused by the small difference in abnormal data when obtaining the recognition accuracy, thereby improving the authenticity of the accuracy evaluation.

[0068] It should be noted that if the defect identification software is less sensitive to the current fault, the adaptive identification accuracy of the fault type will be lower when the new abnormal data is compared in the database. As for the abnormal data itself, theoretically, the smaller the difference in the fluctuation degree of the abnormal data, the greater the possibility that it will show the same fault. However, there are also large differences in the fluctuation degree of abnormal data corresponding to the same fault. At the same time, when using the confusion matrix to calculate the accuracy of fault identification, it only indicates the correct or incorrect performance of the software in the historical sample data, which cannot accurately represent the current fault identification accuracy. It can only be used as a reference for the accuracy base. Therefore, it is necessary to calculate the probability of misjudgment in the fault identification process through the difference performance between the abnormal data and the sensitivity calculated above.

[0069] Specifically, based on different abnormal data and the degree of fluctuation of different abnormal data, the similarity between different abnormal data is obtained, as follows:

[0070]

[0071] Where, E2 a is the fluctuation degree of the a-th abnormal data, E2 b is the fluctuation degree of the bth abnormal data, c is a preset hyperparameter, the purpose of which is to prevent the denominator from being 0. In this embodiment, c=0.1 is used for description. PC a,b is the Pearson correlation coefficient between the a-th anomaly data and the b-th anomaly data, and S' is the similarity factor between the a-th anomaly data and the b-th anomaly data.

[0072] Obtain the similarity factors between all different abnormal data, perform linear normalization on the similarity factors between all different abnormal data, and use the obtained results as the similarity between different abnormal data.

[0073] It should be noted that PC a,b The value range is [-1,1]. The closer the value is to 0, the smaller the correlation is. The closer it is to 1, the positive correlation is, and -1 is the negative correlation. Here, for the correlation between abnormal data, the closer it is to positive correlation, the higher the similarity is. Therefore, the larger the value is, the higher the correlation between the a-th abnormal data and the b-th abnormal data is. |E2 a -E2 b | represents the difference in the degree of fluctuation between any two abnormal data. The smaller the value, the more similar the data is. Therefore, it is expressed in fractions. At this time, the similarity judgment formula is obtained. The larger the value, the higher the similarity.

[0074] Furthermore, all abnormal data are clustered according to similarity to obtain several clusters of abnormal data, as follows:

[0075] K-means clustering is performed on all abnormal data, and the distance metric is the difference obtained by subtracting the similarity between the abnormal data from 1 to obtain several clusters of abnormal data. It should be noted that this embodiment uses the elbow method to obtain the K value of K-means clustering, and the specific acquisition method is the existing method.

[0076] Furthermore, the fault type of each cluster is obtained based on the abnormal data of the cluster and fault type, as follows:

[0077]

[0078] Where, is the average value of the fluctuation degree of all abnormal data in the k-th cluster, is the average fluctuation degree of all abnormal data of the sth fault type, E3 s,w is the fluctuation degree of the wth abnormal data of the sth fault type, W s is the number of abnormal data of the sth fault type, f k,s is the similarity between the k-th cluster and the s-th fault type, γ is a hyperparameter to avoid the denominator being 0, and this embodiment is described as γ=1.

[0079] Obtain the similarity between the kth cluster and each fault type, and take the fault type corresponding to the maximum similarity as the fault type of the kth cluster.

[0080] It should be noted that Indicates the matching degree between the fault type corresponding to the kth cluster and the sth fault type, The smaller it is, the higher the matching degree; It represents the average difference between the average value of the fluctuation degree of each abnormal data of the s-th fault type and all abnormal data in the k-th cluster. The smaller the value, the higher the matching degree between the s-th fault type and the fault type corresponding to the k-th cluster. Therefore, the inverse of the product of the two is taken, and the fault type corresponding to the maximum similarity is taken as the fault type of the k-th cluster.

[0081] Furthermore, based on the sensitivity of the fault type, the fault type of each cluster, and the cluster, the probability of the fault type of each cluster causing an error is obtained, as follows:

[0082]

[0083] Where, TN k is the number of fault types corresponding to the abnormal data in the kth cluster that are different from the fault types of the kth cluster, TN1 k is the number of abnormal data in the kth cluster; the fault type of the kth cluster is recorded as the first fault type, f1 k is the similarity between the kth cluster and the first fault type; P' k is the possible factor causing error in the fault type of the kth cluster, T k is the sensitivity corresponding to the fault type of the kth cluster.

[0084] Obtain the possible factors of the error caused by the fault type of each cluster, perform linear normalization on the possible factors of the error caused by the fault type of all clusters, and use the obtained results as the possibility of the error caused by the fault type of each cluster.

[0085] It should be noted that T k The larger the value, the greater the possibility of error. It represents the proportion of other fault types in the abnormal data of the kth cluster, that is, the fault types other than the fault types of the kth cluster. The larger the value, the greater the possibility of abnormal data being confused, and thus the greater the possibility of error. k The larger the value is, the higher the credibility is in the data comparison process. At the same time, the frequency itself is obtained through clustering results, so this value is used to compare weighted; thus, the above characteristics are transformed into the same dimension using the Euclidean norm, thus obtaining And normalize it to facilitate subsequent calculations, thereby obtaining the possibility of error in the fault type of each cluster. The larger the value, the lower the accuracy of fault identification should be.

[0086] It should be noted that the abnormal data are clustered according to the differences in the fluctuation characteristics of different abnormal data, and the corresponding fault type is obtained based on the clustering results. Finally, based on the difference between the data characteristics of the fault type and the data characteristics in the current cluster, the possibility of errors in identifying and obtaining the accuracy of this type of fault is judged. This operation avoids the distortion of the accuracy rate caused by only indicating that the correct or incorrect performance of the software in the historical sample data cannot accurately represent the current fault identification accuracy as a reference for the accuracy base, thereby improving the authenticity of the accuracy evaluation.

[0087] At this point, the probability of error caused by the fault type of each cluster is obtained.

[0088] Step S004: Obtain the final accuracy of the defect recognition software based on the possibility of error generation of the fault type of each cluster.

[0089] It should be noted that when using the confusion matrix to obtain the accuracy, it is usually necessary to obtain true positives (the software correctly identifies defects), true negatives (the software correctly identifies areas without defects), false positives (areas that the software mistakenly marks as defective), and false negatives (areas that the software mistakenly marks as non-defective). Then, the accuracy is obtained according to the calculation formula. At this time, the possibility of error is obtained according to the above steps. After using the recognition software to classify a certain type of sensor abnormal data, the value is used to weight the accuracy, thereby completing the correction of the defect recognition software accuracy assessment.

[0090] Specifically, the final accuracy of the defect identification software is obtained based on the probability of error generated by the fault type of each cluster, as follows:

[0091] The accuracy of the defect recognition software is obtained according to the confusion matrix, which is recorded as the initial accuracy of the defect recognition software. It should be noted that obtaining the accuracy of the defect recognition software according to the confusion matrix is an existing method and will not be repeated in this embodiment.

[0092]

[0093] Where Acc is the initial accuracy of the defect recognition software; is the average probability of error generation for all cluster fault types, and Acc' is the final accuracy of the defect identification software.

[0094] It should be noted that the greater the possibility of error in the fault type, the lower the accuracy of fault identification should be. Therefore, an inverse proportional processing is performed this time to correct the initial accuracy and obtain the final accuracy of the defect identification software.

[0095] At this point, the initial accuracy of the defect recognition software is corrected by the possibility of errors caused by the fault types of the cluster, completing the automatic evaluation of the accuracy of the defect recognition software.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for automatically evaluating the accuracy of defect recognition software, characterized in that: The method comprises the following steps: Collect abnormal data of the sensor, classify the abnormal data to obtain several fault types of the sensor and several abnormal data of each fault type; Obtaining an abnormal data segment of each abnormal data according to the abnormal data; obtaining the degree of fluctuation of each abnormal data according to the amplitude of the data point in the abnormal data segment and the mean envelope of the abnormal data segment; The sensitivity of the defect recognition software to each fault type is determined based on the fault type and the degree of fluctuation of different abnormal data of the same fault type. The similarity between different abnormal data and the degree of fluctuation of different abnormal data is determined. All abnormal data are clustered based on the similarity to obtain several clusters of abnormal data. The fault type of each cluster is determined based on the cluster and the abnormal data of the fault type. The possibility of error caused by the fault type of each cluster is determined based on the sensitivity of the fault type, the fault type of each cluster and the cluster. The final accuracy of the defect identification software is obtained based on the possibility of error generated by the fault type of each cluster.

2. The automatic evaluation method for the accuracy of defect recognition software according to claim 1, characterized in that: The specific steps of obtaining the abnormal data segment of each abnormal data according to the abnormal data are as follows: Record any abnormal data of any fault type as target abnormal data; obtain abnormal data points in the target abnormal data through defect recognition software, and record the abnormal data points in the target abnormal data as target abnormal data points; The data segment range determined by TH1 data points to the left of the target abnormal data point is used as an abnormal data segment of the target abnormal data; TH1 is a preset first value.

3. The automatic evaluation method for the accuracy of defect recognition software according to claim 1, characterized in that: The method of obtaining the fluctuation degree of each abnormal data according to the amplitude of the data point in the abnormal data segment and the mean envelope of the abnormal data segment includes the following specific steps: Record any abnormal data of any fault type as target abnormal data, record the abnormal data segment of the target abnormal data as target abnormal data segment; obtain the mean envelope of the target abnormal data segment, record it as target mean envelope; The linearly normalized amplitude of the t-th data point in the target abnormal data segment is recorded as the first amplitude. The product of the absolute value of the difference between the slope of the corresponding point of the t-th data point in the target abnormal data segment in the target mean envelope and the slope of the corresponding point of the t+1-th data point in the target abnormal data segment in the target mean envelope, and the first amplitude, is recorded as the first parameter of the t-th data point in the target abnormal data segment. The average value of the first parameters of all data points in the target abnormal data segment is taken as the fluctuation degree of the target abnormal data segment. The fluctuation degree of the target abnormal data segment is used as the fluctuation degree of the target abnormal data.

4. The automatic evaluation method for the accuracy of defect recognition software according to claim 1, characterized in that: The sensitivity of the defect recognition software to each fault type is obtained based on the fault type and the degree of fluctuation of different abnormal data of the same fault type, including the following specific steps: Any fault type is recorded as the target fault type; The ratio of the number of abnormal data of the target fault type to the number of abnormal data of all fault types is recorded as the first ratio, the absolute value of the difference in the degree of fluctuation between any two different abnormal data of the target fault type is recorded as a second parameter of the target fault type, and the inverse proportional value of the accumulated value of all the second parameters of the target fault type multiplied by the first ratio is used as the sensitivity of the defect identification software to the target fault type.

5. The automatic evaluation method for the accuracy of defect recognition software according to claim 1, characterized in that: The specific steps of obtaining the similarity between different abnormal data according to different abnormal data and the fluctuation degree of different abnormal data are as follows: The inverse proportional value of the absolute value of the difference between the fluctuation degree of the a-th abnormal data and the fluctuation degree of the b-th abnormal data, and the product of the Pearson correlation coefficient of the a-th abnormal data and the b-th abnormal data are used as the similarity factor between the a-th abnormal data and the b-th abnormal data; Obtain the similarity factors between all different abnormal data, perform linear normalization on the similarity factors between all different abnormal data, and use the obtained results as the similarity between different abnormal data.

6. The automatic evaluation method for the accuracy of defect recognition software according to claim 1, characterized in that: The specific steps of clustering all abnormal data according to similarity to obtain several clusters of abnormal data are as follows: K-means clustering is performed on all abnormal data, and the distance metric is the difference between 1 and the similarity between abnormal data, to obtain several clusters of abnormal data.

7. The automatic evaluation method for the accuracy of defect recognition software according to claim 1, characterized in that: The specific steps of obtaining the fault type of each cluster based on the abnormal data of the cluster and the fault type are as follows: Where, is the average value of the fluctuation degree of all abnormal data in the k-th cluster, is the average fluctuation degree of all abnormal data of the sth fault type, E3 s,w is the fluctuation degree of the wth abnormal data of the sth fault type, W s is the number of abnormal data of the sth fault type, f k,s is the similarity between the kth cluster and the sth fault type, and γ is a hyperparameter to avoid the denominator being 0; Obtain the similarity between the kth cluster and each fault type, and take the fault type corresponding to the maximum similarity as the fault type of the kth cluster.

8. The automatic evaluation method for the accuracy of defect recognition software according to claim 1, characterized in that: The method of obtaining the possibility of an error caused by the fault type of each cluster based on the sensitivity of the fault type, the fault type of each cluster, and the cluster includes the following specific steps: Where, TN k is the number of fault types corresponding to the abnormal data in the kth cluster that are different from the fault types of the kth cluster, TN1 k is the number of abnormal data in the kth cluster; the fault type of the kth cluster is recorded as the first fault type, f1 k is the similarity between the kth cluster and the first fault type; P' k is the possible factor causing error in the fault type of the kth cluster, T k is the sensitivity corresponding to the fault type of the kth cluster; The probability of error caused by the fault type of each cluster is obtained based on the possible factors.

9. The automatic evaluation method for the accuracy of defect recognition software according to claim 8, characterized in that: The method of obtaining the possibility of the error caused by the fault type of each cluster based on the possible factors includes the following specific steps: Obtain the possible factors of the error caused by the fault type of each cluster, perform linear normalization on the possible factors of the error caused by the fault type of all clusters, and use the obtained results as the possibility of the error caused by the fault type of each cluster.

10. The automatic evaluation method for the accuracy of defect recognition software according to claim 1, characterized in that: The final accuracy of the defect identification software is obtained based on the probability of error generated by the fault type of each cluster, including the following specific steps: Obtain the accuracy of the defect recognition software based on the confusion matrix, which is recorded as the initial accuracy of the defect recognition software; The final accuracy of the defect recognition software is obtained by subtracting the difference between 1 and the average probability of the fault types of all clusters generating errors and multiplying it by the initial accuracy of the defect recognition software.

Images