Regression data analysis method based on unsupervised learning, computer device and medium

Abstract

This application relates to the field of computer technology and provides a regression data analysis method, computer device, and medium based on unsupervised learning. The method includes: analyzing regression logs using a natural language processing model to determine the natural language analysis results associated with the regression logs; determining the error type associated with the regression logs based on the natural language analysis results associated with the regression logs using an unsupervised learning model; and determining whether the regression logs contain anomalous regression test cases based on the natural language analysis results associated with the regression logs using an anomaly detection model, and, when the regression logs contain the anomalous regression test cases, determining the error type associated with the anomalous regression test cases. This provides efficient error localization and saves on localization components, thus helping to improve verification efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a regression data analysis method, computer equipment, and media based on unsupervised learning. Background Technology

[0002] In applications such as data centers, large-scale AI models, automotive-grade chips, and high-performance memory, the complexity and scale of chip design are increasing daily, thus placing higher demands on chip design simulation and verification. Chip design simulation and verification relies on a large number of regression test cases and regression test data, requiring error detection and fault localization to provide a basis for code modification and design changes. However, existing regression data analysis methods either rely on manual analysis and report generation by verification personnel, leading to inefficiency, or on automated scripts for information aggregation and report generation, but such automated scripts require a sufficient number of verification project samples and data labels, making it difficult to adapt to the updates in chip design projects. Unsupervised learning, also known as unsupervised machine learning, can be used to analyze unlabeled datasets, discovering potential structures and patterns by analyzing the statistical characteristics and similarities of data without human intervention. However, existing unsupervised learning techniques are mainly developed for applications such as data analysis, recommendation systems, and natural language processing, without fully considering the differences and special characteristics of chip design applications, thus making it difficult to provide efficient error localization.

[0003] Therefore, this application provides a regression data analysis method, computer equipment and medium based on unsupervised learning, which can be applied to the simulation verification application scenario in chip design projects. It can provide efficient error localization and save on positioning components, thus helping to improve verification efficiency. Summary of the Invention

[0004] Firstly, this application provides a regression data analysis method based on unsupervised learning. The regression data analysis method includes: analyzing regression logs using a natural language processing model to determine the natural language analysis results associated with the regression logs, wherein the regression logs record regression data log information for each of the multiple regression test cases obtained by applying multiple regression test cases to the design under test, and the natural language analysis results associated with the regression logs include simulation results and key feature extraction results from the regression data log information of each of the multiple regression test cases; determining the error type associated with the regression logs based on the natural language analysis results associated with the regression logs using an unsupervised learning model, wherein the unsupervised learning model is trained to identify multiple error types, and the error type associated with the regression logs belongs to one or more of the multiple error types; and determining whether the regression logs include abnormal regression test cases using an anomaly detection model based on the natural language analysis results associated with the regression logs, and when the regression logs include the abnormal regression test cases, determining the error type associated with the abnormal regression test cases, wherein the error type associated with the abnormal regression test cases is different from each of the multiple error types.

[0005] Through the first aspect of this application, in the application scenario of chip design simulation verification, unsupervised learning is used to analyze the statistical characteristics and similarities of data to discover potential structures and patterns. By training an unsupervised learning model, various error types can be identified, thereby performing pattern recognition on error information in regression logs to determine specific error information and save time spent locating error use cases. It helps to determine whether subsequent error use cases are related to the current use case and helps to quickly determine whether related use cases are caused by the same error. By comparing error reporting types, multiple regressions and multiple localizations can be avoided. Considering the differences and special characteristics of chip design application scenarios, anomaly detection methods are applied to identify abnormal regression use cases and to investigate other error types that cannot be classified by clustering algorithms. Using anomaly detection models, patterns or events inconsistent with normal behavior or expected patterns can be found in the data. Thus, by using unsupervised learning methods, no human intervention is required, which can provide efficient error localization and save localization components, thereby helping to improve verification efficiency.

[0006] In one possible implementation of the first aspect of this application, the unsupervised learning model applies a clustering algorithm, which includes at least the K-means clustering algorithm.

[0007] In one possible implementation of the first aspect of this application, the anomaly detection model is to perform vector normalization on the natural language analysis results associated with the regression logs and then apply a standard score algorithm.

[0008] In one possible implementation of the first aspect of this application, the error type associated with the anomaly regression test case is determined by the anomaly detection model at least before the error type associated with the regression log is determined by the unsupervised learning model, and the anomaly regression test case determined by the anomaly detection model is excluded from the analysis objects of the unsupervised learning model.

[0009] In one possible implementation of the first aspect of this application, the error type associated with the regression log is determined by the unsupervised learning model based on the natural language analysis results associated with the regression log, including: when the simulation result in the regression data log information of the first regression test case in the plurality of regression test cases is a failure and the key feature extraction result in the regression data log information of the first regression test case includes the keywords ERROR and ACT / EXP, the error type associated with the regression log is determined to be an error type of chip output result error.

[0010] In one possible implementation of the first aspect of this application, determining the error type associated with the regression log based on the natural language analysis results associated with the regression log through the unsupervised learning model further includes: when the simulation result in the regression data log information of the second regression test case among the plurality of regression test cases fails and the key feature extraction result in the regression data log information of the second regression test case includes the ERROR keyword and the ASSERT keyword, determining that the error type associated with the regression log includes the error type of signal change error.

[0011] In one possible implementation of the first aspect of this application, determining the error type associated with the regression log based on the natural language analysis results associated with the regression log through the unsupervised learning model further includes: when the simulation result in the regression data log information of the third regression test case among the plurality of regression test cases is a failure and the key feature extraction result in the regression data log information of the third regression test case includes the keyword ERROR and the keyword "WATCHDOG TIMEOUT", determining that the error type associated with the regression log includes the error type of signal timeout error.

[0012] In one possible implementation of the first aspect of this application, the method of determining the error type associated with the regression log based on the natural language analysis results associated with the regression log through the unsupervised learning model further includes: when the simulation result in the regression data log information of the fourth regression test case among the plurality of regression test cases is a failure and the key feature extraction result in the regression data log information of the fourth regression test case includes the keywords ERROR and REGISTER, the method of determining the error type associated with the regression log includes the error type of register read / write error.

[0013] In one possible implementation of the first aspect of this application, determining the error type associated with the regression log based on the natural language analysis results associated with the regression log through the unsupervised learning model further includes: when the simulation result in the regression data log information of the fifth regression test case among the plurality of regression test cases fails and the key feature extraction result in the regression data log information of the fifth regression test case includes the keyword ERROR and the keyword "UVM HAS X", determining that the error type associated with the regression log includes the error type of metastable state error.

[0014] In one possible implementation of the first aspect of this application, the error type associated with the regression log is determined based on the natural language analysis results associated with the regression log through the unsupervised learning model, and further includes: when the simulation result in the regression data log information of the sixth regression test case among the plurality of regression test cases fails and the key feature extraction result in the regression data log information of the sixth regression test case includes the keyword ERROR and the keyword "PROJECT SCOREBOARD", the error type associated with the regression log is determined to be an error type of packet data comparison error.

[0015] In one possible implementation of the first aspect of this application, the anomaly detection model determines whether the regression log includes the anomalous regression test case based on the natural language analysis results associated with the regression log. Furthermore, when the regression log includes the anomalous regression test case, the error type associated with the anomalous regression test case is determined, including: when the simulation result in the regression data log information of the seventh regression test case among the plurality of regression test cases is missing, the key feature extraction result in the regression data log information of the seventh regression test case does not include the ERROR keyword, and the regression data log information of the seventh regression test case records changes in simulation time, the error type associated with the anomalous regression test case is determined to include a resource shortage error type.

[0016] In one possible implementation of the first aspect of this application, the anomaly detection model determines whether the regression log includes the anomalous regression test case based on the natural language analysis results associated with the regression log. Furthermore, when the regression log includes the anomalous regression test case, the error type associated with the anomalous regression test case is determined, including: when the key feature extraction result in the regression data log information of the eighth regression test case among the plurality of regression test cases includes the keyword "unexpected termination" or the keyword "hostname", the error type associated with the anomalous regression test case is determined to include a machine anomaly error type.

[0017] In one possible implementation of the first aspect of this application, the maximum test case simulation time of the plurality of regression test cases is obtained by quantile estimation of the historical test case simulation time data of the design under test.

[0018] Secondly, embodiments of this application also provide a computer device, the computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method according to any of the above-mentioned implementations.

[0019] Thirdly, embodiments of this application also provide a computer-readable storage medium storing computer instructions that, when executed on a computer device, cause the computer device to perform a method according to any of the above-described implementations.

[0020] Fourthly, embodiments of this application also provide a computer program product, the computer program product including instructions stored on a computer-readable storage medium, which, when executed on a computer device, cause the computer device to perform a method according to any of the above-described aspects. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a regression data analysis method based on unsupervised learning provided in this application embodiment;

[0023] Figure 2A schematic diagram of a regression data analysis tool provided in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0026] It should be understood that in the description of this application, "at least one" means one or more, and "multiple" means two or more. In addition, the words "first," "second," etc., unless otherwise stated, are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance or order.

[0027] Figure 1 This is a flowchart illustrating a regression data analysis method based on unsupervised learning, provided as an embodiment of this application. Figure 1 As shown, the regression data analysis method includes the following steps.

[0028] Step S101: Analyze the regression logs using a natural language processing model to determine the natural language analysis results associated with the regression logs. The regression logs record the regression data log information of each of the multiple regression test cases obtained by applying multiple regression test cases to the design under test. The natural language analysis results associated with the regression logs include the simulation results and key feature extraction results in the regression data log information of each of the multiple regression test cases.

[0029] Step S103: Using an unsupervised learning model, based on the natural language analysis results associated with the regression log, determine the error type associated with the regression log, wherein the unsupervised learning model is trained to identify multiple error types, and the error type associated with the regression log belongs to one or more of the multiple error types.

[0030] Step S105: Using the anomaly detection model, based on the natural language analysis results associated with the regression log, determine whether the regression log includes anomaly regression test cases. If the regression log includes the anomaly regression test cases, determine the error type associated with the anomaly regression test cases, wherein the error type associated with the anomaly regression test cases is different from each of the multiple error types.

[0031] Figure 1The regression data analysis method shown, in step S101, uses a natural language processing (NLP) model to analyze the regression logs and determine the associated NLP analysis results. Here, the NLP model utilizes NLP techniques to perform NLP analysis on the regression logs, obtaining the associated NLP analysis results. This facilitates subsequent error summarization and reporting, and clustering of vectorized error information to obtain clustering results. It also helps in applying anomaly detection methods to identify anomalous regression test cases and investigate other error types that cannot be classified by clustering algorithms. For example, timeouts and internal infinite loops are often machine-related, such as insufficient computing resources or task termination due to insufficient time. Therefore, using a NLP model helps identify specific error information, saving time spent locating erroneous test cases. Furthermore, it helps determine whether subsequent erroneous test cases are related to the current test case and helps quickly determine whether related test cases are caused by the same error. Furthermore, after identifying a series of erroneous use cases, and modifying the verification code or notifying the design team to modify the erroneous code, it's unnecessary to run the regression analysis again to determine if the modifications are complete. Instead, the regression data analysis method described above can be used to compare the reported error types to determine if the regression errors have been completely corrected, thus avoiding multiple regressions and re-identification. In this way, based on the regression data log information collected by the automated verification regression script, unsupervised learning methods are used to analyze the error information. This allows for a rapid summary of the regression error information, enabling more efficient error identification, saving identification time, and improving verification efficiency.

[0032] See Figure 1The regression log records the regression data log information of each regression test case obtained by applying multiple regression test cases to the design under test. The natural language analysis results associated with the regression log include simulation results and key feature extraction results from the regression data log information of each regression test case. Unsupervised learning is used to analyze the statistical characteristics and similarities of the data to discover potential structures and patterns. Therefore, with unsupervised learning, there is no need to worry about sample labeling; only the extraction of log features needs to be considered. In some embodiments, natural language processing models are used for natural language analysis. Corresponding error information can be extracted using regular expressions, or the log can be split using a lexical method, and the text can be parameterized and attributed to obtain the attributes of the regression log. Then, clustering can be performed, for example, using the K-means algorithm to divide the data into K different groups or clusters for data analysis and pattern recognition. Clustering algorithms can quickly and conveniently obtain clustering results, which can then be used to predict the approximate type of the error log, providing guidance, and simultaneously statistically analyzing the error distribution of the error test cases based on the results.

[0033] Continue reading Figure 1In step S103, based on the natural language analysis results associated with the regression logs, an unsupervised learning model is used to determine the error types associated with the regression logs. The unsupervised learning model is trained to identify multiple error types, and the error types associated with the regression logs belong to one or more of these error types. Thus, during training, an unsupervised learning model can be trained to identify multiple error types, for example, by using a K-class algorithm for keyword filtering to perform pattern recognition on error information in the regression logs. In some embodiments, the results can be presented to the user via command-line statistics, analyzing error use cases within existing regression summary information. In some embodiments, based on the entire regression analysis tool, natural language processing methods are used to extract key error information. Specifically, the natural language processing method can employ inverse maximum matching to extract valid statements from the error information. For example, an error log may contain multiple error messages. To ensure accurate location, the first error message can be extracted and subjected to inverse maximum matching to extract the corresponding error information. Then, the keywords of the error information are vectorized (word2vec) to obtain vectorized error information. In some examples, when the error messages are relatively simple, they can be considered as small models. Therefore, a Bag of Words (BOW) technique can be used to vectorize the keywords in the error messages and save the segmented error key information and sentences for individual viewing. Alternatively, the bag of words technique can be replaced with word embedding technology, which helps to achieve more advanced semantic understanding and error analysis.

[0034] Continue reading Figure 1In step S105, based on the natural language analysis results associated with the regression log, an anomaly detection model is used to determine whether the regression log includes anomalous regression test cases. If the regression log includes such anomalous regression test cases, the error type associated with those cases is determined. This error type is distinct from each of the multiple error types. Considering the clustering and classification of regression logs, some regression test cases are difficult to group into a single category. For example, when a regression test case enters an infinite loop, no obvious error message appears, but the feedback might be from a compilation simulation tool or other components, and not a user-defined error message. Therefore, it could be any error message that is clearly different from other test cases. Thus, an anomaly detection model is needed to identify anomalous regression test cases. Using an anomaly detection model, patterns or events inconsistent with normal behavior or expected patterns can be discovered from the data. In chip design simulation verification applications, anomaly detection models can identify errors different from typical error test cases. For example, errors caused by insufficient machine resources or errors generated by the compilation simulation tool itself. Such errors are clearly different from other error test cases with explicit error types. Furthermore, because it's impossible to extract phrases like "simulation result failed" or other judgment criteria from the regression test case logs, and the runtime logs may be abruptly interrupted or empty, these factors mean that logs related to anomalous regression test cases may have identifiable characteristics compared to logs of general error test cases. Therefore, after log vectorization, basic statistical methods can be used for anomaly detection, such as the Z-Score method (also called the Z-score algorithm or standard score algorithm). This relies on the assumption that the data follows a specific statistical distribution, such as outliers typically being distributed at extreme tails. Anomalies can then be identified based on the distance and standard deviation of data points from the mean. In some embodiments, after obtaining the vectors corresponding to the logs, the Z-Score anomaly detection method is used to detect anomalous information in the error logs. Generally, most error test cases contain error information, or the corresponding error information is in a bag of words. After standardizing the log vectors to obtain the mean and standard deviation, the Z-score value is calculated for each error log. If this value is too large, it indicates an anomalous log. Optionally, identified anomalous logs, or rather, identified anomalous regression test cases, may not participate in clustering. Therefore, after screening the anomalous logs, the remaining log vectors can be used for clustering, and the K-means algorithm can be directly used as the clustering method to obtain the clustering results.The K-means algorithm, as a clustering method, iteratively finds the centroid of each cluster and assigns each data point (corresponding to each error log) to the nearest cluster based on these centroids. Thus, utilizing unsupervised learning methods, it provides efficient error localization and saves on localization components without human intervention, thereby improving verification efficiency.

[0035] Continue reading Figure 1 In some embodiments, after obtaining the regression log set, the logs are classified, successful logs are filtered, and error logs are classified. Then, error logs can be clustered to obtain a summary of error attribution analysis, and anomaly detection can be performed to filter out abnormal logs. In some embodiments, error logs can be segmented and extracted; if no error information is found in the log, it can be inferred that it is a machine error or a compilation error. Furthermore, after segmentation and extraction, a thesaurus can be constructed to obtain the corresponding vectors for the logs. Specifically, for error log segmentation and extraction, for each log, a corresponding vector can be obtained based on the words contained in the error information; for example, thesaurus technology can be used to extract phrases such as "watchdog not reached," "repair failed," or "firmware sweep error."

[0036] In short, Figure 1 The regression data analysis method based on unsupervised learning, as shown, is applied in the simulation verification scenario of chip design. Utilizing unsupervised learning, it analyzes the statistical characteristics and similarities of data to discover potential structures and patterns. By training an unsupervised learning model, it identifies various error types, thereby performing pattern recognition on error information in regression logs to determine specific error information and save time spent locating error use cases. It helps determine whether subsequent error use cases are related to the current use case and quickly judges whether related use cases are caused by the same error. Comparison of error reporting types avoids multiple regressions and multiple localizations. Considering the differences and special characteristics of chip design applications, anomaly detection methods are applied to identify abnormal regression use cases and investigate other error types that cannot be classified by clustering algorithms. Using anomaly detection models, patterns or events inconsistent with normal behavior or expected patterns can be discovered from the data. Thus, by using unsupervised learning methods, no human intervention is required, providing efficient error localization and saving localization components, thereby improving verification efficiency.

[0037] Figure 2 This is a schematic diagram of a regression data analysis tool provided in an embodiment of this application. Figure 2As shown, the regression data analysis tool 200 includes a natural language processing model 201, an unsupervised learning model 202, and an anomaly detection model 203. Utilizing the regression data analysis tool 200 in the application scenario of chip design simulation verification, unsupervised learning is used to analyze the statistical characteristics and similarities of the data to discover potential structures and patterns. The unsupervised learning model 202 is trained to identify various error types, thereby performing pattern recognition on error information in the regression log, determining specific error information, and saving time spent locating error use cases. It helps determine whether subsequent error use cases are related to the current use case, and helps quickly determine whether related use cases are caused by the same error. Comparison of error reporting types can avoid multiple regressions and multiple localizations. Considering the differences and special characteristics of chip design application scenarios, anomaly detection methods are applied to identify abnormal regression use cases and investigate other error types that cannot be classified by clustering algorithms. Using the anomaly detection model, patterns or events inconsistent with normal behavior or expected patterns can be discovered from the data. Thus, using unsupervised learning methods, no human intervention is required, providing efficient error localization and saving localization components, which helps improve verification efficiency.

[0038] See Figure 1 and Figure 2 In one possible implementation, the unsupervised learning model applies a clustering algorithm, including at least K-means clustering. Thus, by utilizing unsupervised learning methods, no human intervention is required, which can provide efficient error localization and save on localization components, thereby improving verification efficiency.

[0039] In one possible implementation, the anomaly detection model performs vector standardization on the natural language analysis results associated with the regression logs before applying a standard score algorithm. Thus, using the anomaly detection model, patterns or events inconsistent with normal behavior or expected patterns can be discovered from the data. After obtaining the vectors corresponding to the logs, the Z-Score anomaly detection method is used to detect anomalous information in the error logs. For example, most error test cases have error messages, or the corresponding error messages are in a bag of words. After standardizing the log vectors, the mean and standard deviation are obtained. Then, the Z-score (zscore) value is calculated for each error log. If this value is too large, it indicates that it is an anomalous log.

[0040] In one possible implementation, the error type associated with the anomalous regression test case is determined by the anomaly detection model, at least before the error type associated with the regression log is determined by the unsupervised learning model. Furthermore, the anomalous regression test cases determined by the anomaly detection model are excluded from the analysis of the unsupervised learning model. Thus, considering the clustering and classification of regression logs, some regression test cases are difficult to group into a single category using clustering methods. For example, when a regression test case gets stuck in an infinite loop, no obvious error message appears, but the feedback information may be from a compilation simulation tool or other components, and it is not a user-defined error message. Therefore, it could be any other error message that is clearly abnormal from other test cases. Therefore, an anomaly detection model is needed to identify anomalous regression test cases. Using the anomaly detection model, patterns or events inconsistent with normal behavior or expected patterns can be discovered from the data. After obtaining the regression log set, the logs are classified, successful logs are filtered, and error logs are classified. Then, error logs can be clustered to obtain a summary of error attribution analysis, and anomaly detection can be performed on the error logs to filter out anomalous logs. This improves overall efficiency.

[0041] In one possible implementation, the unsupervised learning model, based on the natural language analysis results associated with the regression logs, determines the error type associated with the regression logs. This includes determining the error type associated with the regression logs as an error type where the simulation result in the regression log information of the first regression test case out of the plurality of regression test cases is a failure, and the key feature extraction results in the regression log information of the first regression test case include the keywords ERROR and ACT / EXP. Thus, clustering algorithms such as K-means can be used to perform pattern recognition from the error information in the regression logs through keyword filtering. Here, taking the first regression test case as an example, the corresponding error type is the error type where the chip output result is a failure. Specifically, the regression log contains the keyword ERROR and the keyword ACT / EXP, and the log ends with a simulation result failure. This type of error is generally discovered by chip verification personnel who check the output results of the design under test and find errors that do not meet expectations, such as comparing with the output results of a reference model or with expected results set in other ways. In this way, by training an unsupervised learning model to identify multiple error types, pattern recognition can be performed on error information in regression logs to determine specific error information, saving time spent locating error use cases. It helps to determine whether subsequent error use cases are related to the current use case, and helps to quickly determine whether related use cases are caused by the same error. By comparing error reporting types, multiple regressions and multiple localizations can be avoided.

[0042] In one possible implementation, the error type associated with the regression log is determined based on the natural language analysis results associated with the regression log using the unsupervised learning model. This further includes determining that the error type associated with the regression log includes a signal change error when the simulation result in the regression data log information of the second regression test case out of the plurality of regression test cases fails and the key feature extraction result in the regression data log information of the second regression test case includes the ERROR keyword and the ASSERT keyword. Thus, a clustering algorithm such as the K-means algorithm can be used to perform pattern recognition from the error information in the regression log through keyword filtering. Here, taking the second regression test case as an example, the corresponding error type is a signal change error. Specifically, the regression log contains both the ERROR keyword and the ASSERT keyword. This type of error is generally generated by automatically checking whether signal changes meet expectations in assertion statements in hardware description languages ​​(e.g., System Verilog). It can be considered a type of assertion check error. Assertion statements, sometimes called declaration statements, can check the occurrence of specific conditions or event sequences. Assertions can be used to check whether changes in specific signals conform to expected signal declarations. For example, a logic level signal, such as a reset signal, should be high according to the design, but the output is found to be low. Error types related to signal change errors involve the logic control aspects of chip design. Therefore, by training an unsupervised learning model to identify various error types, pattern recognition can be performed on error information in regression logs to determine specific error information, saving time spent locating erroneous use cases. This helps determine whether subsequent erroneous use cases are related to the current use case, and helps quickly determine whether related use cases are caused by the same error. By comparing error reporting types, multiple regressions and multiple localizations can be avoided.

[0043] In one possible implementation, the error type associated with the regression log is determined based on the natural language analysis results associated with the regression log using the unsupervised learning model. This further includes determining the error type associated with the regression log as a signal timeout error when the simulation result in the regression data log information of the third regression test case out of the plurality of regression test cases is a failure and the key feature extraction result in the regression data log information of the third regression test case includes the keywords ERROR and "WATCHDOGTIMEOUT". Thus, clustering algorithms such as K-means can be used to perform pattern recognition from the error information in the regression log through keyword filtering. Here, taking the third regression test case as an example, the corresponding error type is a signal timeout error. Specifically, the regression log contains the keyword ERROR and the keyword WATCHDOG TIMEOUT. This type of error generally occurs when the simulation verification environment waits for a valid signal but fails to receive a valid signal within a set time. This can be considered a type of simulation verification environment timeout error. For example, no valid signal is received after a predetermined time has elapsed, or the signal remains unchanged after a predetermined time in an invalid state. In this way, by training an unsupervised learning model to identify multiple error types, pattern recognition can be performed on error information in regression logs to determine specific error information, saving time spent locating error use cases. It helps to determine whether subsequent error use cases are related to the current use case, and helps to quickly determine whether related use cases are caused by the same error. By comparing error reporting types, multiple regressions and multiple localizations can be avoided.

[0044] In one possible implementation, the error type associated with the regression log is determined based on the natural language analysis results associated with the regression log using the unsupervised learning model. This further includes determining the error type associated with the regression log as a register read / write error when the simulation result in the regression log information of the fourth regression test case out of the multiple regression test cases is a failure and the key feature extraction results in the regression log information of the fourth regression test case include the keywords ERROR and REGISTER. Thus, clustering algorithms such as K-means can be used to perform pattern recognition from the error information in the regression log through keyword filtering. Here, taking the fourth regression test case as an example, the corresponding error type is a register read / write error. Specifically, the regression log contains both the ERROR and REGISTER keywords. This type of error generally indicates an unexpected error during register read / write operations. The K-means algorithm classifies these as register check errors, meaning testers need to locate register transfer-level errors. Examples include write failures or empty reads, caused by register code errors. In this way, by training an unsupervised learning model to identify multiple error types, pattern recognition can be performed on error information in regression logs to determine specific error information, saving time spent locating error use cases. It helps to determine whether subsequent error use cases are related to the current use case, and helps to quickly determine whether related use cases are caused by the same error. By comparing error reporting types, multiple regressions and multiple localizations can be avoided.

[0045] In one possible implementation, the error type associated with the regression log is determined based on the natural language analysis results associated with the regression log using the unsupervised learning model. This further includes determining that the error type associated with the regression log is a metastable error when the simulation result in the regression log information of the fifth regression test case out of the plurality of regression test cases is a failure and the key feature extraction result in the regression log information of the fifth regression test case includes the keywords ERROR and "UVM HASX". Thus, a clustering algorithm such as the K-means algorithm can be used to perform pattern recognition from the error information in the regression log through keyword filtering. Here, taking the fifth regression test case as an example, the corresponding error type is a metastable error. The metastable error type corresponds to metastability under a general verification methodology, or in other words, metastability under the definition of any verification methodology. Specifically, the regression log contains the keyword ERROR and the keywords UVM HASX appear. This type of error typically occurs during simulation when register codes exhibit an X-state. The K-means algorithm categorizes these as X-state check errors (XZ CHECK ERROR), meaning testers need to pinpoint the cause of this unstable signal. For example, a single-bit signal is either 0 or 1, while metastability refers to the inability to determine whether it's 0 or 1. Metastability can exist in system signal states or device states, including register metastability. Therefore, by training an unsupervised learning model to identify various error types, pattern recognition can be performed on error information in regression logs to determine specific error information, saving time spent locating erroneous test cases. This helps determine whether subsequent erroneous test cases are related to the current test case and quickly identify whether related test cases are caused by the same error. Comparing error reporting types can avoid multiple regressions and repeated localization.

[0046] In one possible implementation, the error type associated with the regression log is determined based on the natural language analysis results associated with the regression log using the unsupervised learning model. This further includes determining the error type associated with the regression log as a packet data alignment error when the simulation result in the regression data log information of the sixth regression test case out of the plurality of regression test cases is a failure and the key feature extraction result in the regression data log information of the sixth regression test case includes the keywords "ERROR" and "PROJECTSCOREBOARD". Thus, clustering algorithms such as the K-means algorithm can be used to perform pattern recognition from the error information in the regression log through keyword filtering. Here, taking the sixth regression test case as an example, the corresponding error type is a packet data alignment error. Specifically, the regression log contains the keyword "ERROR" and the keyword "PROJECTSCOREBOARD". This type of error is generally a packet data alignment error, a critical step in chip verification. The K-means algorithm classifies these as packet check errors, meaning that testers need to locate the cause of the packet corruption after passing through the chip, such as register code errors. For example, the data sent by the sender is compared with the data processed by intermediate modules or nodes, such as data link errors or packet loss. In this way, by training an unsupervised learning model to identify various error types, pattern recognition can be performed on error information in regression logs to determine specific error information, saving time spent locating error use cases. This helps determine whether subsequent error use cases are related to the current use case and quickly judges whether related use cases are caused by the same error. Comparing error reporting types can avoid multiple regressions and multiple localization attempts.

[0047] In one possible implementation, the anomaly detection model, based on the natural language analysis results associated with the regression log, determines whether the regression log includes the anomalous regression test case. Furthermore, when the regression log includes the anomalous regression test case, the error type associated with the anomalous regression test case is determined, including: when the simulation result in the regression data log information of the seventh regression test case among the multiple regression test cases is missing, the key feature extraction result in the regression data log information of the seventh regression test case does not include the ERROR keyword, and the regression data log information of the seventh regression test case records changes in simulation time, the error type associated with the anomalous regression test case is determined to include a resource shortage error type. Thus, applying anomaly detection methods, such as Z-Score, can be applied to all test cases to identify anomalous regression test case logs in the current regression. This can be used to investigate other error types that cannot be classified by K-means, such as timeouts and internal infinite loops. These types of problems are often machine-related, such as insufficient computing resources or task termination due to insufficient time. There are also some test cases that cannot be simply categorized using clustering methods; these errors require special consideration. For example, when a test case gets stuck in an infinite loop and no obvious error message is displayed, but rather an error message from the simulation compilation tool or other components instead of a user-defined error message; or any other error message that is clearly different from other test cases; in this case, anomaly detection methods are needed to identify the abnormal test case. Here, insufficient machine computing resources, insufficient test time, etc., may result in timeout messages, repeated attempts to write to a certain area of ​​memory failing (insufficient storage resources), or repeated attempts to call a certain process failing (insufficient computing resources). By identifying such key error messages, taking the seventh regression test case as an example, they can serve as anomaly markers. Therefore, when analyzing the error log information of regression test cases, if one or more such anomaly markers appear, it can be considered to belong to other error types that cannot be classified according to conventional k-means, that is, it can be determined that the regression log includes the abnormal regression test case. Specifically, the regression log does not contain any ERROR keywords, and the log does not display simulation results at the end, meaning it does not show pass or fail; therefore, the simulation results are missing. However, the simulation time is variable; for example, the last time displayed in the simulation log differs from the previous time, which will be identified as a host timeout. This anomaly is generally caused by machine congestion or a slow simulation process, preventing the test cases from completing within the runtime set by the regression script, such as 24 hours. Chip verification personnel do not need to locate the problem; they can detect this anomaly from the regression log. For example, the log may show that the simulation suddenly stopped.The simulation test case timeout was caused by anomalies due to insufficient resources. Therefore, considering the differences and specificities in chip design application scenarios, anomaly detection methods are applied to identify anomalous regression test cases and to investigate other error types that cannot be classified by clustering algorithms. Anomaly detection models can discover patterns or events in the data that are inconsistent with normal behavior or expected patterns, which is beneficial for improving verification efficiency and quality in conjunction with unsupervised learning.

[0048] In one possible implementation, the anomaly detection model, based on the natural language analysis results associated with the regression log, determines whether the regression log includes the anomalous regression test case. Furthermore, when the regression log includes the anomalous regression test case, the error type associated with the anomalous regression test case is determined. This includes determining that the error type associated with the anomalous regression test case includes machine error types when the key feature extraction result in the regression data log information of the eighth regression test case among the multiple regression test cases includes the keywords "unexpected termination" or "hostname". Specifically, taking the eighth regression test case as an example, the regression log contains keywords such as "unexpected termination" and "hostname". This anomaly is identified as a machine error (host error), and chip verification personnel do not need to locate it; this anomaly can be detected from the regression log, for example, a machine error. Thus, considering the differences and special characteristics of chip design application scenarios, applying anomaly detection methods to determine anomalous regression test cases and investigate other error types that cannot be classified by clustering algorithms, and utilizing anomaly detection models, can discover patterns or events inconsistent with normal behavior or expected patterns from the data, which is beneficial for improving verification efficiency and quality in conjunction with unsupervised learning.

[0049] In one possible implementation, the maximum simulation time for the multiple regression test cases is obtained by estimating the quantiles of historical simulation time data for the design under test. This utilizes quantile estimation, a statistical method, to determine a reasonable simulation time. First, a histogram of the simulation time distribution over recent days is plotted. For example, with a timeout set to 10 hours, 95% of the test cases complete on the first day, and 90% complete on the second day. A completion rate model is then established using quantile estimation, setting the target completion rate to 100% and calculating a reasonable upper limit for the timeout, such as 15 hours as given by the algorithm. Finally, a model matching algorithm is used to identify hardware errors. This improves verification efficiency.

[0050] Figure 3This is a schematic diagram of a computing device 300 provided in an embodiment of this application. The computing device 300 includes one or more processors 310, a communication interface 320, and a memory 330. The processors 310, communication interface 320, and memory 330 are interconnected via a bus 340. Optionally, the computing device 300 may further include an input / output interface 350, which is connected to input / output devices for receiving user-set parameters, etc. The computing device 300 can be used to implement some or all of the functions of the device embodiment or system embodiment in the above-described embodiments of this application; the processor 310 can also be used to implement some or all of the operation steps of the method embodiment in the above-described embodiments of this application. For example, the specific implementation of various operations performed by the computing device 300 can be referred to the specific details in the above embodiments, such as the processor 310 being used to execute some or all of the steps or operations in the above-described method embodiments. For example, in the embodiments of this application, the computing device 300 can be used to implement some or all of the functions of one or more components in the above-described device embodiments. In addition, the communication interface 320 can be used specifically for communication functions necessary to implement the functions of these devices and components, and the processor 310 can be used specifically for processing functions necessary to implement the functions of these devices and components.

[0051] It should be understood that, Figure 3 The computing device 300 may include one or more processors 310, and the multiple processors 310 may collaboratively provide processing power in a parallel connection mode, a serial connection mode, a serial-parallel connection mode, or an arbitrary connection mode; or the multiple processors 310 may form a processor sequence or a processor array; or the multiple processors 310 may be divided into a main processor and an auxiliary processor; or the multiple processors 310 may have different architectures, such as adopting a heterogeneous computing architecture. Furthermore, Figure 3 The structural and functional descriptions of the computing device 300 shown are exemplary and non-limiting. In some exemplary embodiments, the computing device 300 may include... Figure 3 The diagram shows more or fewer components, or combinations of some components, or splitting of some components, or different arrangements of components.

[0052] The processor 310 can have various specific implementations. For example, it may include one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or a data processing unit (DPU). This application does not impose specific limitations on these embodiments. The processor 310 can also be a single-core or multi-core processor. The processor 310 can be a combination of a CPU and hardware chips. These hardware chips can be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLDs can be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof. The processor 310 can also be implemented using logic devices with built-in processing logic, such as FPGAs or digital signal processors (DSPs). The communication interface 320 can be a wired interface or a wireless interface, used to communicate with other modules or devices. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., and the wireless interface can be a cellular network interface or a wireless LAN interface, etc.

[0053] Memory 330 may be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 330 may also be volatile memory, which may be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). The memory 330 can also be used to store program code and data, so that the processor 310 can call the program code stored in the memory 330 to execute some or all of the operation steps in the above method embodiments, or to execute the corresponding functions in the above device embodiments. Furthermore, the computing device 300 may include, compared to... Figure 3 The number of components displayed may be more or less, or there may be different component configurations.

[0054] Bus 340 can be a Peripheral Component Interconnect Express (PCIe) bus, or an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. Bus 340 can be divided into address bus, data bus, control bus, etc. In addition to the data bus, bus 340 can also include a power bus, a control bus, and a status signal bus. However, for clarity, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0055] The methods and devices provided in this application are based on the same inventive concept. Since the principles by which the methods and devices solve problems are similar, the embodiments, implementation methods, examples, or methods of implementation of the methods and devices can be referred to each other, and repeated details will not be repeated. This application also provides a system comprising multiple computing devices, the structure of each computing device of which can refer to the structure of the computing devices described above. The functions or operations achievable by this system can refer to the specific implementation steps in the above method embodiments and / or the specific functions described in the above device embodiments, and will not be repeated here.

[0056] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed on a computer device (such as one or more processors), they can implement the method steps described in the above method embodiments. The specific implementation of the above method steps by the processor of the computer-readable storage medium can refer to the specific operations described in the above method embodiments and / or the specific functions described in the above device embodiments, and will not be repeated here.

[0057] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. This application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Embodiments of this application can be implemented wholly or partially by software, hardware, firmware, or any other combination. When implemented in software, the above embodiments can be implemented wholly or partially as a computer program product. This application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that contains one or more sets of available media. Available media can be magnetic media (such as floppy disks, hard disks, and magnetic tapes), optical media, or semiconductor media. Semiconductor media can be solid-state drives, random access memory, flash memory, read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, or any other suitable form of storage medium.

[0058] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. Each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. The steps in the methods of the embodiments of this application can be adjusted in order, combined, or deleted according to actual needs; the modules in the systems of the embodiments of this application can be divided, combined, or deleted according to actual needs. If these modifications and variations of the embodiments of this application fall within the scope of the claims of this application and their equivalents, then this application also intends to include these modifications and variations.

Claims

1. A method of regression data analysis based on unsupervised learning, characterized in that, The regression data analysis method comprises: analyzing the regression log by a natural language processing model to determine natural language analysis results associated with the regression log, wherein the regression log records regression data log information of each of a plurality of regression test cases applied to a design under test, and the natural language analysis results associated with the regression log include simulation results and key feature extraction results in the regression data log information of each of the plurality of regression test cases; determining an error type associated with the regression log based on the natural language analysis results associated with the regression log by an unsupervised learning model, wherein the unsupervised learning model is trained to identify a plurality of error types, and the error type associated with the regression log belongs to one or more of the plurality of error types; and determining whether an abnormal regression test case is included in the regression log based on the natural language analysis results associated with the regression log by an anomaly detection model, and determining an error type associated with the abnormal regression test case when the abnormal regression test case is included in the regression log, wherein the error type associated with the abnormal regression test case is different from each of the plurality of error types.

2. The regression data analysis method according to claim 1, characterized by, The unsupervised learning model is a clustering algorithm, and the clustering algorithm at least comprises a K-means clustering algorithm.

3. The regression data analysis method of claim 1, wherein, The anomaly detection model is a standard score algorithm applied after a vector normalization operation on the natural language analysis results associated with the regression log.

4. The regression data analysis method of claim 1, wherein, Determining the error type associated with the abnormal regression test case by the anomaly detection model occurs at least before determining the error type associated with the regression log by the unsupervised learning model, and the abnormal regression test case determined by the anomaly detection model is excluded from the analysis object of the unsupervised learning model.

5. The regression data analysis method of claim 1, wherein, Determining the error type associated with the regression log based on the natural language analysis results associated with the regression log by the unsupervised learning model comprises: when a simulation result in regression data log information of a first regression test case in the plurality of regression test cases is a failure and key feature extraction results in the regression data log information of the first regression test case include an ERROR keyword and an ACT / EXP keyword, determining that the error type associated with the regression log includes a chip output result error type.

6. The regression data analysis method according to claim 5, wherein, Determining the error type associated with the regression log based on the natural language analysis results associated with the regression log by the unsupervised learning model further comprises: when a simulation result in regression data log information of a second regression test case in the plurality of regression test cases is a failure and key feature extraction results in the regression data log information of the second regression test case include the ERROR keyword and an ASSERT keyword, determining that the error type associated with the regression log includes a signal change error type.

7. The regression data analysis method according to claim 6, characterized in that, The determining, by the unsupervised learning model, the error type associated with the regression log based on the natural language analysis result associated with the regression log further includes: When the simulation result in the regression data log information of a third regression test case in the plurality of regression test cases is a failure and the key feature extraction result in the regression data log information of the third regression test case includes an ERROR keyword and a "WATCHDOG TIMEOUT" keyword, the determining the error type associated with the regression log includes an error type of a signal timeout error.

8. The regression data analysis method of claim 7, wherein, The determining, by the unsupervised learning model, the error type associated with the regression log based on the natural language analysis result associated with the regression log further includes: When the simulation result in the regression data log information of a fourth regression test case in the plurality of regression test cases is a failure and the key feature extraction result in the regression data log information of the fourth regression test case includes an ERROR keyword and a REGISTER keyword, the determining the error type associated with the regression log includes an error type of a register read-write error.

9. The regression data analysis method according to claim 8, characterized by, The determining, by the unsupervised learning model, the error type associated with the regression log based on the natural language analysis result associated with the regression log further includes: When the simulation result in the regression data log information of a fifth regression test case in the plurality of regression test cases is a failure and the key feature extraction result in the regression data log information of the fifth regression test case includes an ERROR keyword and a "UVM HAS X" keyword, the determining the error type associated with the regression log includes an error type of a metastable error.

10. The regression data analysis method of claim 9, wherein, The determining, by the unsupervised learning model, the error type associated with the regression log based on the natural language analysis result associated with the regression log further includes: When the simulation result in the regression data log information of a sixth regression test case in the plurality of regression test cases is a failure and the key feature extraction result in the regression data log information of the sixth regression test case includes an ERROR keyword and a "PROJECT SCOREBOARD" keyword, the determining the error type associated with the regression log includes an error type of a packet data comparison error.

11. The regression data analysis method of claim 10, wherein, The determining, by the unsupervised learning model, the error type associated with the regression log based on the natural language analysis result associated with the regression log further includes: When the simulation result in the regression data log information of a seventh regression test case in the plurality of regression test cases is missing, the key feature extraction result in the regression data log information of the seventh regression test case does not include an ERROR keyword, and the regression data log information of the seventh regression test case records a change in simulation time, the determining the error type associated with the abnormal regression test case includes an error type of a resource shortage.

12. The regression data analysis method of claim 11, wherein, The abnormality detection model is used to determine whether the abnormal regression test case is included in the regression log based on the natural language analysis result associated with the regression log, and when the abnormal regression test case is included in the regression log, determine an error type associated with the abnormal regression test case, including: When the key feature extraction result in the regression data log information of the eighth regression test case in the plurality of regression test cases includes an "unexpected termination" keyword or a "hostname" keyword, it is determined that the error type associated with the abnormal regression test case includes an error type of machine abnormality.

13. The regression data analysis method of claim 11, wherein, The maximum test case simulation time of the plurality of regression test cases is obtained by quantile estimation on historical test case simulation time data of the design under test.

14. A computer device, comprising: The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the method according to any one of claims 1-13 when executing the computer program.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, which, when executed on a computer device, cause the computer device to perform the method according to any one of claims 1-13.

Images