A method for automatically registering test records

Abstract

The present invention discloses a method for automatically registering test records. S1, a test result confirmation module confirms whether a test result is correct by querying whether there is a marked correct end file under a test subject directory; S2, an error stage identification module determines the stage at which an error occurs, whether it is a compilation error, a link error or an operation error, by querying the compilation output log file and the operation output log file in each test subject directory under the directory structure; S3, an error type determination module confirms a specific error type by identifying keyword information in an error message; S4, an information screening registration module screens out specific error information and automatically registers it into a warehouse; S5, a record supplement registration module manually supplements and registers the error cause that cannot be automatically written, and uses a submit storage command to store the information into a warehouse. The present invention can effectively solve the problem of low efficiency caused by manual arrangement of test records.

Description

Technical Field

[0001] The invention relates to a test record automatic registration method, belonging to the technical field of compiler testing. Background Art

[0002] The correctness and robustness of the parallel compiler are closely related to the granularity and scale of the parallel compiler test. The parallel compiler test usually includes syntax extension test, compilation optimization test and runtime library function test. Syntax extension test needs to perform functional coverage test for each syntax function point, including the declaration and use of extension types, the recognition and processing of compilation instructions, the parsing and transformation of keywords, etc. Compilation optimization test needs to perform correctness test and optimization effect test for each compilation optimization technology; runtime library function test needs to perform parameter combination test for each runtime API function, and try to test multiple possible value combinations of multiple parameters, and strive to achieve full coverage test of function paths. Because the design of the runtime library is related to the parallel scale of the system, it is necessary to perform multiple tests at different parallel scales, such as CPU scale, node scale, machine room scale, quarter system scale, half system scale, whole system scale, etc.

[0003] The testing and debugging and tuning of the parallel compiler complement each other. If a problem is found during testing, the debugging and tuning of the parallel compiler is submitted for resolution. After the problem is resolved, the test is submitted for confirmation. Therefore, testing is repeated. In order to detect problems in a timely manner and shorten the system debugging cycle, the testing efficiency must be improved.

[0004] Automatic testing is an effective way to improve testing efficiency. The automatic testing of parallel compilers includes two processes: compiling and running the subject. In order to facilitate automatic control, the parallel compiler automatic testing system is designed based on the directory structure. Only one test target code is generated in one directory, and the test results are also arranged according to the directory structure. This architectural design method has a simple structure and clear organization. The automatic testing system sets the writing specifications for the test subject, requiring that a symbolic correct end file be generated when the test subject ends correctly. This is the correct identification of the subject test. If an exception occurs during the subject test, the automatic testing system will direct the output information to the compilation output log and the operation output log.

[0005] After the parallel compiler automatic test is completed, the test results need to be sorted and registered. For test topics that run correctly and end normally, automatic registration of test results is relatively easy, and you only need to check whether there is a correct end file; for topics that have abnormalities during compilation and operation, the registration of test results is more complicated.

[0006] There are two types of exceptions in the compilation process: one is a compilation error when compiling to generate a .o file, and the other is a link error when linking to generate the target code. The error message of the compilation error includes syntax unsupported, syntax error, compiler processing error, compiler internal error, etc. Sometimes a single error will report a lot of similar information, especially the compiler internal error, which will list the entire call stack, making it very difficult to determine the error site. The error message of the link error includes the inability to find defined global variables and functions, etc. If the same function is called in multiple places, it will report multiple errors, and the error information is redundant.

[0007] There are many abnormal conditions that occur during the operation, including floating-point exceptions, segment violations, overflows, out-of-bounds accesses, job submission exceptions, timeouts, no result output, no screen output due to timeouts, wrong results, etc. The error messages are reported by the operating system, the runtime library, and the test topics. They are of various types, with inconsistent formats and high information redundancy.

[0008] The characteristic of automatic testing is to start the test at a set time. Ideally, after the test is started, the compilation, operation and result registration of the test subject are a fully automatic process. However, in reality, various exceptions may occur during the test process, and the operation output information is complicated. It is very difficult to automatically register the test results and automatically identify and classify the errors. In the past, the test results were manually sorted and registered by the tester in view of the complicated and diverse error information and the large amount of redundancy. It was necessary to count the correct and error information in each directory, which was a cumbersome process, heavy workload and low efficiency. Summary of the invention

[0009] The purpose of the present invention is to provide a test record automatic registration method to solve the problem of low efficiency of manual registration.

[0010] To achieve the above object, the technical solution adopted by the present invention is: to provide a test record automatic registration method, set the writing specification of the test subject, require the generation of a symbolic correct end file when the test subject is correctly ended, and set the correct end file in a unified manner; set a compilation output log file and a running output log file in each test subject directory;

[0011] Based on the following functional modules:

[0012] The test result confirmation module is used to confirm whether the test environment and test results are correct, wherein the test environment includes CPU information and job resource information that can be obtained through the test log;

[0013] The error phase identification module is used to determine the phase where the error occurred by querying the compilation output log file and the operation output log file in each test subject directory under the directory structure, whether it is an error in the compilation process or an abnormality in the operation process;

[0014] The error type determination module is used to determine the specific type of the error; for compilation errors, the error types include syntax support, syntax error, compiler processing error, and compiler internal error; for link errors, the error types include global variable related errors and function related errors; for link errors, the error types include floating point exceptions, segment violations, overflows, out-of-bounds access, job submission exceptions, timeout without screen output, timeout not ended, no result output, and result error; different error types have different keywords in the error message;

[0015] Information screening and registration module, used to screen out specific error information and automatically register it into the database;

[0016] The record supplement registration module is used to manually supplement the registration of error reasons that cannot be automatically written through a combination of manual participation and automatic registration;

[0017] The following steps are involved:

[0018] S1. The test result confirmation module confirms whether the test result is correct by checking whether there is a marked correct end file in the test subject directory. If the test result is correct, the test environment information and test result are directly entered into the test record library. If the test result is incorrect, the error stage identification module is entered;

[0019] S2, the error stage identification module determines the stage where the error occurs by querying the compilation output log file and the operation output log file in each test subject directory under the directory structure, whether it is a compilation error, a link error or an operation error;

[0020] S3, the error type determination module confirms the specific error type by identifying the keyword information in the error message;

[0021] S4, the information screening and registration module screens out the specific error information and automatically registers it into the database;

[0022] S5. The record supplement registration module will manually supplement the registration of error reasons that cannot be automatically written, and the information can be entered into the database using the submit command.

[0023] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:

[0024] The present invention proposes a test record automatic registration method, which adopts a fully automatic way to effectively manage the test results, searches for the test output of the test subject in the directory structure, integrates the functions of automatically registering the test results, automatically identifying the error status, automatically determining the error type, automatically screening the error information and registering it into a database, and automatically analyzing the basic error causes, and provides a semi-automatic record incremental supplementary registration function, which can effectively solve the low efficiency problem caused by manual sorting of test records. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Attached Figure 1 This is a module structure diagram of the present invention. DETAILED DESCRIPTION

[0026] Embodiment: The present invention provides a test record automatic registration method, sets the writing specification of the test subject, requires generating a symbolic correct end file when the test subject is correctly ended, and sets the correct end file in a unified manner; sets a compilation output log file and a running output log file in each test subject directory;

[0027] Based on the following functional modules:

[0028] The test result confirmation module is used to confirm whether the test environment and test results are correct, wherein the test environment includes CPU information and job resource information that can be obtained through the test log;

[0029] The error phase identification module is used to determine the phase where the error occurred by querying the compilation output log file and the operation output log file in each test subject directory under the directory structure, whether it is an error in the compilation process or an abnormality in the operation process;

[0030] The error type determination module is used to determine the specific type of the error; for compilation errors, the error types include syntax support, syntax error, compiler processing error, and compiler internal error; for link errors, the error types include global variable related errors and function related errors; for link errors, the error types include floating point exceptions, segment violations, overflows, out-of-bounds access, job submission exceptions, timeout without screen output, timeout not ended, no result output, and result error; different error types have different keywords in the error message;

[0031] Information screening and registration module, used to screen out specific error information and automatically register it into the database;

[0032] The record supplement registration module is used to manually supplement the registration of error reasons that cannot be automatically written through a combination of manual participation and automatic registration;

[0033] The following steps are involved:

[0034] S1. The test result confirmation module confirms whether the test result is correct by checking whether there is a marked correct end file in the test subject directory. If the test result is correct, the test environment information and test result are directly entered into the test record library. If the test result is incorrect, the error stage identification module is entered;

[0035] S2, the error stage identification module determines the stage where the error occurs by querying the compilation output log file and the operation output log file in each test subject directory under the directory structure, whether it is a compilation error, a link error or an operation error;

[0036] S3, the error type determination module confirms the specific error type by identifying the keyword information in the error message;

[0037] S4, the information screening and registration module screens out the specific error information and automatically registers it into the database;

[0038] S5. The record supplement registration module will manually supplement the registration of error reasons that cannot be automatically written, and the information can be entered into the database using the submit command.

[0039] The above embodiment is further explained as follows:

[0040] The present invention proposes a test record automatic registration method, including automatic modules such as test result confirmation, error stage identification, error type determination, information screening registration, and a record supplement registration module requiring manual assistance, the structure of which is as follows: Figure 1 shown.

[0041] The directory structure based on automatic test record registration and the "one target code one directory" feature of the directory architecture are conducive to the correct and incorrect judgment of test cases and the query and screening of error information;

[0042] Search for the test output of the test subject in the directory structure, integrate the functions of automatically registering test results, automatically identifying error status, automatically determining error type, automatically filtering error information and registering it in the database, and automatically analyzing basic error causes, so as to effectively manage the test results;

[0043] The record supplement registration adopts a template structure. The required supplement registration header information and closing information are automatically generated during the test process. Manual record supplementation only needs to register the key items in the template according to regulations.

[0044] In order to effectively record the test conditions, the test record items are set as follows:

[0045] Name: Test topic name. A directory is a test topic, so the test topic name is the directory name.

[0046] Run: running status, the values ​​include no, right, error: 'no' means that the subject is not run in this test; 'right' means that the test is correct; 'error' means that the test is incorrect;

[0047] err_phase: error phase, indicating compile error, link error or run error;

[0048] err_type: error type;

[0049] info: error information;

[0050] reason: the reason for the error. This item may need to be filled in manually;

[0051] cversion: compiler version.

[0052] machine: test platform;

[0053] cpu: the number of CPUs used for the test;

[0054] cores: the number of computing cores contained in each CPU;

[0055] date: test date, 8 digits, arranged in year, month and day;

[0056] checker: a tester or error checker;

[0057] solve: Error solution.

[0058] The functions of each module are described in detail below:

[0059] 1) Test result confirmation

[0060] The function of the test result confirmation module is to confirm whether the test environment and test subject results are correct. The test environment includes CPU information, job resource information, etc., which can be obtained through the test log.

[0061] The automatic testing system sets the writing specifications for the test subjects, requiring that a symbolic correct end file be generated when the test subject is correctly completed. The "one target code, one directory" feature of the directory structure allows the correct end file to be set in a unified manner. Therefore, by querying whether the symbolic file exists in the test subject directory, it is possible to confirm whether the test result is correct; if the test result is correct, the test environment information and test results are directly entered into the test record library; if the test result is incorrect, the error phase identification module is entered.

[0062] 2) Error phase identification

[0063] The function of the error stage identification module is to determine the stage where the error occurred, whether it was an error in the compilation process or an abnormality in the running process. In the directory structure, there are compilation output log files and running output log files in each test subject directory. By querying the log files, you can identify whether it is a compilation error, link error or running error.

[0064] 3) Determine the error type

[0065] The function of the error type determination module is to determine the specific type of the error.

[0066] For compilation errors, there are syntax unsupported, syntax errors, compiler processing errors, and compiler internal errors. Different compilation errors have different keywords in the error information, and the specific error type can be confirmed by identifying the keyword information;

[0067] For link errors, the error types include global variable related errors and function related errors;

[0068] There are many types of runtime errors, including floating-point exceptions, segment violations, overflows, out-of-bounds accesses, job submission exceptions, timeouts without screen output, timeouts not ending, no result output, wrong results, etc. Different error types can be identified and confirmed by searching for different keywords.

[0069] 4) Information screening and registration

[0070] The information screening and registration module is used to screen out specific error information and automatically register it into the database.

[0071] For compilation error information, the more complex filtering is the information filtering when the compiler has internal errors, because the compiler will list the calling function stack when an error occurs internally. This information is more useful when the compiler is checking errors, so the entire calling function stack will be filtered into the library.

[0072] The main method used to filter link error information is to remove duplicates, because the link error information has a lot of redundancy. For example, if the same variable is used multiple times, if the variable is undefined, multiple similar error messages will be reported. When the information is registered in the database, only one error message is needed for an undefined variable, and the same is true for undefined functions.

[0073] The biggest workload of the information screening and registration module is the screening of operation error information. There are many sources of operation error information, including those reported by the operating system, Internet software, runtime library, and test topics. The types are complicated, the formats are not uniform, and the information redundancy is large. The information screening adopts the field interception strategy. According to different error types, tools such as awk, sed, and cut are used to intercept useful information from the field to remove redundant information and difficult-to-recognize characters.

[0074] 5) Record supplement registration

[0075] The record supplement registration module requires manual participation. The error causes of the test subjects are registered in the test record items. Some error causes can be automatically written according to the runtime error log, such as the operating environment does not meet the subject operation conditions, and the subject that requires an even number of computing cores is allocated an odd number of computing cores; some error causes must be entered after the designer checks the error, which requires supplementary registration of the record. The "error solution" item in the test record can only be entered through supplementary registration, because this information needs to be checked and corrected before it can be filled in.

[0076] Record supplement registration is a combination of manual participation and automatic registration. When the test results are automatically registered, the record supplement registration header information file and the record supplement registration closing information file have been automatically generated. You only need to manually register in the supplement registration template according to the following format: db['subject name']['reason']="error reason". After manual registration, use the submit command to enter the information into the database.

[0077] When the above-mentioned test record automatic registration method is adopted, the test results are effectively managed in a fully automatic manner, and the test outputs of the test subjects in the directory structure are searched. It integrates the functions of automatic registration of test results, automatic identification of error status, automatic determination of error type, automatic screening of error information and registration into the database, and automatic analysis of basic error causes. It also provides a semi-automatic record incremental supplementary registration function, which can effectively solve the inefficiency problem caused by manual sorting of test records.

[0078] In order to facilitate a better understanding of the present invention, the terms used in this article are briefly explained below:

[0079] Directory structure: Set up a separate directory file management structure for each test target code corresponding to the subject.

[0080] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatically registering test records, Features: Set the writing standards for test topics, require that a symbolic correct end file be generated when the test topic is correctly completed, and set the correct end file in a unified way; Set up compilation output log files and run output log files in each test topic directory; Based on the following functional modules: The test result confirmation module is used to confirm whether the test environment and test results are correct, wherein the test environment includes CPU information and job resource information obtained through the test log; The error phase identification module is used to determine the phase where the error occurred by querying the compilation output log file and the operation output log file in each test subject directory under the directory structure, whether it is an error in the compilation process or an abnormality in the operation process; The error type determination module is used to determine the specific type of the error; for compilation errors, the error types include syntax support, syntax error, compiler processing error, and compiler internal error; for link errors, the error types include global variable related errors and function related errors; for link errors, the error types include floating point exceptions, segment violations, overflows, out-of-bounds access, job submission exceptions, timeout without screen output, timeout not ended, no result output, and result error; different error types have different keywords in the error message; Information screening and registration module, used to screen out specific error information and automatically register it into the database; The record supplement registration module is used to manually supplement the registration of error reasons that cannot be automatically written through a combination of manual participation and automatic registration; The following steps are involved: S1. The test result confirmation module confirms whether the test result is correct by checking whether there is a marked correct end file in the test subject directory. If the test result is correct, the test environment information and test result are directly entered into the test record library. If the test result is incorrect, the error stage identification module is entered; S2, the error stage identification module determines the stage where the error occurs by querying the compilation output log file and the operation output log file in each test subject directory under the directory structure, whether it is a compilation error, a link error or an operation error; S3, the error type determination module confirms the specific error type by identifying the keyword information in the error message; S4, the information screening and registration module screens out the specific error information and automatically registers it into the database; S5. The record supplement registration module will manually supplement the registration of error reasons that cannot be automatically written, and the information can be entered into the database using the submit command.

Images