Test case operation result analysis method, electronic equipment and medium

Abstract

The invention relates to the technical field of chips, in particular to a test case operation result analysis method, electronic equipment and a medium, and the method comprises the following steps: S1, obtaining operation log information corresponding to a test case tx; s2, analyzing the running log information corresponding to the tx, and obtaining a position information set Utx where all preset failure text identifiers exist in the running log information corresponding to the tx; and S3, traversing each Ukx in the Utx, judging whether a condition that a preset failure text identifier exists in a preset configuration file exists or not, if so, deleting the Ukx from the Utx, if not, retaining the Ukx, if the processed Utx is null, determining that the tx operation result is passed, and otherwise, determining that the tx operation result is not passed. According to the invention, the accuracy of test case operation result analysis is improved.

Description

Technical Field

[0001] The present invention relates to the field of chip technology, and in particular to a test case running result analysis method, electronic equipment and medium. Background Art

[0002] To ensure that chips function properly under various conditions and meet design specifications and industry standards, the chip verification process requires constructing a large number of chip test cases and performing regression testing based on these test cases to verify the chip. The pass rate during chip regression testing measures the successful execution rate of verification cases—that is, the percentage of test cases in the verification test set that pass without error. The pass rate is a key quality indicator, typically used to measure the chip's passability at each testing stage. Accurately obtaining the pass rate requires accurately obtaining the execution results of each test case. In the prior art, the test case execution results are determined by analyzing the test case log to see if they contain strings indicating test case failure. However, some strings indicating test case failure in the log file are legitimate and cannot be directly used as a criterion for determining test case failure. Therefore, the prior art methods for analyzing test case execution results can result in many misjudgments and low accuracy. Therefore, improving the accuracy of test case execution result analysis has become a pressing technical issue. Summary of the Invention

[0003] The present invention aims to provide a test case execution result analysis method, electronic device and medium, which improve the accuracy of the test case execution result analysis.

[0004] According to a first aspect of the present invention, a method for analyzing test case execution results is provided, comprising:

[0005] Step S1: Get test case t x In the corresponding run log information, the value of x ranges from 1 to X, where X is the total number of test cases;

[0006] Step S2, analysis t x Corresponding running log information, get t x The location information set Ut of all preset failure text identifiers in the corresponding operation log information x ={U1 x ,U2 x ,...,U k x ,...,U p(x) x}, U k x t xThe location information of the kth preset failure text mark in the corresponding operation log information. The value range of k is 1 to p(x), and p(x) is t x The total number of preset failure text identifiers in the corresponding operation log information;

[0007] Step S3, traverse Ut x Each U k x , determine whether there is a preset failure text mark set in the preset configuration file, if so, then U k x From {U1 x ,U2 x ,...,U k x ,...,U p(x) x}, if it does not exist, keep U k x , if the processed Ut x If it is empty, determine t x The result of the operation is passed, otherwise, determine t x The running result is failed.

[0008] According to a second aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to execute the method described in the first aspect of the present invention.

[0009] According to a third aspect of the present invention, a computer-readable storage medium is provided, storing computer-executable instructions, wherein the computer instructions are used to execute the method according to the first aspect of the present invention.

[0010] The present invention has significant advantages and beneficial effects compared to existing technologies. Through the above technical solution, the test case execution result analysis method, electronic device, and medium provided by the present invention can achieve considerable technological advancement and practicality, and have wide industrial application value, and have at least the following beneficial effects:

[0011] The present invention first obtains the location information set of all preset failure text identifiers in the operation log of each test case, and then queries them one by one to determine whether there is a preset failure text identifier, and then deletes it from the location information set, thereby avoiding misjudgment of the operation result and improving the accuracy of the test case operation result analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 Flowchart of the test case execution result analysis method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0015] According to a first aspect of the present invention, a method for analyzing test case execution results is provided. Figure 1 As shown, including:

[0016] Step S1: Get test case t x In the corresponding run log information, the value of x ranges from 1 to X, where X is the total number of test cases.

[0017] Step S2, analysis t x Corresponding running log information, get t x The location information set Ut of all preset failure text identifiers in the corresponding operation log information x ={U1 x ,U2 x ,...,U k x ,...,U p(x) x}, U k x t x The location information of the kth preset failure text mark in the corresponding operation log information. The value range of k is 1 to p(x), and p(x) is t x The total number of preset failure text identifiers in the corresponding run log information.

[0018] Among them, U k x Specifically, it may include the coordinates of the first letter and the text length information corresponding to the preset failure text mark. The preset failure text mark may be set to "error".

[0019] Step S3, traverse Ut x Each U k x , determine whether there is a preset failure text mark set in the preset configuration file, if so, then U k x From {U1 x ,U2 x ,...,U k x ,...,U p(x) x}, if it does not exist, keep U k x , if the processed Ut x If it is empty, determine t x The result of the operation is passed, otherwise, determine t x The running result is failed.

[0020] It should be noted that after the processing of steps S1 to S3, the U k x Delete from the location information collection when Ut x When it is not empty, it means t x The corresponding running result must be failed.

[0021] As an embodiment, step S1 includes:

[0022] Step S11: Obtain the chip design code to be tested and target test case reconstruction information, wherein the target test case reconstruction information includes user identification, configuration information of the design to be tested, operating parameters, number of cycles, test case group and operating time.

[0023] Step S12: Generate test cases required for chip regression testing based on the target test case reconstruction information.

[0024] Step S13: compile the chip design code to generate an executable file of the chip design to be tested.

[0025] Step S14: running test cases required for chip regression testing in parallel based on the executable file designed for the chip to be tested, and generating running logs corresponding to the test cases.

[0026] As an embodiment, in step S11, the target test case reconstruction information is {C0, (C1, B1), (C2, B2), ..., (C n ,B n ),...,(C N ,B N )}, C0 is the shared test case configuration information corresponding to the chip design under test, C nThe shared test case configuration information corresponding to the nth sub-design of the chip under test, B n The nth group of test case reconstruction information list for the chip under test design, n ranges from 1 to N, N is the total number of sub-designs of the chip under test design, B n ={F1 n ,F2 n ,...,F i n ,...,F a(n) n}, F i n Reconstruct the information of the i-th test case in the n-th test case reconstruction information list. The value of i ranges from 1 to a(n), and a(n) is B n The total number of corresponding test case reconstruction information.

[0027] As an example, each F i n Including one or more of user identification, configuration information of the design to be tested, operating parameters, number of cycles, test case group and running time; C0, C n Including one or more of the design configuration information to be tested, operating parameters, number of cycles and running time. It should be noted that the user identifier and test case group are maintained separately for each test case reconstruction information. The design configuration information to be tested, operating parameters, number of cycles and running time can be used as shared test case configuration information, that is, the design configuration information to be tested, operating parameters, number of cycles and running time of the test cases can be set in batches through the test case configuration information. Among them, the number of cycles is set according to the specific test regression requirements, for example, it is set to 10 times or 20 times. The running time can be specifically set to execute within a preset time interval or once every preset time period. The preset time interval can specifically be daytime period, nighttime period, morning period, afternoon period, etc. The preset time period can specifically be one day, one week, etc.

[0028] As an embodiment, the configuration information of the design to be tested is used to set the composition form of each component of the chip design to be tested, and the composition form includes a first form and a second form, the first form is a form including component ports and internal logic, and the second form is a form including only component ports. Taking the design to be tested as an example including the first component, the second component, the third component and the fourth component, assuming that the object to be verified is the first component, then the corresponding configuration information of the design to be tested is to set the first component to the first form, and the second component, the third component and the fourth component to the second form. Assuming that the verification object is the interaction between the first component and the second component, then the first component and the second component are set to the first form, and the third component and the fourth component are set to the second form.

[0029] As an embodiment, the design configuration information to be tested includes master configuration information and multiple slave configuration information. The master configuration information is the configuration information with the highest frequency of use. When no information is specified in the design configuration information to be tested, the master configuration information is set by default. When slave configuration information needs to be set, the corresponding slave configuration information needs to be specified in the design configuration information to be tested. By setting the master configuration information and multiple slave configuration information, the default configuration can be directly used in most scenarios, further improving the efficiency of constructing the target test case reconstruction structure, thereby improving the efficiency of test case generation.

[0030] As an embodiment, the operating parameters include environment variables, variables required by the preset script, and options that need to be passed to the electronic design automation (EDA) tool during the chip test case running process, wherein the environment variables can specifically be Linux environment variables, the preset script is a script for executing the chip test case, and the script type can specifically be Python, Perl, Ruby, Rust, Ymal, etc.

[0031] As an embodiment, the test case group includes at least one test case, and the test case group is divided based on chip functional characteristics or based on running time period. Based on chip functional characteristics, test cases that verify the same chip functional characteristics under the same design configuration information to be tested can be divided into one test case group. In order to improve resource utilization, it is also possible to divide the test cases with the same or similar running time into one test case group based on running time period.

[0032] As an embodiment, the test case group further includes a first identifier and a second identifier. If the first identifier is set, the current regression test process needs to generate test cases in the test case group. If the second identifier is set, the current regression test process does not need to generate test cases in the test case group. By setting the test case group identifier, flexible configuration of regression testing can be achieved. When a test case group does not need to participate in the current regression, the test case group can be set to the second identifier without deleting the test case group. When the test case group is needed in a subsequent regression test process, the test case group only needs to be set to the first identifier.

[0033] It should be noted that multi-level shared test case configuration information is set in the target test case reconstruction structure, which reduces the maintenance amount of test case configuration information and facilitates global changes of the same test case configuration information, thereby reducing the error rate.

[0034] As an embodiment, step S12 includes:

[0035] Step S121: parse the target test case reconstruction structure to obtain C0, and then obtain each C in parallel. n and B n .

[0036] It should be noted that, by reconstructing the structure based on the target test case, the shared test case configuration information only needs to be parsed once and can maintain consistency, which improves the efficiency of test case construction and avoids errors.

[0037] Step S122, parallel analysis of each B n , get each F i n .

[0038] Step S123: Based on each F i n 、F i n Corresponding C n , C0 generates all chip test cases for the chip design to be tested.

[0039] As an embodiment, step S123 includes:

[0040] Step S1231: Get F i n Corresponding user ID and test case group {T1 in ,T2 in ,...,T j in ,,...,T b(in) in}, T j inF i n The corresponding j-th test case information, j ranges from 1 to b(in), b(in) is F i n The number of test cases in the corresponding test case group.

[0041] It is understandable that when i and n take different values, the corresponding value of b(in) may also be different.

[0042] Step S1232: If F i n If there are preset parameters in the i n The preset parameters in are determined as T j in Corresponding target parameters, otherwise, execute step S1233, the preset parameters are the design configuration information to be tested, operating parameters, number of cycles, test case group or running time.

[0043] It should be noted that each F i n The user ID and test case group must be set. The design configuration information to be tested, running parameters, number of cycles, test case group or running time can be set or not. If not set, it will be generated based on the shared test case configuration information.

[0044] Step S1233: If C n If there are preset parameters in the n The preset parameters in are determined as T j in Corresponding target parameters, otherwise, execute step S234.

[0045] Step S1234: If a preset parameter is set in C0, the preset parameter of C0 is determined as T j in If the target parameter is correct, execute step S1235; otherwise, generate a prompt message and end the process.

[0046] Through steps S1232 to S1234, when F i n When preset parameters are set in F i n Set the preset parameters as target parameters. The preset parameters can be one or more of the following: the configuration information of the design to be tested, the operating parameters, the number of cycles, the test case group, and the operating time. i n If there is no preset parameter in the n Find the corresponding preset parameters in C, if they exist,n The preset parameters in are determined as the corresponding target parameters. n If it does not exist in C0, the preset parameters corresponding to the settings in C0 are determined as the target parameters. It can be seen that the preset parameters that work on the global design under test are set in C0, and the preset parameters that work on the sub-design of the design under test are set in C n In the test case reconstruction information of each group of personalized i n In , multi-level configuration of test case reconstruction information is implemented.

[0047] Step S1235: Based on each T j in , each T j in The corresponding user ID and each T j in All target parameters corresponding to each T j in Corresponding test cases.

[0048] As an embodiment, step S14 includes:

[0049] Step S141: Test cases required for chip regression testing x Assign the corresponding number of CPUs P x And memory storage space Q x , the value of x ranges from 1 to X, where X is the total number of test cases required for chip regression testing.

[0050] Step S142: Design an executable file based on the chip to be tested and run all t x , generating each run t x The corresponding test case running record, the running log includes the actual number of CPUs required for running P1 x And the actual memory storage space required Q1 x .

[0051] Step S143: When t x When the corresponding test case running records reach R, get the R records closest to the current one. x P1 in the corresponding test case run record x The data P2 after rounding the mean x and Q1 x The mean of the rounded data Q2 x , P x Updated to P x =P2 x +P0 x , Q x Updated to Qx =Q2 x +Q0 x , P0 x Q0 is the preset CPU quantity adjustment value. x is the preset memory storage space adjustment value, and R is the preset test case running record accumulation threshold.

[0052] Through steps S141 to S143, the resources allocated to the test case can be dynamically adjusted based on the actual running status of the test case, thereby avoiding resource waste and improving resource utilization.

[0053] As an embodiment, the situation in which a preset failure text identifier is set in the preset configuration file includes: the preset failure text identifier and the first several characters and / or the last several characters form a preset phrase, and the attribute information corresponding to the preset failure text identifier is the preset attribute information. Still taking the example that the preset failure text identifier can be specifically set to "error", there is a register named error_state, then the situation in which a preset failure text identifier exists includes the 6 characters after "error" and error forming error_state. In addition, there are some situations in which the "error" result needs to be run. For such situations, the attribute information corresponding to "error" is set to the preset attribute information.

[0054] As an embodiment, the preset configuration file includes five levels of configuration files {LF1, LF2, LF3, LF4, LF5}, where LF1 is a global preset configuration file, which is effective for multiple global test cases. LF2 is a list of preset configuration files corresponding to the chip design to be tested, which is effective for the test cases corresponding to the entire chip design to be tested. LF3 is a list of preset configuration files corresponding to the chip design configuration to be tested, which is effective for the test cases corresponding to the chip design configuration to be tested. LF4 is a list of preset configuration files corresponding to the functional features of the chip design to be tested, which is effective for the test cases corresponding to the functional features of the chip design to be tested. LF5 is a list of preset configuration files corresponding to the test case, which is effective for the corresponding test case.

[0055] As an embodiment, the step S1 further includes:

[0056] Step S11: Get test case t x Corresponding configuration information (DS x ,DC x ,DF x ), where DS x t x Corresponding chip design under test, DC x t x Corresponding chip design configuration to be tested, DF x t xThe corresponding chip design features under test, the value of x ranges from 1 to X, and X is the total number of test cases.

[0057] As an embodiment, step S3 includes:

[0058] Step S31: Initially set the configuration file level v=1.

[0059] Step S32, traverse Ut x Each U k x , based on t x Each U in the corresponding operation log information k x The text information at level v configuration file LF v The preset failure text mark is set in the match, and the successful U k x From Ut x Deleted.

[0060] Step S33: Determine the current Ut x Is it empty? If it is empty, determine t x If the operation result is passed, the process ends; otherwise, step S34 is executed.

[0061] Step S34: If v=5, determine t x If the result of the operation is failure, the process ends. If v<5, set v=v+1 and return to step S32.

[0062] It should be noted that, through steps S31 to S34 , it is possible to determine step by step whether the situation in the configuration file exists in the location information set, and delete all location information in the location set that may cause misjudgment from the location set.

[0063] In order to further improve the accuracy of the test case running result analysis, for some situations that cannot be directly determined, downgrade processing can be performed first, that is, downgraded from the fail level to the warning level, and then the warning level information can be further judged. As an embodiment, each situation with a preset failure text identifier set in the preset configuration file is also correspondingly provided with a processing identifier, and the processing identifier includes a first processing identifier and a second processing identifier. The first processing identifier corresponds to an identifier that can be directly determined as a misjudgment, and the second processing identifier corresponds to an identifier that cannot be directly determined as a misjudgment and needs to be downgraded first and then further judged. The step S3 includes:

[0064] Step C31: Initially set the configuration file level v=1.

[0065] Step C32: traverse Ut x Each U kx , based on t x Each U in the corresponding operation log information k x The text information at level v configuration file LF v The case where there is a preset failure text identifier set in the match is matched, and the U that is successfully matched and whose processing identifier is the first processing identifier is k x From Ut x and delete the U that is matched successfully and whose processing identifier is the second processing identifier. k x Store to t x The corresponding preset warning position set Uw x middle.

[0066] Step C33: If the current Ut x is not empty, then execute step C34. If the current Ut x If it is empty, execute step C35;

[0067] Step C34: If v=5, determine t x If the result of the operation is failure, the process ends. If v<5, set v=v+1 and return to step C32.

[0068] Step C35: If the current Uw x If it is empty, determine t x If the result is passed, the process ends; otherwise, execute step C36;

[0069] Step C36: Determine Uw x U in k x Is there an error at the position? If so, determine t x If the result of the operation is not passed, the process ends; otherwise, determine t x The running result is passed, and the process ends.

[0070] It should be noted that, through steps C31 to C36, it is determined step by step whether the situation in the configuration file exists in the location information set, and the U of the first processing mark is k x From Ut x Delete the second processing mark U k x Store to t x The corresponding preset warning position set Uw x In the first step, the downgrade process is performed, from the error level to the warning level. When all configuration files are judged, if Ut x If it is not empty, determine t xThe result of the operation is not passed, otherwise, further confirmation of the current Uw x Is it empty? If not, further judgment is needed on Uw x U in k x Steps C31 to C36 further improve the accuracy of the test case running result analysis.

[0071] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of the steps can be performed in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. A process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0072] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.

[0073] An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.

[0074] The embodiment of the present invention first obtains the location information set of all preset failure text identifiers in the operation log of each test case, and then queries them one by one to determine whether there are preset failure text identifiers, and then deletes them from the location information set to avoid misjudgment of the operation results and improve the accuracy of the test case operation result analysis.

[0075] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the present profession can make slight changes or modifications to equivalent embodiments using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A test case running result analysis method, characterized in that: include: Step S1: Get test case t x In the corresponding run log information, the value of x ranges from 1 to X, where X is the total number of test cases; Step S2, analysis t x Corresponding running log information, get t x The location information set Ut of all preset failure text identifiers in the corresponding operation log information x ={U1 x ,U2 x ,...,U k x ,...,U p(x) x }, U k x t x The location information of the kth preset failure text mark in the corresponding operation log information. The value range of k is 1 to p(x), and p(x) is t x The total number of preset failure text identifiers in the corresponding operation log information; Step S3, traverse Ut x Each U k x , determine whether there is a preset failure text mark set in the preset configuration file, if so, then U k x From {U1 x ,U2 x ,...,U k x ,...,U p(x) x }, if it does not exist, keep U k x , if the processed Ut x If it is empty, determine t x The result of the operation is passed, otherwise, determine t x The running result is failed.

2. The method according to claim 1, characterized in that The situations where the preset failure text identifier exists in the preset configuration file include: the preset failure text identifier and the first and / or last characters form a preset phrase, and the attribute information corresponding to the preset failure text identifier is the preset attribute information.

3. The method according to claim 1, characterized in that The preset configuration files include five levels of configuration files {LF1, LF2, LF3, LF4, LF5}, LF1 is a global preset configuration file, LF2 is a preset configuration file list corresponding to the chip design to be tested, LF3 is a preset configuration file list corresponding to the chip design configuration to be tested, LF4 is a preset configuration file list corresponding to the functional features of the chip design to be tested, and LF5 is a preset configuration file list corresponding to the test case.

4. The method according to claim 3, characterized in that The step S1 further includes: Step S11: Get test case t x Corresponding configuration information (DS x ,DC x ,DF x ), where DS x t x Corresponding chip design under test, DC x t x Corresponding chip design configuration to be tested, DF x t x The corresponding chip design features under test, the value of x ranges from 1 to X, and X is the total number of test cases.

5. The method according to claim 4, characterized in that The step S3 comprises: Step S31, initially setting the configuration file level v=1; Step S32, traverse Ut x Each U k x , based on t x Each U in the corresponding operation log information k x The text information at level v configuration file LF v The preset failure text mark is set in the match, and the successful U k x From Ut x Delete in; Step S33: Determine the current Ut x Is it empty? If it is empty, determine t x If the result is passed, the process ends; otherwise, step S34 is executed; Step S34: If v=5, determine t x If the result of the operation is failure, the process ends. If v<5, set v=v+1 and return to step S32.

6. The method according to claim 4, characterized in that Each case where a preset failure text flag is set in the preset configuration file also corresponds to a processing flag, and the processing flag includes a first processing flag and a second processing flag. Step S3 includes: Step C31, initially setting the configuration file level v=1; Step C32: traverse Ut x Each U k x , based on t x Each U in the corresponding operation log information k x The text information at level v configuration file LF v The case where there is a preset failure text identifier set in the match is matched, and the U that is successfully matched and whose processing identifier is the first processing identifier is k x From Ut x and delete the U that is matched successfully and whose processing identifier is the second processing identifier. k x Store to t x The corresponding preset warning position set Uw x middle; Step C33: If the current Ut x is not empty, then execute step C34. If the current Ut x If it is empty, execute step C35; Step C34: If v=5, determine t x If the result is "fail", the process ends. If v < 5, set v = v + 1 and return to step C32. Step C35: If the current Uw x If it is empty, determine t x If the result is passed, the process ends; otherwise, execute step C36; Step C36: Determine Uw x U in k x Is there an error at the position? If so, determine t x If the result of the operation is not passed, the process ends; otherwise, determine t x The running result is passed, and the process ends.

7. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-executable instructions are stored, and the computer-executable instructions are used to execute the method according to any one of the preceding claims 1 to 6.

Images