Software automatic evaluation method, electronic equipment and computer readable medium

Abstract

The invention discloses an automatic software evaluation method, electronic equipment and a computer readable medium, and belongs to the technical field of electronic design automation software, and the method comprises the steps: obtaining a test demand, obtaining a first value of at least one index corresponding to each test case obtained by operating each test case through comparison version software, and taking the first value as first test data, obtaining a second numerical value of at least one index corresponding to each test case obtained by running each test case by the to-be-evaluated version software, and taking the second numerical value as second test data; and based on the first test data and the second test data corresponding to each test case, determining a difference value of each index corresponding to each test case, and determining an evaluation result of the to-be-evaluated version software according to the difference value of each index corresponding to each test case. The evaluation result of the to-be-edited software is automatically evaluated based on the test requirement, so that the evaluation efficiency is improved, errors caused by manual evaluation are avoided, and the performance of the software can be comprehensively evaluated.

Description

Technical Field

[0001] This application relates to the field of electronic design automation software technology, and more specifically, to an automatic evaluation method for software, an electronic device, and a computer-readable medium. Background Technology

[0002] Field-Programmable Gate Arrays (FPGAs) are highly flexible hardware platforms that have seen rapid growth in applications such as industrial control, communication systems, and artificial intelligence accelerators due to their excellent programmability and performance advantages. Consequently, the accompanying Electronic Design Automation (EDA) software also needs continuous iteration and upgrades to adapt to increasingly complex design scenarios.

[0003] EDA software, as the core driving force of FPGA hardware design, encompasses a series of key stages including logic synthesis, placement and routing, and static timing analysis, along with numerous toolchains. Each process typically requires evaluation using multiple test metrics, making its development extremely cumbersome. Every modification during EDA development necessitates regression testing of tested cases to ensure its normality and stability, which is labor-intensive. Furthermore, the metrics can easily influence each other, making it difficult to evaluate the progress of changes. The development and testing of EDA software lack a comprehensive assessment of its various characteristics and overall performance, and the absence of an effective automated scoring mechanism makes quality evaluation after each regression test time-consuming and error-prone. Summary of the Invention

[0004] This application proposes an automated software evaluation method, electronic device, and computer-readable medium to improve upon the aforementioned deficiencies.

[0005] In a first aspect, this application provides an automatic software evaluation method applied to a processor of an electronic device. The method includes: obtaining test requirements, the test requirements including reference version information and version information to be evaluated; determining multiple test cases based on the test requirements; obtaining a first value of at least one indicator corresponding to each test case obtained by running each test case of the reference version software, as first test data; and obtaining a second value of at least one indicator corresponding to each test case obtained by running each test case of the version to be evaluated software, as second test data; determining a difference value of each indicator corresponding to each test case based on the first test data and the second test data corresponding to each test case, the difference value representing the value obtained by subtracting the corresponding first value from the second value; and determining an evaluation result of the version to be evaluated software based on the difference value of each indicator corresponding to each test case.

[0006] Optionally, in one possible implementation, obtaining the first value of at least one indicator corresponding to each test case obtained by running each test case of the comparison version software as first test data, and obtaining the second value of at least one indicator corresponding to each test case obtained by running each test case of the version software to be evaluated as second test data, includes: obtaining the first value of at least one indicator corresponding to each test case obtained by running each test case of the comparison version software as first test data from a preset database, and obtaining the second value of at least one indicator corresponding to each test case obtained by running each test case of the version software to be evaluated as second test data.

[0007] Optionally, in one possible implementation, the method further includes: if there is no first value of at least one indicator corresponding to each test case obtained by running each test case with the reference version software in the preset database, then by running each test case with the reference version software, a first test result corresponding to each test case is obtained; the first value of the corresponding indicator is extracted from each first test result corresponding to the test case, and the first value of the indicator corresponding to each test case is used as the first test data of each test case.

[0008] Optionally, in one possible implementation, the method further includes: if there is no second value of at least one indicator corresponding to each test case obtained by running each test case of the software version to be evaluated in the preset database, then by running each test case of the software version to be evaluated, a second test result corresponding to each test case is obtained; the second value of the corresponding indicator is extracted from each second test result corresponding to the test case, and the second value of the indicator corresponding to each test case is used as the second test data of each test case.

[0009] Optionally, in one possible implementation, determining the evaluation result of the software version to be evaluated based on the difference value of each indicator corresponding to each test case includes: determining the optimization amount of each indicator corresponding to each test case based on the difference value of each indicator corresponding to each test case, the first value of each indicator, and the preset rules of each indicator; determining the indicator score of each indicator corresponding to each test case based on the optimization amount of each indicator corresponding to each test case, the preset score of each indicator, the preset maximum optimization amount of each indicator, and the preset maximum deterioration amount of each indicator; and determining the evaluation result of the software version to be evaluated based on the indicator score of each indicator corresponding to each test case.

[0010] Optionally, in one possible implementation, determining the optimization amount of each indicator corresponding to each test case based on the difference value of each indicator corresponding to each test case, the first value of each indicator, and the preset rule of each indicator includes: if there is an indicator whose value range does not include 0 among the indicators corresponding to the test case, the indicator whose value range does not include 0 is taken as the first indicator; the optimization value of each first indicator is determined based on the ratio of the difference value of the first indicator to the corresponding first value; the optimization sign of each first indicator is determined based on the preset rule of each first indicator; the optimization amount of each first indicator is determined based on the optimization value of each first indicator and the corresponding optimization sign; if there is an indicator whose value range includes 0 among the indicators corresponding to the test case, the indicator whose value range includes 0 is taken as the second indicator; the optimization value of each second indicator is determined based on the difference value of each second indicator; the optimization sign of each second indicator is determined based on the preset rule of each second indicator; the optimization amount of each second indicator is determined based on the optimization value of each second indicator and the corresponding optimization sign.

[0011] Optionally, in one possible implementation, determining the indicator score for each indicator corresponding to each test case based on the optimization amount, preset score, preset maximum optimization amount, and preset maximum degradation amount of each indicator corresponding to each test case includes: if there are indicators among the indicators corresponding to the test case that do not correspond to a preset minimum degradation amount and a preset minimum optimization amount, the indicators that do not correspond to a preset minimum degradation amount and a preset minimum optimization amount are used as third indicators, and the indicator score for each of the third indicators corresponding to the test case is determined to be C1. Where A represents the optimization amount of the third indicator, A1 represents the preset maximum deterioration amount of the third indicator, A2 represents the preset maximum optimization amount of the third indicator, and B represents the preset score corresponding to the third indicator.

[0012] Optionally, in one possible implementation, the method further includes: if among the metrics corresponding to the test case there are metrics corresponding to a preset minimum degradation amount and a preset minimum optimization amount, the metrics corresponding to the preset minimum degradation amount and the preset minimum optimization amount are used as fourth metrics, and the metrics of each of the fourth metrics corresponding to the test case are divided into C2: Wherein, D represents the optimization amount of the fourth indicator, D1 represents the preset maximum deterioration amount of the fourth indicator, D2 represents the preset minimum deterioration amount of the fourth indicator, D3 represents the preset minimum optimization amount of the fourth indicator, D4 represents the preset maximum optimization amount of the fourth indicator, and E represents the preset score corresponding to the indicator.

[0013] Optionally, in one possible implementation, the sum of the preset scores for each indicator corresponding to each test case is a preset first score. The step of determining the evaluation result of the software version to be evaluated based on the indicator scores for each indicator corresponding to each test case includes: obtaining the total indicator score for each test case based on the indicator scores for each indicator corresponding to each test case; determining the second score for the software version to be evaluated based on the average of the total indicator scores for each test case; and determining the evaluation result of the software version to be evaluated based on the second score and the first score.

[0014] Secondly, this application also provides an electronic device, comprising: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory, the one or more application programs are configured to be executed by the one or more processors, and the one or more application programs are configured to perform the methods described above.

[0015] Thirdly, this application also provides a computer-readable medium storing processor-executable program code, which, when executed by the processor, causes the processor to perform the above-described method.

[0016] This application provides an automatic software evaluation method. First, test requirements are obtained, including information on a reference version of the software and information on the version to be evaluated. Based on the test requirements, multiple test cases are determined. Second, a first value for at least one indicator corresponding to each test case is obtained from running each test case on the reference version software, serving as first test data. A second value for at least one indicator corresponding to each test case is obtained from running each test case on the version to be evaluated, serving as second test data. Then, based on the first test data and the second test data for each test case, a difference value for each indicator corresponding to each test case is determined. The difference value represents the value obtained by subtracting the corresponding first value from the second value. Finally, the evaluation result of the version to be evaluated is determined based on the difference value for each indicator corresponding to each test case.

[0017] This application determines multiple test cases based on testing requirements, identifies the corresponding metric difference value for each test case, and determines the evaluation result of the version to be evaluated based on the metric difference value corresponding to each test case. On the one hand, manual evaluation is inefficient and prone to errors. Compared with manual evaluation methods, this application automatically evaluates the evaluation result of the software version based on testing requirements, which not only improves the efficiency of evaluation but also avoids errors caused by manual evaluation, enabling a comprehensive evaluation of software performance. On the other hand, since different testers may interpret the same test result differently, the accuracy of manual evaluation is low, and there is a lack of unified evaluation rules for test results. The embodiments of this application unify the evaluation rules and improve the accuracy of the evaluation results.

[0018] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of the automatic evaluation method for software provided in an embodiment of this application is shown; Figure 2 A flowchart of an automatic evaluation method for software provided in another embodiment of this application is shown; Figure 3 A flowchart of an automatic evaluation method for software provided in another embodiment of this application is shown; Figure 4 A flowchart of an automatic evaluation method for software provided in another embodiment of this application is shown; Figure 5 A structural block diagram of the electronic device provided in an embodiment of this application is shown; Figure 6 A structural block diagram of a computer-readable storage medium provided in an embodiment of this application is shown. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. The components of the embodiments of the present application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without inventive effort are within the scope of protection of the present application.

[0022] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] Field-Programmable Gate Arrays (FPGAs) are highly flexible hardware platforms that have seen rapid growth in applications such as industrial control, communication systems, and artificial intelligence accelerators due to their excellent programmability and performance advantages. Consequently, the accompanying Electronic Design Automation (EDA) software also needs continuous iteration and upgrades to adapt to increasingly complex design scenarios.

[0024] EDA software, as the core driving force of FPGA hardware design, encompasses a series of key stages including logic synthesis, placement and routing, and static timing analysis, along with numerous toolchains. Each process typically uses multiple test metrics for evaluation, making its development extremely cumbersome. Every modification during EDA development requires regression testing of tested cases to ensure its normality and stability, which is labor-intensive. Furthermore, the metrics can easily influence each other, making it difficult to accurately evaluate the progress of changes. The development and testing of EDA software lacks a comprehensive evaluation of its various characteristics and overall performance, and the absence of an effective automatic scoring mechanism makes quality assessments after each regression test time-consuming and prone to errors.

[0025] Therefore, this application provides an automatic software evaluation method, an electronic device, and a computer-readable medium to solve or partially solve the above-mentioned problems.

[0026] Please see Figure 1The document illustrates a flowchart of an automatic software evaluation method provided in an embodiment of this application, applied to a processor of an electronic device. The method specifically includes steps S101 to S105.

[0027] Step S101: Obtain test requirements, which include the software's reference version information and the version to be evaluated.

[0028] It's important to note that when an EDA software is improved, its performance needs to be tested. Furthermore, EDA has numerous functional modules, and improvements are not always made to the entire EDA. Therefore, the testing requirements differ depending on the specific improvements made to the EDA. Test requirements, or testing objectives, are used to verify whether newly added or modified functions meet the design requirements.

[0029] For example, the test requirements are determined by the developers based on the improvements. The test requirements include, but are not limited to, functional correctness testing, compatibility testing, performance testing, stability testing, user experience testing, and security testing. For example, the test requirement is "to verify the performance improvement of the routing algorithm of the EDA software to be evaluated under circuit designs of different complexity levels when performing digital circuit design".

[0030] The testing requirements include information on the reference version of the software and the version to be evaluated. The reference version information indicates the version to be evaluated, such as the previous stable version or the previous release version. The version to be evaluated refers to the improved EDA software version, that is, the EDA software version whose performance needs to be evaluated.

[0031] Step S102: Determine multiple test cases based on the test requirements.

[0032] Understandably, test cases are used to describe the specific steps of how to test a certain function, and typically include: test objective (the function to be verified), preconditions (environmental setup before testing), operation steps (how to execute the test), expected result (what should happen under correct conditions), actual result (the actual situation after the test is executed), and pass or fail status (whether the test was successful), etc.

[0033] A single test requirement can correspond to multiple test cases. These test cases comprehensively check whether the software meets the requirement from different perspectives, scenarios, and data combinations.

[0034] For example, based on the testing requirements, five test cases are determined: "New Project Function Test", "Open Existing Project Function Test", "Component Library Loading Function Test", "Component Search Function Test", and "Schematic Drawing Function Test".

[0035] In one alternative embodiment, a first test case is determined based on testing requirements, and the first test case that has been tested in the comparison version is used as the test case.

[0036] For example, based on the testing requirements, seven first test cases are determined: "New Project Functionality Test", "Open Existing Project Functionality Test", "Component Library Loading Functionality Test", "Component Search Functionality Test", "Schematic Drawing Functionality Test", "Schematic Retention Functionality Test", and "Schematic Printing Functionality Test". Among these, the "Schematic Retention Functionality Test" and "Schematic Printing Functionality Test" test cases were not tested in the comparison version; therefore, the "New Project Functionality Test", "Open Existing Project Functionality Test", "Component Library Loading Functionality Test", "Component Search Functionality Test", and "Schematic Drawing Functionality Test" from the first test cases are selected as test cases.

[0037] Step S103: Obtain the first value of at least one indicator corresponding to each test case obtained by running each test case of the comparison version software, as the first test data; and obtain the second value of at least one indicator corresponding to each test case obtained by running each test case of the version software to be evaluated, as the second test data.

[0038] It should be noted that the EDA software runs each test case to obtain test result data, and the value of the corresponding indicator for that test case can be determined based on the test result data. Each test case corresponds to at least one indicator, and the indicator corresponding to each test case is determined based on the specific test case.

[0039] For example, the test case "New Project Function Test" has two indicators: "Whether the project was successfully created" and "Whether the project name, path and other attributes are set correctly"; the test case "Schematic Drawing Function Test" has two indicators: "Whether the schematic can be drawn normally" and "Whether the connection of the schematic is correct"; the test case "PCB Layout Function Test" has two indicators: "Whether the component layout is easy to operate" and "Whether the position of the components after layout can be adjusted"; and the test case "Signal Integrity Analysis Function Test" has two indicators: "Whether the signal integrity analysis results are accurate" and "Whether the analysis report is detailed".

[0040] The first test data consists of the first value of at least one metric corresponding to each test case obtained by running each test case of the comparison version software, and the second test data consists of the second value of at least one metric corresponding to each test case obtained by running each test case of the version to be evaluated. In other words, the first test data includes each test case and its corresponding metric value corresponding to the comparison version, and the second test data includes each test case and its corresponding metric value corresponding to the version to be evaluated. Each test case corresponds to both the first and second test data. It should be noted that different metric values ​​represent different performance characteristics.

[0041] Step S104: Based on the first test data and the second test data corresponding to each test case, determine the difference value of each indicator corresponding to each test case, wherein the difference value represents the value obtained by subtracting the corresponding first value from the second value.

[0042] It should be noted that the first test data for each test case includes the metric values ​​for the comparison version, and the second test data for each test case includes the metric values ​​for the version to be evaluated. The difference value of each metric for each test case can be determined based on the first and second test data for each test case.

[0043] For example, the test case "Signal Integrity Analysis Function Test" corresponds to two metrics: "Is the signal integrity analysis result accurate?" and "Is the analysis report detailed?". In the first test data for this test case, the value for "Is the signal integrity analysis result accurate?" is 8, and the value for "Is the analysis report detailed?" is 7. In the second test data for this test case, the value for "Is the signal integrity analysis result accurate?" is 9, and the value for "Is the analysis report detailed?" is 7. Therefore, the difference value for the metric "Is the signal integrity analysis result accurate?" is 9-8=1, and the difference value for the metric "Is the analysis report detailed?" is 7-7=0. Thus, the differences between the two metrics for this test case are 1 and 0, respectively. Similarly, the difference value for each metric for each test case can be obtained.

[0044] Step S105: Determine the evaluation result of the version of software to be evaluated based on the difference value of each indicator corresponding to each test case.

[0045] Understandably, the evaluation result of the software version to be evaluated is determined based on the difference value of each metric corresponding to each test case.

[0046] In one optional embodiment, the indicator score for each indicator corresponding to each test case is determined based on the difference value of each indicator corresponding to each test case, and the evaluation result of the software version to be evaluated is determined based on the indicator score for each indicator corresponding to each test case.

[0047] This application provides an automatic software evaluation method. First, test requirements are obtained, including information on a reference version of the software and information on the version to be evaluated. Based on the test requirements, multiple test cases are determined. Second, a first value for at least one indicator corresponding to each test case is obtained from running each test case on the reference version software, serving as first test data. A second value for at least one indicator corresponding to each test case is obtained from running each test case on the version to be evaluated, serving as second test data. Then, based on the first test data and the second test data for each test case, a difference value for each indicator corresponding to each test case is determined. The difference value represents the value obtained by subtracting the corresponding first value from the second value. Finally, the evaluation result of the version to be evaluated is determined based on the difference value for each indicator corresponding to each test case.

[0048] This application determines multiple test cases based on testing requirements, identifies the corresponding metric difference value for each test case, and determines the evaluation result of the version to be evaluated based on the metric difference value corresponding to each test case. On the one hand, manual evaluation is inefficient and prone to errors. Compared with manual evaluation methods, this application automatically evaluates the evaluation result of the software version based on testing requirements, which not only improves the efficiency of evaluation but also avoids errors caused by manual evaluation, enabling a comprehensive evaluation of software performance. On the other hand, since different testers may interpret the same test result differently, the accuracy of manual evaluation is low, and there is a lack of unified evaluation rules for test results. The embodiments of this application unify the evaluation rules and improve the accuracy of the evaluation results.

[0049] Please see Figure 2 The document illustrates a flowchart of an automatic software evaluation method provided in an embodiment of this application, applied to a processor of an electronic device. The method specifically includes steps S201 to S209.

[0050] Step S201: Obtain test requirements, which include the software's reference version information and the version to be evaluated.

[0051] Step S202: Determine multiple test cases based on the test requirements.

[0052] Step S203: Obtain the first value of at least one indicator corresponding to each test case obtained by running each test case of the comparison version software from the preset database, as the first test data; and obtain the second value of at least one indicator corresponding to each test case obtained by running each test case of the version software to be evaluated, as the second test data.

[0053] It should be noted that the preset database is used to store data. If the preset database stores a first value of at least one indicator corresponding to each test case obtained by running each test case of the comparison version software, and a second value of at least one indicator corresponding to each test case obtained by running each test case of the version to be evaluated software, then the first value of at least one indicator corresponding to each test case obtained by running each test case of the comparison version software is obtained from the preset database as the first test data, and the second value of at least one indicator corresponding to each test case obtained by running each test case of the version to be evaluated software is obtained as the second test data.

[0054] Step S204: If the preset database does not contain the first value of at least one indicator corresponding to each test case obtained by running each test case with the reference version software, then the first test result corresponding to each test case is obtained by running each test case with the reference version software.

[0055] If the preset data does not contain a first value for at least one metric corresponding to each test case obtained by running each test case with the reference version software, then each test case is run with the reference version EDA software to obtain a first test result for each test case. The first test result includes log information, and the first value for each metric corresponding to that test case can be determined based on the log information.

[0056] Step S205: Extract the first value of the corresponding indicator from the first test result corresponding to each test case, and use the first value of the indicator corresponding to each test case as the first test data of each test case.

[0057] Since the first test result includes the test log information, the first value of the corresponding indicator can be extracted from the first test result corresponding to the test case, and the first value of the indicator corresponding to each test case can be used as the first test data of each test case.

[0058] In one optional embodiment, the first value of the corresponding indicator is extracted from the first test result corresponding to each test case and stored in a preset database, which can provide comparative data for subsequent evaluation of new versions of software.

[0059] Step S206: If the preset database does not contain a second value for at least one metric corresponding to each test case obtained by running each test case of the software version to be evaluated, then the software version to be evaluated runs each test case to obtain a second test result corresponding to each test case.

[0060] If the preset database does not contain a second value for at least one metric corresponding to each test case obtained by running each test case of the software version to be evaluated, then by running each test case of the software version to be evaluated, a second test result corresponding to each test case is obtained. The second test result includes log information, and the second value of each metric corresponding to the test case can be determined based on the log information.

[0061] Step S207: Extract the second value of the corresponding indicator from the second test result corresponding to each test case, and use the second value of the indicator corresponding to each test case as the second test data of each test case.

[0062] Since the second test result includes test log information, the second value of the corresponding indicator can be extracted from the second test result corresponding to the test case, and the second value of the indicator corresponding to each test case can be used as the second test data of each test case.

[0063] In one optional embodiment, the second value of the corresponding indicator is extracted from the second test result corresponding to each test case and stored in a preset database, which can provide comparative data for subsequent evaluation of new versions of software.

[0064] Step S208: Based on the first test data and the second test data corresponding to each test case, determine the difference value of each indicator corresponding to each test case, wherein the difference value represents the value obtained by subtracting the corresponding first value from the second value.

[0065] Step S209: Determine the evaluation result of the version of software to be evaluated based on the difference value of each indicator corresponding to each test case.

[0066] In this embodiment, it can be ensured that the first test data and the second test data corresponding to each test case are obtained. Based on the first test data and the second test data corresponding to each test case, the indicator value corresponding to each test case is determined. Then, based on the indicator value corresponding to each test case, the evaluation result of the software version to be evaluated is determined, which can achieve comprehensive evaluation.

[0067] In one optional embodiment, if some test cases have missing first test data or some test cases have missing second test data, the following approach can be used: (1) This test case is not included in the scoring calculation; (2) If some test cases are missing first test data, the test cases with missing data shall be used as the first test cases. The first test data corresponding to the first test case of the software version closest to the comparison version shall be obtained from the preset database and used as the first test data of the first test case of the comparison version. If some test cases are missing second test data, the test cases with missing data shall be used as the second test cases. The first test data corresponding to the second test case of the software version closest to the version to be evaluated shall be obtained from the preset database and used as the first test data of the second test case of the version to be evaluated.

[0068] (3) Determine that the metric corresponding to the test case is 0.

[0069] Please see Figure 3 The diagram illustrates a flowchart of an automatic software evaluation method provided in this application embodiment, applied to a processor of an electronic device. The method specifically includes steps S301 to S307.

[0070] Step S301: Obtain test requirements, which include the software's reference version information and the version to be evaluated.

[0071] Step S302: Determine multiple test cases based on the test requirements.

[0072] Step S303: Obtain the first value of at least one indicator corresponding to each test case obtained by running each test case of the comparison version software, as the first test data; and obtain the second value of at least one indicator corresponding to each test case obtained by running each test case of the version software to be evaluated, as the second test data.

[0073] Step S304: Based on the first test data and the second test data corresponding to each test case, determine the difference value of each indicator corresponding to each test case, wherein the difference value represents the value obtained by subtracting the corresponding first value from the second value.

[0074] Step S305: Based on the difference value of each metric corresponding to each test case, the first value of each metric, and the preset rules of each metric, determine the optimization amount of each metric corresponding to each test case.

[0075] It should be noted that the difference value, the first value of each indicator, and the preset rule for each indicator for each test case mean that each indicator for each test case includes these three parameters: difference value, first value, and preset rule.

[0076] Understandably, different test cases correspond to different metrics. These metrics are evaluation metrics, and different metric values ​​represent different performance levels. The value range for each metric can be different. For example, the metric "Is the signal integrity analysis result accurate?" has a value range of 0-10. A value of "0" indicates that the signal integrity analysis is completely inaccurate, a value of "4" indicates that the signal integrity analysis is partially accurate, and a value of "10" indicates that the signal integrity analysis is completely accurate. Similarly, the metric "Is the analysis report detailed?" has a value range of 0% to 100%. A value of "0" indicates that the analysis report is completely incomplete, a value of "50%" indicates that the analysis report is partially detailed, and a value of "100%" indicates that the analysis report is completely detailed.

[0077] The first value for each indicator represents the indicator value corresponding to the test cases of the comparison version of the software. The second value for each indicator represents the indicator value corresponding to the test cases of the software to be evaluated. The difference value represents the difference between the second value and the first value. Since the value ranges of different indicators may be different, it is necessary to determine the optimization amount for each indicator based on the preset rules for each indicator. The indicator optimization amount represents the amount of optimization to be done for the corresponding indicator in the software to be evaluated.

[0078] Specifically, step S305 may include the following steps: Step S3051: If there is an indicator whose value range does not include 0 among the indicators corresponding to the test case, take the indicator whose value range does not include 0 as the first indicator, determine the optimized value of each first indicator based on the ratio of the difference value of the first indicator to the corresponding first value, and determine the optimized symbol of each first indicator based on the preset rule of each first indicator.

[0079] Step S3052: Determine the optimization amount for each first indicator based on the optimized value and the corresponding optimization symbol for each first indicator.

[0080] For example, the value range of the metric "CPU utilization" in the test case "simulator parallel computing efficiency" is (0, 100%). The first value corresponding to the metric "CPU utilization" is "80%" and the first value corresponding to the metric "CPU utilization" is "60%". Therefore, the metric "CPU utilization" corresponding to this test case is determined to be the first metric. The difference value of the first metric is "-20%". The optimization value of the first metric is determined to be (60%-80%) / 80%=-0.25. The preset rule of the first metric is "the lower the CPU utilization, the better". Therefore, the optimization sign of the first metric is "+". The optimization amount of the first metric "CPU utilization" is determined to be 0.25.

[0081] Step S3053: If there is an indicator whose value range includes 0 among the indicators corresponding to the test case, take the indicator whose value range includes 0 as the second indicator, determine the optimized value of the second indicator based on the difference value of each second indicator, and determine the optimized symbol of each second indicator based on the preset rule of each second indicator.

[0082] Step S3054: Based on the optimized value of each second indicator and the corresponding optimized symbol, determine the optimization amount of each second indicator.

[0083] For example, the value range of the metric "clock offset" for the test case "hold-time violation" is (-100, 100). The first value corresponding to the metric "hold-time violation" is "0", and the second value corresponding to the metric "hold-time violation" is "60". Therefore, the metric "hold-time violation" for this test case is determined to be the second metric. The difference value of this second metric is "60", and the optimization value of this second metric is determined to be 60. The preset rule for this second metric is "the smaller the hold-time violation, the better". Therefore, the optimization symbol for this second metric is "-". The optimization amount for the second metric "hold-time violation" is determined to be -60.

[0084] For example, please refer to Table 1 below. Test case A corresponds to two test modules, namely module 1 and module 2. Test case A has 5 metrics, namely metric 1, metric 2, metric 3, metric 4, and metric 5. Here, "Golden" represents the reference version, "Ref" represents the version to be evaluated, and "Property" represents the preset rule for that metric. The difference value of each metric and the optimization amount of each metric are determined based on the first and second values ​​of each metric in the test case. Similarly, the metric for each test case can be obtained.

[0085] Table 1 Step S306: Based on the optimization amount of each metric corresponding to each test case, the preset score of each metric, the preset maximum optimization amount of each metric, and the preset maximum deterioration amount of each metric, determine the metric score of each metric corresponding to each test case.

[0086] In one optional embodiment, step S306 includes the following steps: Step S3061: If any of the metrics corresponding to the test case does not correspond to a preset minimum degradation amount and a preset minimum optimization amount, the metric that does not correspond to the preset minimum degradation amount and the preset minimum optimization amount is taken as a third metric, and the metrics of each of the third metrics corresponding to the test case are divided into C1: Where A represents the optimization amount of the third indicator, A1 represents the preset maximum deterioration amount of the third indicator, A2 represents the preset maximum optimization amount of the third indicator, and B represents the preset score corresponding to the third indicator.

[0087] In one optional embodiment, step S306 includes the following steps: Step S3062: If the metrics corresponding to the test case include metrics with preset minimum degradation and preset minimum optimization, the metrics with preset minimum degradation and preset minimum optimization are used as fourth metrics, and the metrics of each fourth metric corresponding to the test case are divided into C2: Wherein, D represents the optimization amount of the fourth indicator, D1 represents the preset maximum deterioration amount of the fourth indicator, D2 represents the preset minimum deterioration amount of the fourth indicator, D3 represents the preset minimum optimization amount of the fourth indicator, D4 represents the preset maximum optimization amount of the fourth indicator, and E represents the preset score corresponding to the indicator.

[0088] For example, based on the indicators in Table 1, the preset score for indicator 1 is 20, the preset score for indicator 2 is 10, the preset score for indicator 3 is 20, the preset score for indicator 4 is 30, and the preset score for indicator 5 is 20. The preset maximum optimization amount and preset maximum degradation amount, etc., for each indicator are preset as follows: Indicator 1: The preset maximum optimization amount is 100%, the preset maximum degradation amount is -100%, the preset minimum optimization amount is 10%, and the preset minimum degradation amount is -10%.

[0089] Indicator 2: The preset maximum optimization amount is 20%, and the preset maximum deterioration amount is -20%.

[0090] Indicator 3: The preset maximum optimization amount is 30%, the preset maximum degradation amount is -30%, the preset minimum optimization amount is 10%, and the preset minimum degradation amount is -10%.

[0091] Indicator 4: The preset maximum optimization amount is 100%, and the preset maximum deterioration amount is -100%.

[0092] Indicator 5: The preset maximum optimization amount is 400, and the preset maximum deterioration amount is -200.

[0093] The specific metrics for the five metrics corresponding to test case A are as follows: Indicator 1: The preset score for Indicator 1 is 20. Indicator 1 is the fourth indicator. The actual optimization amount is +5%, which does not meet the minimum optimization requirement (10%). The score for Indicator C2 is 20.

[0094] Indicator 2: The preset score for Indicator 2 is 10. Indicator 2 is the third indicator, and the actual optimization amount is +10%. It is calculated according to the linear relationship, that is, the indicator score C1 = 10 + 2 * 10 * 10% / (20% - (-20%)) = 15.

[0095] Indicator 3: The preset score for Indicator 3 is 20. Indicator 3 is the fourth indicator. The actual optimization amount is -50%, which is less than the preset maximum deterioration amount of -30%. The indicator score C2 is 0.

[0096] Indicator 4: The preset score for Indicator 4 is 30. Indicator 4 is the third indicator, with an actual optimization amount of +20%. Calculated according to the linear relationship, the indicator score C1 = 30 + 2 * 30 * 20% / (100% - (-100%)) = 36.

[0097] Indicator 5: The preset score for Indicator 5 is 20. Indicator 5 is the third indicator. The actual optimization amount is +200. Calculated according to the linear relationship, the indicator score C1 = 20 + 2*20*200 / (400-(-200))≈33.3.

[0098] Similarly, we can obtain the score for each metric for each test case. Furthermore, we can determine that the total score for all metrics for test case A is 20 + 15 + 0 + 36 + 33.3 = 104.3, and similarly, we can obtain the total score for each test case.

[0099] Step S307: Determine the evaluation result of the software version to be evaluated based on the index score of each index corresponding to each test case.

[0100] Specifically, step S307 includes the following steps: Step S3071: Obtain the total indicator score for each test case based on the indicator score of each indicator corresponding to each test case.

[0101] As can be seen from the foregoing embodiments, by obtaining the index score for each metric corresponding to each test case, the total index score for each test case can be obtained.

[0102] It should be noted that the sum of the preset scores for each metric corresponding to each test case is the preset first score.

[0103] Step S3072: Determine the second score of the version to be evaluated based on the average of the total index scores for each test case.

[0104] For example, please refer to Table 2, which shows the total score of the comparison version. It should be noted that the test requirement corresponds to three test cases: Test Case A, Test Case B, and Test Case C. The total score for each of the three test cases in the comparison version is 100. Therefore, the average of the total score for the three test cases in the comparison version is 100. This is a manually set average of 100 for the total score of the three test cases in the comparison version, meaning the first score is 100.

[0105] Table 2 Please see Figure 3 The table shows the total score of each test case in the version to be evaluated. According to Table 3, the test requirements correspond to three test cases, namely test case A, test case B and test case C. The average total score of the three test cases in the version to be evaluated is 94.1, which is the second score of 94.4.

[0106] Table 3 Step S3073: Determine the evaluation result of the software version to be evaluated based on the second score and the first score.

[0107] In one alternative embodiment, if the second score is greater than the first score, it indicates that the evaluation result of the software version to be evaluated has not achieved progressive improvement.

[0108] For example, the evaluation result of the software version to be evaluated is determined based on the first score and the second score. Continuing with the above embodiments, if the second score of 94.1 is less than the first score of 100, then the evaluation result of the software version to be evaluated is determined to be an improvement that has not achieved progress.

[0109] In one alternative embodiment, if the second score is N times greater than the first score, where N is a number greater than 1.1, it indicates that the evaluation result of the software version to be evaluated has not achieved any progressive improvement.

[0110] Please see Figure 4 It shows a flowchart of an automated software evaluation method. Test Case Execution Management Module: Responsible for interacting with the test case library. Specific functions include: (a) Select a set of test cases according to a certain strategy.

[0111] (b) Run all test cases in the selected test case set.

[0112] Indicator data extraction module: Extracts corresponding indicator data from the output of EDA software.

[0113] Data Management Module: Responsible for interacting with the database that stores historical test case data.

[0114] Data Comparison Module: This module compares the test results of the software version to be evaluated (Ref) with the control version (Golden), calculating the differences in metrics between the two versions. Based on the nature of the metrics, it determines the difference value for each metric corresponding to each test case.

[0115] The scoring report module scores each test case across various metrics based on the difference values ​​for each metric and the preset score for each metric. All metric scores are summed to obtain the test case score. Finally, the average score of all test cases is used to represent the software's score, and a comprehensive scoring report is generated.

[0116] First, a comparison task list (test requirements) is obtained. The data management module determines the target data to be compared based on the comparison task list and reads the target data from the historical test database. If historical data exists (the first and second test data corresponding to the indicator), the obtained historical data is compared. The scoring report module obtains the test case score (total indicator score) for each test case and then generates a scoring report. If historical data (the first and second test data corresponding to the indicator) does not exist in the historical test database, the test case execution management module reads the target test cases from the test cases based on the task list and runs the EDA software based on the target test cases to obtain test result data. The indicator data extraction module extracts the corresponding indicator data from the EDA software output and sends it to the data management module. The obtained historical data is then compared, and the scoring report module obtains the test case score (total indicator score) for each test case and then generates a scoring report.

[0117] Please see Figure 5 This diagram illustrates a structural block diagram of an electronic device 700 provided in an embodiment of this application. The electronic device 700 can be an in-vehicle infotainment system, which can be installed in a vehicle. The electronic device 700 in this application may include one or more of the following components: a processor 711, a memory 712, and one or more application programs, wherein the processor 711 is electrically connected to the memory 712, and the one or more programs are configured to execute the methods described in the foregoing embodiments of the test methods.

[0118] Processor 711 may include one or more processing cores. Processor 711 connects to various parts within the electronic device 700 using various interfaces and lines, and performs various functions and processes data of the electronic device 700 by running or executing instructions, programs, code sets, or instruction sets stored in memory 712, and by calling data stored in memory 712. Optionally, processor 711 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 711 may integrate one or more of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and computer programs; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 711 and may be implemented separately through a communication chip. Specifically, the methods described in the foregoing embodiments can be executed by one or more processors 711.

[0119] In some implementations, memory 712 may include random access memory (RAM) or read-only memory (ROM). Memory 712 can be used to store instructions, programs, code, code sets, or instruction sets. Memory 712 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described below, etc. The data storage area may also store data created by the electronic device 700 during use.

[0120] Please refer to Figure 6 This diagram illustrates a structural block diagram of a computer-readable medium provided in an embodiment of this application. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0121] The computer-readable medium 800 may be an electronic storage device such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable medium 800 includes a non-transitory computer-readable storage medium. The computer-readable medium 800 has storage space for program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

[0122] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0123] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0124] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0125] In this embodiment, the modules / units included in the various devices and products described in the above embodiments can be software modules / units, hardware modules / units, or a combination of both.

[0126] For example, for various devices and products applied to or integrated into a chip, each module / unit can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs that run on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits; for various devices and products applied to or integrated into a chip module, each module / unit can be implemented using hardware methods such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units... It can be implemented using software programs that run on the processor integrated within the chip module. The remaining modules / units (if any) can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its modules / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining modules / units (if any) can be implemented using hardware methods such as circuits.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An automatic evaluation method for software, characterized in that, A processor applied to an electronic device, the method comprising: Obtain test requirements, which include the software's reference version information and the version to be evaluated; Based on the aforementioned testing requirements, multiple test cases were determined. Obtain a first value of at least one indicator corresponding to each test case obtained by running each test case of the comparison version software, as first test data; and obtain a second value of at least one indicator corresponding to each test case obtained by running each test case of the version software to be evaluated, as second test data. Based on the first test data and the second test data corresponding to each test case, the difference value of each indicator corresponding to each test case is determined, and the difference value represents the value obtained by subtracting the corresponding first value from the second value. The evaluation result of the software version to be evaluated is determined based on the difference value of each metric corresponding to each test case.

2. The method according to claim 1, characterized in that, The step of obtaining a first value of at least one indicator corresponding to each test case obtained by running each test case of the comparison version software as first test data, and obtaining a second value of at least one indicator corresponding to each test case obtained by running each test case of the version software to be evaluated as second test data, includes: The first value of at least one indicator corresponding to each test case obtained by running each test case of the comparison version software from a preset database is used as the first test data, and the second value of at least one indicator corresponding to each test case obtained by running each test case of the version software to be evaluated is used as the second test data.

3. The method according to claim 2, characterized in that, Also includes: If the preset database does not contain a first value for at least one indicator corresponding to each test case obtained by running each test case with the reference version software, then the first test result corresponding to each test case is obtained by running each test case with the reference version software. Extract the first value of the corresponding metric from the first test result corresponding to each test case, and use the first value of the metric corresponding to each test case as the first test data of each test case.

4. The method according to claim 2, characterized in that, Also includes: If the preset database does not contain a second value for at least one metric corresponding to each test case obtained by running each test case of the software version to be evaluated, then the second test result corresponding to each test case is obtained by running each test case of the software version to be evaluated. Extract the second value of the corresponding metric from the second test result corresponding to each test case, and use the second value of the metric corresponding to each test case as the second test data of each test case.

5. The method according to claim 1, characterized in that, The step of determining the evaluation result of the software version to be evaluated based on the difference value of each indicator corresponding to each test case includes: Based on the difference value of each metric corresponding to each test case, the first value of each metric, and the preset rules of each metric, determine the optimization amount of each metric corresponding to each test case; Based on the optimization amount of each metric corresponding to each test case, the preset score of each metric, the preset maximum optimization amount of each metric, and the preset maximum deterioration amount of each metric, the metric score of each metric corresponding to each test case is determined. The evaluation result of the software version to be evaluated is determined based on the index score of each index corresponding to each test case.

6. The method according to claim 5, characterized in that, The step of determining the optimization amount for each metric corresponding to each test case based on the difference value of each metric corresponding to each test case, the first value of each metric, and the preset rules of each metric includes: If there is an indicator whose value range does not include 0 among the indicators corresponding to the test case, the indicator whose value range does not include 0 is taken as the first indicator. Based on the ratio of the difference value of the first indicator to the corresponding first value, the optimized value of each first indicator is determined. Based on the preset rule of each first indicator, the optimized symbol of each first indicator is determined. Based on the optimized value of each first indicator and the corresponding optimized symbol, the optimization amount of each first indicator is determined. If there is an indicator whose value range includes 0 among the indicators corresponding to the test case, the indicator whose value range includes 0 corresponding to the test case is taken as the second indicator, the optimized value of the second indicator is determined based on the difference value of each second indicator, and the optimized symbol of each second indicator is determined based on the preset rule of each second indicator. Based on the optimized value and corresponding optimized symbol of each second indicator, the optimization amount of each second indicator is determined.

7. The method according to claim 5, characterized in that, The determination of the indicator score for each indicator corresponding to each test case, based on the optimization amount, preset score, preset maximum optimization amount, and preset maximum degradation amount for each indicator corresponding to each test case, includes: If any of the metrics corresponding to the test case does not correspond to a preset minimum degradation amount or a preset minimum optimization amount, the metric that does not correspond to a preset minimum degradation amount or a preset minimum optimization amount is taken as a third metric, and each of the third metrics corresponding to the test case is classified into C1: Where A represents the optimization amount of the third indicator, A1 represents the preset maximum deterioration amount of the third indicator, A2 represents the preset maximum optimization amount of the third indicator, and B represents the preset score corresponding to the third indicator.

8. The method according to claim 7, characterized in that, Also includes: If any of the metrics corresponding to the test case has a preset minimum degradation amount and a preset minimum optimization amount, then the metric with the preset minimum degradation amount and the preset minimum optimization amount is used as the fourth metric, and each of the fourth metrics corresponding to the test case is classified into C2: Wherein, D represents the optimization amount of the fourth indicator, D1 represents the preset maximum deterioration amount of the fourth indicator, D2 represents the preset minimum deterioration amount of the fourth indicator, D3 represents the preset minimum optimization amount of the fourth indicator, D4 represents the preset maximum optimization amount of the fourth indicator, and E represents the preset score corresponding to the indicator.

9. The method according to claim 5, characterized in that, The sum of the preset scores for each metric corresponding to each test case is the preset first score. The determination of the evaluation result of the software version to be evaluated based on the metric scores for each metric corresponding to each test case includes: The total score for each test case is obtained based on the score of each metric corresponding to each test case. The second score of the version to be evaluated is determined based on the average of the total score for each test case. The evaluation result of the software version to be evaluated is determined based on the second score and the first score.

10. An electronic device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory, the one or more applications are configured to be executed by the one or more processors, and the one or more applications are configured to perform the method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-9.

Images