A performance evaluation method and system of a car-machine automation test equipment

Abstract

The application discloses a kind of performance evaluation method and system of car machine automation test equipment, comprising: collecting the historical test data of car machine automation test equipment in preset test period, construct original data sample, and carry out data preprocessing to it, form standardization data sequence;According to the dynamic grouping of standardization data sequence according to test scene type, each group corresponds to a test scene, and each group data is arranged in descending order according to the preset performance index;Based on the data arranged in descending order, the performance guarantee rate under each test scene is calculated respectively, and the performance guarantee rate is defined as the probability that car machine automation test equipment reaches preset performance threshold under specific test scene;Correlate performance index and guarantee rate threshold under each test scene, and analyze it, output the performance evaluation result of car machine automation test equipment.The present application is helpful for fine performance optimization and troubleshooting of technical personnel, improve the overall integrity and stability of car machine system.

Description

Technical Field

[0001] This invention relates to the technical field of performance testing of vehicle automation equipment, and in particular to a performance evaluation method and system for vehicle automation testing equipment. Background Technology

[0002] With the rapid development of automotive intelligence and connectivity technologies, the functionality and complexity of in-vehicle infotainment systems (IVS) have significantly increased, and their performance and reliability have become one of the important indicators for ensuring the overall quality of vehicles. To effectively ensure the performance stability of IVS, automated testing technology is widely used in the development and quality control of IVS. Automated testing equipment simulates various real-world usage scenarios to conduct automated operation tests on IVS to obtain performance data under different operating conditions and loads, thereby evaluating the overall performance of the system. However, current performance evaluation methods for automated testing equipment mostly adopt static or fixed scenario analysis, which cannot fully reflect the true performance of the equipment under different usage environments.

[0003] The existing performance evaluation methods for vehicle infotainment system automation testing equipment have the following shortcomings: First, they lack effective utilization of historical test data, making it impossible to establish targeted and representative performance analysis samples; second, most test analysis methods do not consider the differences between different test scenarios, resulting in a lack of dynamic adaptability and accuracy in the evaluation results; in addition, existing technologies have not yet formed a reasonable performance index correlation analysis mechanism, making it difficult to systematically reflect the actual performance of vehicle infotainment system automation testing equipment in multiple scenarios and dimensions.

[0004] These defects significantly reduce the accuracy and objectivity of test results, affecting the overall assessment of the reliability of the vehicle infotainment system. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this section, the abstract and title of the invention. Such simplifications or omissions shall not be used to limit the scope of the present invention.

[0006] In view of the aforementioned existing problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by this invention is that the existing performance evaluation methods for vehicle-mounted automated testing equipment have problems such as insufficient utilization of historical data analysis, lack of dynamic scenario adaptability, and insufficient in-depth correlation analysis of performance indicators.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: collecting historical test data of vehicle-mounted automated testing equipment within a preset test cycle, constructing original data samples, and performing data preprocessing to form standardized data sequences;

[0009] The standardized data sequence is dynamically grouped according to the test scenario type, with each group corresponding to a test scenario, and the data in each group is sorted in descending order according to preset performance indicators.

[0010] Based on the data sorted in descending order, the performance guarantee rate for each test scenario is calculated. The performance guarantee rate is defined as the probability that the vehicle automation test equipment reaches a preset performance threshold in a specific test scenario.

[0011] The system correlates performance indicators and guarantee rate thresholds under various test scenarios, analyzes them, and outputs performance evaluation results for the vehicle automation test equipment.

[0012] As a preferred embodiment of the performance evaluation method for the vehicle-mounted automated testing equipment described in this invention, the historical test data includes test response time, test success rate, error type codes, and system resource utilization.

[0013] The preset test cycle includes:

[0014] When the vehicle infotainment software update cycle is less than or equal to 30 days, the testing cycle is set to 7 days.

[0015] When the vehicle infotainment software update cycle is greater than 30 days but less than 90 days, the testing cycle is set to 14 days.

[0016] When the vehicle infotainment software update cycle is greater than or equal to 90 days, the testing cycle is set to 30 days.

[0017] As a preferred embodiment of the performance evaluation method for the vehicle-mounted automated testing equipment described in this invention, the historical test data is preprocessed to form a standardized data sequence, including:

[0018] Missing values ​​in the collected historical test data are filled by interpolation;

[0019] Then, normalize the completed data.

[0020] Based on the needs of different testing scenarios, a sliding window method is used to smooth the normalized data and generate a standardized data sequence.

[0021] As a preferred embodiment of the performance evaluation method for the vehicle-mounted automated testing equipment described in this invention, the standardized data sequence is dynamically grouped according to the test scenario type, and the data in each group is arranged in descending order according to preset performance indicators, including:

[0022] Each set of standardized data is classified according to the preset test scenario type identifier, and dynamically grouped according to the corresponding scenario identifier label;

[0023] Within each group, the data is sorted from highest to lowest according to the numerical value of the preset performance index, resulting in a descending order data sequence.

[0024] As a preferred embodiment of the performance evaluation method for the vehicle infotainment system automated testing equipment described in this invention, the test scenario types include in-vehicle entertainment system stress test, navigation system multi-path planning test, voice interaction concurrent test, and vehicle network security verification test;

[0025] The preset performance indicators include system response time, peak CPU utilization, memory leak rate, task completion success rate, and abnormal interruption frequency.

[0026] As a preferred embodiment of the performance evaluation method for the vehicle-mounted automated testing equipment described in this invention, the performance guarantee rate for each test scenario is calculated based on the data sorted in descending order, including:

[0027] For each test scenario group, count the number n data points within that group that meet the performance metric value of not less than a preset performance threshold, arranged in descending order. p and the total amount of data n in this group t ;

[0028] The performance guarantee rate R is calculated using the following formula:

[0029]

[0030] Where R is the performance guarantee rate in the corresponding test scenario, and n p To meet the performance threshold requirements, the data sample size n t This represents the total sample size in this test scenario;

[0031] The performance guarantee rates for multiple test scenarios were calculated and recorded.

[0032] As a preferred embodiment of the performance evaluation method for the vehicle-mounted automated testing equipment described in this invention, the step of associating performance indicators and guarantee rate thresholds under various test scenarios and analyzing them includes:

[0033] For each test scenario, establish a correlation table between performance metrics and corresponding performance guarantee rates;

[0034] Preset performance guarantee rate thresholds for each test scenario;

[0035] Statistical analysis methods are used to identify performance indicators that fail to reach the performance guarantee rate threshold in certain test scenarios and generate risk warnings.

[0036] A comprehensive performance index matrix is ​​generated based on the performance metrics, performance guarantee rates, and thresholds for each test scenario.

[0037] Based on the comprehensive performance index matrix and the importance weights of the test scenarios, the comprehensive performance score S is calculated using a weighted calculation method.

[0038]

[0039] Where I represents the total number of test scenarios, and R... i For the performance guarantee rate of the i-th test scenario, w i The importance weight of the i-th scene;

[0040] The weights for stress testing of in-vehicle entertainment systems are defined as w1 = 0.25, multi-path planning testing of navigation systems is w2 = 0.25, concurrent voice interaction testing is w3 = 0.30, and vehicle network security verification testing is w4 = 0.20.

[0041] As a preferred embodiment of the performance evaluation method for the vehicle infotainment automated testing equipment described in this invention, the output of performance evaluation results for the vehicle infotainment automated testing equipment includes:

[0042] Based on the comprehensive performance score S and the risk warnings for each test scenario, a graphical performance evaluation report is generated. The evaluation report includes the performance guarantee rate, performance indicator compliance status, potential performance risks and comprehensive score for each test scenario.

[0043] Based on the calculated range of the comprehensive performance score S, the equipment performance is divided into several levels, where:

[0044] When S≥90%, it is judged as excellent, that is, the equipment performance is excellent;

[0045] When 80% ≤ S < 90%, it is judged as good, that is, the equipment performance is relatively good;

[0046] When 70% ≤ S < 80%, it is judged as average, that is, the equipment performance is medium.

[0047] When S < 70%, it is judged as poor, meaning that there is a large room for optimization in the equipment performance;

[0048] For each test scenario, when the performance guarantee rate R under that scenario is... i Less than the scenario-preset performance guarantee rate threshold T i If so, the scenario is marked as a performance risk, where:

[0049] If R i <T i And R i≥T i -10%, generating a medium risk warning;

[0050] If R i <T i -10%, generating a high-risk warning;

[0051] If R i ≥T i If so, it will generate a risk-free or qualified result.

[0052] A preferred embodiment of the performance evaluation system for the vehicle-mounted automated testing equipment described in this invention includes: one or more processors;

[0053] The memory stores operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the process of the performance evaluation method for the aforementioned vehicle automation testing equipment.

[0054] As a preferred embodiment of the computer-readable medium for storing software according to the present invention, the software includes instructions executable by one or more computers, the instructions causing the one or more computers to perform operations, the operations including the process of the performance evaluation method of the aforementioned vehicle automation testing equipment.

[0055] The beneficial effects of this invention are:

[0056] 1. By collecting historical test data from vehicle-mounted automated testing equipment within a preset test cycle, original data samples are constructed, and data consistency is achieved to form a standardized data sequence. This enables effective standardization and unified processing of test data, which can eliminate data deviations caused by data condition aggregation and environmental differences in different test cycles. This is conducive to obtaining a reliable and consistent data foundation, thereby effectively improving the accuracy of evaluation results and the efficiency of data processing.

[0057] 2. By dynamically grouping standardized data sequences according to test scenario types, with each group representing a specific test scenario, and arranging the data in each group in descending order according to default performance indicators, the system achieves accurate identification and classification management of the characteristics of different test scenarios, greatly improving the dynamic flexibility of the evaluation method and ensuring a deep understanding and accurate judgment of the performance of the vehicle system under different scenarios.

[0058] 3. By calculating the performance guarantee rate for each test scenario based on the data sorted in descending order, the transformation from qualitative analysis to quantitative analysis was achieved, which more clearly revealed the performance compliance of the vehicle automation testing equipment in different scenarios, thereby helping to improve equipment performance;

[0059] 4. By effectively integrating performance indicators and guarantee rate data from different test scenarios, the system can systematically reflect the overall performance status of the equipment, provide effective decision-making basis, help technicians to conduct refined performance optimization and troubleshooting, and improve the overall integrity and stability of the vehicle system. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0061] Figure 1 This is a flowchart illustrating the performance evaluation method of the vehicle-mounted automated testing equipment shown in this invention. Detailed Implementation

[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0063] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.

[0064] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0065] According to an embodiment of the present invention, in combination Figure 1 The flowchart shown illustrates a performance evaluation method for in-vehicle automated testing equipment, which specifically includes the following steps:

[0066] S1. Collect historical test data from the vehicle-mounted automated testing equipment within a preset test cycle, construct raw data samples, and preprocess the data to form standardized data sequences. Note the following in this step:

[0067] Based on the update cycle of the vehicle infotainment software version, a preset testing cycle is established, including:

[0068] When the vehicle infotainment software update cycle is less than or equal to 30 days, the testing cycle is set to 7 days.

[0069] When the vehicle infotainment software update cycle is greater than 30 days but less than 90 days, the testing cycle is set to 14 days.

[0070] When the vehicle infotainment software update cycle is greater than or equal to 90 days, the testing cycle is set to 30 days.

[0071] Within the defined testing period, historical test data from the vehicle-mounted automated testing equipment will be collected. This historical test data includes, but is not limited to, test response time, test success rate, error type codes, and system resource utilization.

[0072] The collected historical test data is preprocessed, including:

[0073] For missing values ​​in historical test data, time series interpolation is used to fill in the missing data (such as using the mean, median or linear interpolation of the nearest neighboring data to fill in the missing data) to ensure the continuity and integrity of the data.

[0074] The completed data is then normalized to unify the dimensions and scale of the data (e.g., scaling the data to the [0,1] range) to facilitate subsequent grouping and sorting.

[0075] Based on the needs of different testing scenarios, a fixed-window-length moving average algorithm is applied to the data sequence to eliminate short-term fluctuations in the data and improve the stability and representativeness of the data.

[0076] After data preprocessing, a standardized data sequence is formed, which is represented in the form of an n×m dimensional matrix. Each row corresponds to a data sampling point, and each column corresponds to a test index, which serves as the input data for the subsequent performance evaluation process.

[0077] S2. Dynamically group the standardized data sequences according to the test scenario type, with each group corresponding to one test scenario, and sort the data in each group in descending order according to preset performance indicators. Note that the following should be noted in this step:

[0078] Based on the preset test scenario type information of the vehicle-mounted automated testing equipment, extract all test scenario type identifiers, including:

[0079] The stress test result for an in-vehicle entertainment system is marked as A.

[0080] The identifier for multi-path planning tests in navigation systems is B;

[0081] The identifier for concurrent voice interaction testing is C;

[0082] The identifier for vehicle-to-everything (V2X) security verification testing is D.

[0083] Traverse the standardized data sequence generated in step S1, and dynamically divide the data into corresponding groups according to the scene identifier attached to each data record (each sampling point), forming subsets divided by scene type, specifically including:

[0084] Records identified as being from in-vehicle entertainment system stress tests will be grouped into group G. A ;

[0085] Records identified as part of the navigation system multipath planning test in the data are grouped into group G. B ;

[0086] Records identified as part of the voice interaction concurrent test in the data are grouped into group G. C ;

[0087] Records identified as vehicle-to-everything (V2X) security verification tests in the data are grouped into group G. D .

[0088] For example, within each group, preset performance metrics are extracted for each record in that group, including system response time, peak CPU utilization, memory leak rate, task completion success rate, and abnormal interruption frequency.

[0089] For each group G i (i∈A,B,C,D), based on the preset sorting index (such as prioritizing task completion success rate or system response time as the main sorting criteria), the data within the group are sorted in descending order according to the target index value from high to low;

[0090] As an example, if the task completion success rate is chosen as the sorting metric, the higher the success rate, the higher the ranking; if the system response time is chosen as the sorting metric, the lower the response time, the higher the ranking.

[0091] In an optional implementation, after completing the above dynamic grouping and descending sorting, the resulting descending sorted data is expressed in the following form:

[0092] Define i scene groups (i = 4, corresponding to four scenes A, B, C, and D), each group containing n... i Each set of records, after sorting, can be expressed as:

[0093]

[0094] Among them, G i This represents the grouped dataset for the i-th test scenario. Group G represents the grouping. i The i-th data record after descending sorting, n i Group G represents the grouping. i The number of records in the database.

[0095] S3. Based on the data sorted in descending order, calculate the performance guarantee rate for each test scenario. The performance guarantee rate is defined as the probability that the vehicle automation test equipment will reach a preset performance threshold in a specific test scenario. Note that the following should be noted in this step:

[0096] After completing step S2, multiple groups G based on different test scenarios have been obtained. i Furthermore, the data within the group has been sorted in descending order according to the preset performance indicators;

[0097] Based on preset requirements, for each scenario, determine the corresponding performance indicators and performance thresholds, for example:

[0098] The task completion success rate must be no less than 90%.

[0099] The system response time must be less than or equal to 500ms;

[0100] The peak CPU utilization rate should be less than or equal to 80%.

[0101] For each test scenario group G i Perform the following statistical procedure:

[0102] Traversing G i The data records are arranged in descending order;

[0103] For each record, determine whether the target performance metric value meets the corresponding performance threshold;

[0104] Count the number of records that meet the conditions and the total number of records, and calculate the performance guarantee rate for this test scenario;

[0105] For example, the performance guarantee rate calculation formulas for each test scenario are as follows:

[0106]

[0107] Where R is the performance guarantee rate in the corresponding test scenario, and n p To meet the performance threshold requirements, the data sample size n t This represents the total sample size in this test scenario;

[0108] Performance guarantee rate R for the four scenarios respectively A R B R C R D Perform calculations and record and store the results for use in the subsequent performance analysis in step S4.

[0109] S4. Correlate the performance indicators and guarantee rate thresholds under each test scenario, analyze them, and output the performance evaluation results of the vehicle automation testing equipment. Note that the following should be noted in this step:

[0110] For each test scenario, a corresponding performance metric-guarantee rate correlation table is established using the scenario's preset performance metrics (e.g., system response time, peak CPU utilization, memory leak rate, task completion success rate, abnormal interruption frequency) and the calculated performance guarantee rate as core parameters, as illustrated below:

[0111] Table 1. Relationship between performance indicators and guarantee rate

[0112] <![CDATA[G A ]]> Task completion success rate 88％ <![CDATA[G A ]]> System response time 85％ <![CDATA[G B ]]> Peak CPU usage 91％ <![CDATA[G C ]]> Memory leak rate 75％ <![CDATA[G D ]]> abnormal interrupt frequency 80％

[0113] Each key performance indicator in each test scenario is associated with the corresponding performance guarantee rate for subsequent risk analysis.

[0114] For each test scenario, based on business and system requirements, a minimum acceptable performance guarantee rate is set, i.e., the performance guarantee rate threshold T. i ;For example:

[0115] Scene G A (In-vehicle entertainment system stress test) Performance guarantee rate threshold T A =90%;

[0116] Scene G B Performance guarantee rate threshold T (multipath planning test of navigation system) B =85%;

[0117] Scene G C (Voice interaction concurrency test) performance guarantee rate threshold T C =80%;

[0118] Scene G D (Vehicle Network Security Verification Test) Performance Guarantee Rate Threshold T D =85%.

[0119] For each scenario i, the actual calculated performance guarantee rate R i With the preset threshold T i Comparison:

[0120] If R i ≥T i The test scenario was deemed satisfactory.

[0121] If R i <T i The test scenario was deemed unqualified and posed a performance risk, and was assigned a corresponding risk label based on the risk level.

[0122] Each test scenario is output in the report in the format of "Scenario Name + Performance Guarantee Rate + Compliance Status", for example: "Navigation System Multi-Path Planning Test: Performance Guarantee Rate 85%, Compliant", "Voice Interaction Concurrency Test: Performance Guarantee Rate 75%, Not Compliant (Medium Risk - Need Attention)";

[0123] Based on the judgment result, generate corresponding text prompts, such as:

[0124] The in-vehicle entertainment system failed the stress test and is at medium risk. It is recommended to optimize the task completion success rate indicator.

[0125] The voice interaction concurrency test failed to meet the standards, indicating a high risk. The frequency of abnormal interruptions and CPU peak values ​​need to be optimized.

[0126] For example, the comprehensive performance index matrix can be expressed in the following form:

[0127]

[0128] Among them, R i Let T be the performance guarantee rate for the i-th test scenario. i w is the performance guarantee rate threshold for the i-th test scenario. i Risk is the importance weight of the i-th test scenario. i The risk level of the i-th scenario (e.g., high risk, medium risk, no risk);

[0129] Based on the comprehensive performance index matrix and the importance weight of the test scenarios, the comprehensive performance score S is calculated using a weighted calculation method.

[0130]

[0131] Where I represents the total number of test scenarios, and R... i For the performance guarantee rate of the i-th test scenario, w i The importance weight of the i-th scene;

[0132] For example, the stress test weights for in-vehicle entertainment systems are defined as w1 = 0.25, the multi-path planning test weights for navigation systems are w2 = 0.25, the concurrent voice interaction test weights are w3 = 0.30, and the vehicle network security verification test weights are w4 = 0.20.

[0133] Furthermore, the output includes performance evaluation results for the vehicle automation testing equipment, including:

[0134] Based on the overall performance score S and the risk warnings for each test scenario, a graphical performance evaluation report is generated. The evaluation report includes the performance guarantee rate, performance indicator compliance, potential performance risks and overall score for each test scenario.

[0135] Based on the calculated range of comprehensive performance scores S, the equipment performance is divided into several levels, among which:

[0136] When S≥90%, it is judged as excellent, that is, the equipment performance is excellent;

[0137] When 80% ≤ S < 90%, it is judged as good, that is, the equipment performance is relatively good;

[0138] When 70% ≤ S < 80%, it is judged as average, that is, the equipment performance is medium.

[0139] When S < 70%, it is judged as poor, meaning that there is a large room for optimization in the equipment performance;

[0140] For each test scenario, when the performance guarantee rate R under that scenario is... i Less than the scenario-preset performance guarantee rate threshold T i If so, the scenario is marked as a performance risk, where:

[0141] If R i <T i And R i ≥T i -10%, generating a medium risk warning;

[0142] If R i <T i -10%, generating a high-risk warning;

[0143] If R i ≥T i If so, it will generate a risk-free or qualified result.

[0144] In an optional implementation, based on the scoring results and risk warnings, the system automatically generates visual charts, specifically including:

[0145] The horizontal axis of the bar chart for performance guarantee rates for each test scenario represents the name of each test scenario, and the vertical axis represents the corresponding R. i Label the performance threshold T for each scenario i Reference line, indicating whether the standard is met or not;

[0146] The performance risk distribution radar chart displays the risk level distribution (e.g., no risk, medium risk, high risk) under each test scenario.

[0147] The comprehensive score and performance level display interface generates a graphical rating table that includes a comprehensive score of S and performance levels of Excellent / Good / Average / Poor.

[0148] It should be noted that the risk warning is not only based on the performance guarantee rate R. i With threshold T iThe comparison results also take into account the importance weights of each test scenario. i To generate targeted risk severity level recommendations, specifically including:

[0149] When R i <T i -10% and w i When the value is ≥0.30, it is judged as high risk - key attention, and it is recommended to optimize immediately and pay close attention to this scenario;

[0150] When R i <T i -10% and 0.15≤w i If the value is less than 0.30, it is considered high-risk and optimization is recommended.

[0151] When R i <T i And R i ≥T i -10% and w i When the value is ≥0.30, it is judged as medium risk - requiring attention, and it is recommended to include it in the short-term optimization plan;

[0152] When R i <T i And R i ≥T i -10% and 0.15≤w i When the value is less than 0.30, it is considered a medium-risk value, and medium-term optimization is recommended.

[0153] When R i <T i And w i When the value is less than 0.15, it is considered low risk, and optimization is recommended based on available resources.

[0154] When R i ≥T i At that time, it was determined to be risk-free and no optimization was needed.

[0155] For example, for scenarios with a risk level of medium or higher, customized optimization suggestions are output as follows:

[0156] If it is a high-risk-key or high-risk scenario, improve the performance bottleneck indicators (such as CPU utilization, task success rate, etc.), increase the resource scheduling capability of the scenario, and optimize the testing strategy (such as introducing higher frequency stress tests and optimizing data processing flow). It is recommended to start a special performance optimization project immediately and retest regularly.

[0157] If the scenario is of medium risk - requires attention or medium risk, analyze the secondary indicators that cause performance degradation (such as memory leak rate and response latency), perform local optimization, and recommend including them in the next stage of optimization plan. The optimization priority will be determined based on the subsequent system load and test results.

[0158] For low-risk scenarios, focus on scenario development trends and optimize as needed based on subsequent testing results.

[0159] Specifically, based on the defined rating levels, the evaluation report will present the overall performance score and corresponding performance level in detail in the conclusion section, including:

[0160] When S≥90%, the report outputs: "Overall performance score: XX%, Grade: Excellent, the vehicle automation test equipment has excellent performance and the overall system performance is stable";

[0161] When 80%≤S<90%, the report outputs: "Overall performance score: XX%, Grade: Good, the vehicle automation test equipment performs well, and some scenarios can be further optimized."

[0162] When 70%≤S<80%, the report outputs: "Overall performance score: XX%, level: average, there are multiple medium-risk or some high-risk scenarios, it is recommended to optimize step by step";

[0163] When S < 70%, the report outputs: "Overall performance score: XX%, grade: poor, there are multiple high-risk scenarios, the system performance is obviously insufficient, and comprehensive optimization is needed as soon as possible."

[0164] For example, the output structure of the evaluation report is shown in the table below:

[0165] Table 2. Structure of the Assessment Report

[0166]

[0167]

[0168] In optional implementations, the evaluation results are generated into electronic reports in PDF, HTML, or JSON format, which are automatically pushed to the user interface for display or stored in a local / cloud database for subsequent review and analysis.

[0169] The aforementioned data preprocessing method for the collected data can be implemented using existing technologies, and will not be described in detail in this embodiment.

[0170] In addition to the above embodiments, other aspects of the present invention also disclose a performance evaluation system for vehicle automation testing equipment, including one or more processors and a memory.

[0171] The memory is used to store operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the flow of the performance evaluation method of the vehicle automation testing equipment described in the foregoing embodiments, in particular... Figure 1 The flowchart of the method is shown.

[0172] Other aspects disclosed in the embodiments of the present invention also propose a computer-readable medium for storing software including instructions executable by one or more computers, which, upon execution, cause the one or more computers to perform operations including the flow of the performance evaluation method for the vehicle-mounted automated testing equipment of the foregoing embodiments, particularly... Figure 1 The flowchart of the method is shown.

[0173] It should be recognized that embodiments of the present invention may be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium.

[0174] The method can be implemented using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program in the computer program, wherein the storage medium is configured such that the computer operates in a specific and predefined manner.

[0175] Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system; however, if required, the program can be implemented in assembly or machine language.

[0176] In any case, the language can be either compiled or interpreted.

[0177] Furthermore, for this purpose, the program can run on programmed application-specific integrated circuits.

[0178] The processes described herein (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.

[0179] Furthermore, the method can be implemented in any suitable computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices.

[0180] Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether portable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein.

[0181] Furthermore, machine-readable code, or parts thereof, can be transmitted via wired or wireless networks.

[0182] When such media includes instructions or programs that combine with a microprocessor or other data processor to implement the steps described above, the invention described herein includes these and other different types of non-transitory computer-readable storage media.

[0183] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A performance evaluation method for vehicle-mounted automated testing equipment, characterized in that, include: Collect historical test data from the vehicle-mounted automated testing equipment within a preset test cycle, construct raw data samples, and preprocess the data to form standardized data sequences; The historical test data includes test response time, test success rate, error type codes, and system resource utilization. The preset test cycle includes: When the vehicle infotainment software update cycle is less than or equal to 30 days, the testing cycle is set to 7 days. When the vehicle infotainment software update cycle is greater than 30 days but less than 90 days, the testing cycle is set to 14 days. When the vehicle infotainment software update cycle is greater than or equal to 90 days, the testing cycle is set to 30 days. The standardized data sequence is dynamically grouped according to the test scenario type, with each group corresponding to a test scenario, and the data in each group is sorted in descending order according to a preset performance index. This includes: classifying each group of standardized data according to a preset test scenario type identifier, and dynamically grouping it according to the identifier label of the corresponding scenario; sorting the data in each group from high to low according to the value of the preset performance index to obtain a descending order data sequence. The test scenarios include stress testing of in-vehicle entertainment systems, multi-path planning testing of navigation systems, concurrent voice interaction testing, and vehicle network security verification testing. The preset performance indicators include system response time, peak CPU utilization, memory leak rate, task completion success rate, and abnormal interruption frequency. Based on the data sorted in descending order, the performance guarantee rate for each test scenario is calculated. The performance guarantee rate is defined as the probability that the vehicle automation test equipment reaches a preset performance threshold in a specific test scenario. For each test scenario group, count the number of data points within that group that meet the performance metric value of not less than a preset performance threshold, arranged in descending order. and the total amount of data within this group ; Calculate the performance guarantee rate using the following formula : in, To ensure the performance rate in the corresponding test scenario, To meet the data sample size requirements for the performance threshold, This represents the total sample size in this test scenario; Calculate and record the performance guarantee rates for multiple test scenarios; The system associates and analyzes performance metrics and guarantee rate thresholds across various test scenarios, including: associating each key performance metric in each test scenario with its corresponding performance guarantee rate for subsequent risk analysis; establishing a correlation table between performance metrics and their corresponding performance guarantee rates for each test scenario; pre-setting performance guarantee rate thresholds for each test scenario; using statistical analysis methods to identify performance metrics that fail to reach their performance guarantee rate thresholds in certain test scenarios and generating risk warnings; and generating a comprehensive performance metric matrix based on the performance metrics, performance guarantee rates, and thresholds for each test scenario. The comprehensive performance index matrix is ​​expressed in the following form: in, Let i be the performance guarantee rate for the i-th test scenario. Let be the performance guarantee rate threshold for the i-th test scenario. Let i be the importance weight of the i-th test scenario. The risk level of the i-th scenario; Based on the comprehensive performance index matrix and the importance weights of the test scenarios, the comprehensive performance score S is calculated using a weighted calculation method. in, This represents the total number of test scenarios. For the first Performance guarantee rate for each test scenario For the first The importance weight of each scenario; Define the stress test weights for in-vehicle entertainment systems Multi-path planning test weights for navigation systems Voice interaction concurrency test weights Weighting of vehicle-to-everything (V2X) security verification test ; Risk warnings are not only based on performance guarantee rates With threshold The comparison results also take into account the importance weight of each test scenario. To generate targeted risk severity level recommendations, specifically including: when and If the scenario is deemed high-risk and requires close monitoring, it is recommended to optimize the scenario immediately and focus on it closely. when and If the risk is deemed high, optimization is recommended as a priority. when and and At that time, it was determined to be of medium risk - requiring attention, and it is recommended to include it in the short-term optimization plan; when and and At that time, it was determined to be of medium risk, and medium-term optimization was recommended; when and If the risk level is low, optimization is recommended based on available resources. when At that time, it was determined to be risk-free and no optimization was needed; Output the performance evaluation results of the aforementioned vehicle automation testing equipment.

2. The performance evaluation method for the vehicle-mounted automated testing equipment according to claim 1, characterized in that, The historical test data is preprocessed to form a standardized data sequence, including: Missing values ​​in the collected historical test data are filled by interpolation; Then, normalize the completed data. Based on the needs of different testing scenarios, a sliding window method is used to smooth the normalized data and generate a standardized data sequence.

3. The performance evaluation method for the vehicle-mounted automated testing equipment according to claim 1, characterized in that, Output the performance evaluation results of the vehicle automation testing equipment, including: Based on the comprehensive performance score S and the risk warnings for each test scenario, a graphical performance evaluation report is generated. The evaluation report includes the performance guarantee rate, performance indicator compliance status, potential performance risks and comprehensive score for each test scenario. Based on the calculated range of the comprehensive performance score S, the equipment performance is divided into several levels, where: when When the equipment is rated as excellent, it is considered to have excellent performance. when When the time is right, it is judged as good, meaning the equipment performance is relatively good; when At that time, it was judged as average, that is, the equipment performance was medium; when When the time is right, it is judged as poor, meaning that there is significant room for improvement in the equipment performance; For each test scenario, the performance guarantee rate under that scenario Less than the performance guarantee rate threshold preset for the scenario If so, the scenario is marked as a performance risk, where: like and Generate a medium-risk warning; like Generate a high-risk warning; like If so, it will generate a risk-free or qualified result.

4. A performance evaluation system for vehicle-mounted automated testing equipment, characterized in that, include: One or more processors; The memory stores operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the flow of a performance evaluation method for an in-vehicle automated testing device as described in any one of claims 1 to 3.

5. A computer-readable medium for storing software, characterized in that: The software includes instructions executable by one or more computers, which, through execution, cause the one or more computers to perform operations, including the flow of a performance evaluation method for an in-vehicle automated testing device as described in any one of claims 1 to 3.

Images