Performance evaluation method and system for automatic testing equipment of in-vehicle infotainment system

Abstract

The invention discloses a performance evaluation method and system for automatic testing equipment of a vehicle machine, and the method comprises the steps: collecting the historical testing data of the automatic testing equipment of the vehicle machine in a preset testing period, constructing an original data sample, and carrying out the data preprocessing of the original data sample, and forming a standardized data sequence; dynamically grouping the standardized data sequence according to the test scene type, each group corresponding to one test scene, and arranging each group of data in a descending order according to a preset performance index; based on the data arranged in the descending order, the performance guarantee rate under each test scene is calculated, and the performance guarantee rate is defined as the probability that the automatic test equipment of the vehicle machine reaches a preset performance threshold value under the specific test scene; and associating the performance indexes and the guarantee rate thresholds in each test scene, analyzing the performance indexes and the guarantee rate thresholds, and outputting a performance evaluation result about the automatic test equipment of the in-vehicle infotainment system. According to the invention, a technician is helped to carry out refined performance optimization and troubleshooting, and the overall integrity and stability of the vehicle-mounted information entertainment system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of performance testing of in-vehicle infotainment system automation equipment, and particularly relates to a method and system for performance evaluation of in-vehicle infotainment system automation testing equipment. Background Art

[0002] With the rapid development of automotive intelligent and connected technologies, the functions and complexity of in-vehicle infotainment systems (hereinafter referred to as in-vehicle systems) have been significantly improved, and their performance and reliability have become one of the important indicators to ensure the overall quality of vehicles. To effectively ensure the performance stability of in-vehicle systems, in-vehicle infotainment system automation testing technology has been widely used in the development and quality control of in-vehicle systems. In-vehicle infotainment system automation testing equipment simulates a variety of actual usage scenarios to conduct automated operation tests on in-vehicle systems, so as to obtain performance data of the system under different working conditions and load conditions, and then evaluate the overall performance of the system. However, most of the current performance evaluation methods for in-vehicle infotainment system automation testing equipment adopt static or fixed scenario analysis, which cannot fully reflect the true performance of the equipment under different usage environments.

[0003] The existing performance evaluation methods for in-vehicle infotainment system automation testing equipment mainly have the following deficiencies: First, there is a lack of effective utilization of historical test data, and it is impossible to establish a performance analysis sample with strong pertinence and representativeness. Second, most test analysis methods do not consider the differences of different test scenarios, resulting in the evaluation results lacking dynamic adaptability and accuracy. In addition, the existing technology has not formed a reasonable performance index correlation analysis mechanism, making it difficult to systematically reflect the actual performance of in-vehicle infotainment system automation testing equipment in multiple scenarios and dimensions.

[0004] These defects significantly reduce the accuracy and objectivity of test results and affect the comprehensive evaluation of the performance and reliability of in-vehicle systems. Summary of the Invention

[0005] The purpose of this part is to outline some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Simplifications or omissions may be made in this part, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this part, the abstract, and the title. Such simplifications or omissions shall not be used to limit the scope of the present invention.

[0006] In view of the above existing problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by the present invention is that the existing performance evaluation methods for in-vehicle infotainment system automation testing equipment have problems such as insufficient utilization of historical data analysis, lack of dynamic scenario adaptability, and insufficient in-depth analysis of performance index correlation.

[0008] To solve the above technical problems, the present invention provides the following technical solutions: collect historical test data of the in-vehicle infotainment (IVI) automation test equipment within a preset test cycle, construct an original data sample, and perform data preprocessing on it to form a standardized data sequence;

[0009] Dynamically group the standardized data sequence according to the test scenario type, with each group corresponding to a test scenario, and sort the data in each group in descending order according to a preset performance index;

[0010] Based on the data sorted in descending order, calculate the performance guarantee rate under each test scenario, where the performance guarantee rate is defined as the probability that the IVI automation test equipment reaches a preset performance threshold under a specific test scenario;

[0011] Associate the performance index and the guarantee rate threshold under each test scenario, and analyze them to output a performance evaluation result of the IVI automation test equipment.

[0012] As a preferred solution of the performance evaluation method of the IVI automation test equipment of the present invention, the historical test data includes test response time, test success rate, error type code, and system resource occupancy rate;

[0013] The preset test cycle includes:

[0014] When the in-vehicle software version update cycle is less than or equal to 30 days, the test cycle is set to 7 days;

[0015] When the in-vehicle software version update cycle is greater than 30 days and less than 90 days, the test cycle is set to 14 days;

[0016] When the in-vehicle software version update cycle is greater than or equal to 90 days, the test cycle is set to 30 days.

[0017] As a preferred solution of the performance evaluation method of the IVI automation test equipment of the present invention, performing data preprocessing on the historical test data to form a standardized data sequence includes:

[0018] Interpolate and complete the missing values in the collected historical test data;

[0019] Then perform normalization processing on the completed data;

[0020] According to the requirements of different test scenarios, use the sliding window method to smooth the normalized data to generate a standardized data sequence.

[0021] As a preferred solution of the performance evaluation method of the IVI automation test equipment of the present invention, dynamically grouping the standardized data sequence according to the test scenario type and sorting the data in each group in descending order according to a preset performance index includes:

[0022] Classify each group of standardized data according to the preset test scenario type identifier, and perform dynamic grouping according to the identification tags corresponding to the scenarios;

[0023] Sort the data within each group in descending order according to the numerical values of the preset performance indicators to obtain a data sequence arranged in descending order.

[0024] As a preferred solution of the performance evaluation method of the in-vehicle infotainment system automation test equipment described in the present invention, the test scenario types include in-vehicle entertainment system stress test, navigation system multi-path planning test, voice interaction concurrency test, and vehicle networking security verification test;

[0025] The preset performance indicators include system response time, peak CPU occupancy rate, memory leakage rate, task completion success rate, and abnormal interruption frequency.

[0026] As a preferred solution of the performance evaluation method of the in-vehicle infotainment system automation test equipment described in the present invention, based on the data arranged in descending order, calculate the performance guarantee rate under each test scenario respectively, including:

[0027] For each test scenario group, count the number n of data in the data arranged in descending order within the group that meet the performance indicator value not lower than the preset performance threshold p and the total amount n of data within the group t ;

[0028] Use the following formula to calculate the performance guarantee rate R:

[0029]

[0030] where R is the performance guarantee rate under the corresponding test scenario, n p is the data sample size that meets the performance threshold requirement, and n t is the total sample size under this test scenario;

[0031] Calculate and record the performance guarantee rates under multiple test scenarios respectively.

[0032] As a preferred solution of the performance evaluation method of the in-vehicle infotainment system automation test equipment described in the present invention, associate the performance indicators and guarantee rate thresholds under each test scenario, and analyze them, including:

[0033] For each test scenario, establish an association relationship table between the performance indicators and the corresponding performance guarantee rate respectively;

[0034] Preset the performance guarantee rate thresholds for each test scenario;

[0035] Adopt statistical analysis methods to identify the performance indicators that do not reach the performance guarantee rate threshold in some test scenarios, and generate risk warnings;

[0036] Generate a comprehensive performance index matrix based on the performance indexes, performance guarantee rates, and thresholds under each test scenario;

[0037] According to the comprehensive performance index matrix and combined with the importance weights of the test scenarios, calculate the comprehensive performance score S, and the score adopts a weighted calculation method:

[0038]

[0039] where I is the total number of test scenarios, R i is the performance guarantee rate of the i-th test scenario, and w i is the importance weight of the i-th scenario;

[0040] Define that the weight of the in-vehicle entertainment system stress test w1 = 0.25, the weight of the navigation system multi-path planning test w2 = 0.25, the weight of the voice interaction concurrency test w3 = 0.30, and the weight of the vehicle networking security verification test w4 = 0.20.

[0041] As a preferred solution of the performance evaluation method of the in-vehicle automation test equipment described in the present invention, output the performance evaluation result of the in-vehicle automation test equipment, including:

[0042] Generate a charted performance evaluation report according to the comprehensive performance score S and the risk prompts under each test scenario, and the evaluation report includes the performance guarantee rate, the compliance of the performance index, the potential performance risk, and the comprehensive score under each test scenario;

[0043] According to the interval value of the calculated comprehensive performance score S, divide the equipment performance into several levels, where:

[0044] When S ≥ 90%, it is determined as excellent, that is, the equipment performance is excellent;

[0045] When 80% ≤ S < 90%, it is determined as good, that is, the equipment performance is good;

[0046] When 70% ≤ S < 80%, it is determined as general, that is, the equipment performance is medium;

[0047] When S < 70%, it is determined as poor, that is, there is a large room for optimization in the equipment performance;

[0048] For each test scenario, when the performance guarantee rate R i is less than the performance guarantee rate threshold T i preset for the scenario, then mark this scenario as a performance risk, where:

[0049] If R i < T i and R i≥T i -10%, generate a medium-risk prompt;

[0050] If R i <T i -10%, generate a high-risk prompt;

[0051] If R i ≥T i , then generate no risk or compliance.

[0052] As a preferred embodiment of the performance evaluation system of the in-vehicle infotainment system automation testing device described in the present invention, it includes: one or more processors;

[0053] A memory storing operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including the process of the performance evaluation method of the in-vehicle infotainment system automation testing device as described above.

[0054] As a preferred embodiment of a computer-readable medium storing software described in the present invention, the software includes instructions executable by one or more computers, and such execution causes the one or more computers to perform operations including the process of the performance evaluation method of the in-vehicle infotainment system automation testing device as described above.

[0055] Advantages of the present invention:

[0056] 1. By collecting historical test data of the in-vehicle infotainment system automation testing device within a preset test cycle, constructing an original data sample, and then performing data consistency to form a standardized data sequence, it realizes the effective standardization and unified processing of test data, can eliminate data deviations caused by data condition aggregation and environmental differences in different test cycles, is conducive to obtaining a reliable and consistent data basis, and thus effectively improves the accuracy of the evaluation results and the efficiency of data processing;

[0057] 2. By dynamically grouping the standardized data sequence according to the test scenario type, with each group corresponding to a specific test scenario, and arranging the data in each group in descending order according to the default performance indicators, it realizes the precise identification and classification management of different test scenario characteristics, greatly improves the dynamic flexibility of the evaluation method, and ensures an in-depth understanding and accurate judgment of the performance of the in-vehicle infotainment system in different scenarios;

[0058] 3. By calculating the performance guarantee rate in each test scenario based on the data arranged in descending order, it realizes the transformation from qualitative analysis to quantitative analysis, more clearly reveals the performance compliance of the in-vehicle infotainment system automation testing device in different scenarios, and thus is conducive to improving the device performance;

[0059] 4. By effectively integrating the performance indicators and guarantee rate data of different test scenarios, the overall performance status of the device can be systematically reflected, providing an effective basis for decision-making, which helps technicians to conduct refined performance optimization and troubleshooting, and improve the overall integrity and stability of the in-vehicle system. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. Among them:

[0061] Figure 1 It is a schematic flow chart of the performance evaluation method for the in-vehicle automation test equipment shown in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0062] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the following will make a detailed description of the specific embodiments of the present invention with reference to the drawings of the specification. Obviously, the described embodiments are some embodiments of the present invention, rather than all embodiments.

[0063] Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0064] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0065] According to an embodiment of the present invention, in combination with Figure 1 the flowchart shown, a performance evaluation method for an in-vehicle automation test equipment specifically includes the following steps:

[0066] S1. Collect the historical test data of the in-vehicle automation test equipment within a preset test period, construct an original data sample, and perform data preprocessing on it to form a standardized data sequence. Among them, it should be noted in this step that:

[0067] According to the update cycle of the in-vehicle software version, the preset test period includes:

[0068] When the update cycle of the in-vehicle software version is less than or equal to 30 days, the test period is set to 7 days;

[0069] When the vehicle computer software version update cycle is greater than 30 days and less than 90 days, the test cycle is set to 14 days;

[0070] When the vehicle software version update cycle is greater than or equal to 90 days, the test cycle is set to 30 days.

[0071] During the determined test cycle, historical test data of the vehicle-mounted automated test equipment is collected. The historical test data includes but is not limited to: test response time, test success rate, error type code, and system resource usage.

[0072] Preprocess the collected historical test data, including:

[0073] For missing values in historical test data, time series interpolation methods are used to fill them (such as using the mean, median or linear interpolation of adjacent data to fill missing data) to ensure the continuity and integrity of the data;

[0074] Then normalize the completed data to unify the dimension and scale of the data (such as scaling the data to the interval [0,1]) to facilitate subsequent grouping and sorting;

[0075] Based on the needs of different test scenarios, a moving average algorithm with a fixed window length is applied to the data sequence to eliminate short-term fluctuations in the data and improve the stability and representativeness of the data;

[0076] After data preprocessing, a standardized data sequence is formed. The standardized data sequence is represented in the form of an n×m dimensional matrix, where each row corresponds to a data sampling point and each column corresponds to a test indicator, which serves as the input data for the subsequent performance evaluation process.

[0077] S2. Dynamically group the standardized data sequence according to the test scenario type, with each group corresponding to a test scenario, and sort the data in descending order according to the preset performance indicators.

[0078] Extract all test scenario type identifiers based on the test scenario type information preset by the vehicle-mounted automated test equipment, including:

[0079] The in-vehicle entertainment system stress test is marked as A;

[0080] The navigation system multi-path planning test is marked as B;

[0081] The voice interaction concurrency test is marked as C;

[0082] The identification mark for the Internet of Vehicles security verification test is D.

[0083] Traverse the standardized data sequence generated in step S1, and dynamically divide the data into corresponding groups according to the scenario identifiers attached to each data record (each sampling point), forming sub-datasets divided by scenario types, specifically including:

[0084] Classify the records in the data marked as in-vehicle entertainment system stress tests into group G A ;

[0085] Classify the records in the data marked as navigation system multi-path planning tests into group G B ;

[0086] Classify the records in the data marked as voice interaction concurrency tests into group G C ;

[0087] Classify the records in the data marked as vehicle networking security verification tests into group G D .

[0088] Exemplarily, within each grouping, extract the preset performance metrics of each record in the group, including system response time, peak CPU occupancy rate, memory leak rate, task completion success rate, and abnormal interruption frequency.

[0089] For each grouping G i (i ∈ A, B, C, D), according to the preset sorting metrics (such as primarily using the task completion success rate or system response time as the main sorting basis), sort the data within the grouping in descending order according to the target metric values;

[0090] As an example, if the task completion success rate is selected as the sorting metric, the higher the success rate, the higher the ranking; if the system response time is selected as the sorting metric, the lower the response time, the higher the ranking.

[0091] In an alternative embodiment, after completing the above dynamic grouping and descending sorting, the expression form of the sorted data obtained is as follows:

[0092] Define a total of i scenario groupings (i = 4, corresponding to four scenarios A, B, C, D), each group contains n i records, and after sorting, each group of data can be expressed as:

[0093]

[0094] Among them, G i represents the grouped data set of the i-th test scenario, represents the i-th data record after descending sorting in group G i , n i represents the number of records in group G i .

[0095] S3. Based on the data arranged in descending order, calculate the performance guarantee rate for each test scenario. The performance guarantee rate is defined as the probability that the in-vehicle infotainment system automation test equipment reaches the preset performance threshold under a specific test scenario. It should be noted in this step that:

[0096] After completing step S2, multiple groups G based on different test scenarios have been obtained i , and the data within the group has been arranged in descending order according to the preset performance indicators;

[0097] According to the preset requirements, for each scenario, determine the corresponding performance indicators and performance thresholds. For example:

[0098] The requirement for the task completion success rate is not less than 90%;

[0099] The requirement for the system response time is less than or equal to 500 ms;

[0100] The requirement for the peak CPU occupancy rate is less than or equal to 80%;

[0101] For each test scenario group G i , perform the following statistical process:

[0102] Traverse the data records arranged in descending order in G i ;

[0103] For the target performance indicator value of each record, determine whether it meets the corresponding performance threshold;

[0104] Count the number of records that meet the conditions and the total number of records, and calculate the performance guarantee rate for this test scenario;

[0105] Exemplarily, the calculation formula for the performance guarantee rate in each test scenario is as follows:

[0106]

[0107] Among them, R is the performance guarantee rate for the corresponding test scenario, and n p is the sample size of the data that meets the performance threshold requirement, and n t is the total sample size for this test scenario;

[0108] Calculate the performance guarantee rates R A , R B , R C , R D for the four scenarios respectively, and record and store the results for use in the performance analysis in the subsequent step S4.

[0109] S4. Correlate the performance indicators and guarantee rate thresholds for each test scenario, analyze them, and output the performance evaluation results for the in-vehicle infotainment system automation test equipment. It should be noted in this step that:

[0110] For each test scenario, using the preset performance metrics of that scenario (e.g., system response time, peak CPU occupancy rate, memory leak rate, task completion success rate, abnormal interruption frequency) and the calculated performance guarantee rate as core parameters, establish a corresponding performance metric - guarantee rate association table, as shown below:

[0111] Table 1. Performance metric - guarantee rate association table

[0112] Scenario type Performance metric Guaranteed rate of this scenario's performance <![CDATA[G A > Success rate of task completion 88％ <![CDATA[G A > System response time 85％ <![CDATA[G B > Peak CPU occupancy rate 91％ <![CDATA[G C > Memory leak rate 75％ <![CDATA[G D > Frequency of abnormal interruption 80％

[0113] For each key performance metric under each test scenario, it is associated with the corresponding performance guarantee rate for subsequent risk analysis;

[0114] For each test scenario, according to business and system requirements, set the minimum acceptable value of the performance guarantee rate, i.e., the performance guarantee rate threshold T i ; For example:

[0115] Scenario G A (In-vehicle entertainment system stress test) performance guarantee rate threshold T A = 90%;

[0116] Scenario G B (Navigation system multi-path planning test) performance guarantee rate threshold T B = 85%;

[0117] Scenario G C (Voice interaction concurrency test) performance guarantee rate threshold T C = 80%;

[0118] Scenario G D (Vehicle networking security verification test) performance guarantee rate threshold T D = 85%.

[0119] For each scenario i, compare the actually calculated performance guarantee rate R i with the preset threshold T i as follows:

[0120] If R i ≥ T i , this test scenario is determined to meet the standard;

[0121] If R i < T i , this test scenario is determined to not meet the standard, there is a performance risk, and assign a corresponding risk label in combination with the risk level;

[0122] Each test scenario is output in the report in the form of "Scenario Name + Performance Assurance Rate + Compliance Status", for example: "Navigation System Multi-Path Planning Test: Performance Assurance Rate 85%, Compliant", "Voice Interaction Concurrent Test: Performance Assurance Rate 75%, Not Compliant (Medium Risk - Attention Required)";

[0123] According to the judgment result, generate corresponding text prompts, such as:

[0124] The stress test of the in-vehicle entertainment system is not compliant, with medium risk. It is recommended to optimize the task completion success rate index;

[0125] The voice interaction concurrent test is not compliant, with high risk. It is necessary to focus on optimizing the abnormal interruption frequency and CPU peak value.

[0126] Exemplarily, the expression form of the comprehensive performance index matrix is as follows:

[0127]

[0128] Among them, R i is the performance assurance rate of the i-th test scenario, T i is the performance assurance rate threshold of the i-th test scenario, w i is the importance weight of the i-th test scenario, Risk i is the risk level of the i-th scenario (such as high risk, medium risk, no risk);

[0129] According to the comprehensive performance index matrix, combined with the importance weight of the test scenario, calculate the comprehensive performance score S. The scoring adopts a weighted calculation method:

[0130]

[0131] Among them, I is the total number of test scenarios, R i is the performance assurance rate of the i-th test scenario, w i is the importance weight of the i-th scenario;

[0132] Exemplarily, define the weight of the in-vehicle entertainment system stress test w1 = 0.25, the weight of the navigation system multi-path planning test w2 = 0.25, the weight of the voice interaction concurrent test w3 = 0.30, and the weight of the vehicle networking security verification test w4 = 0.20.

[0133] Furthermore, output the performance evaluation results of the in-vehicle automation test equipment, including:

[0134] According to the comprehensive performance score S and the risk prompts under each test scenario, generate a charted performance evaluation report. The evaluation report includes the performance assurance rate, performance index compliance status, potential performance risks, and comprehensive score under each test scenario;

[0135] According to the interval value of the calculated comprehensive performance score S, the device performance is divided into several levels, where:

[0136] When S ≥ 90%, it is judged as excellent, that is, the device performance is excellent;

[0137] When 80% ≤ S < 90%, it is judged as good, that is, the device performance is better;

[0138] When 70% ≤ S < 80%, it is judged as average, that is, the device performance is medium;

[0139] When S < 70%, it is judged as poor, that is, there is a large room for optimization in the device performance;

[0140] For each test scenario, when the performance guarantee rate R i is less than the performance guarantee rate threshold T i preset for the scenario, then this scenario is marked as a performance risk, where:

[0141] If R i < T i and R i ≥ T i - 10%, a medium - risk prompt is generated;

[0142] If R i < T i - 10%, a high - risk prompt is generated;

[0143] If R i ≥ T i , then a risk - free or compliant prompt is generated.

[0144] In an alternative implementation, based on the scoring results and risk prompts, the system automatically generates visual charts, specifically including:

[0145] For the bar chart of the performance guarantee rate of each test scenario, the horizontal axis is the name of each test scenario, and the vertical axis is the corresponding R i , and the performance threshold T i reference line for each scenario is marked to show the compliant / non - compliant status;

[0146] The radar chart of performance risk distribution takes each test scenario as a dimension to show the risk level distribution (such as risk - free, medium - risk, high - risk) for each scenario;

[0147] The comprehensive score and performance level display interface generates a graphical score table including the comprehensive score S and the excellent / good / average / poor performance levels.

[0148] It should be noted that the risk prompt is not only based on the performance guarantee rate R i and the threshold T iThe comparison results also comprehensively consider the importance weight w of each test scenario i , to generate targeted risk severity recommendations, including:

[0149] When R i <T i -10% and w i When the value is ≥0.30, it is judged as high risk and needs to be paid close attention to. It is recommended to optimize and focus on this scenario immediately.

[0150] When R i <T i -10% and 0.15≤w i When <0.30, it is judged as high risk and optimization is recommended as a priority;

[0151] When R i <T i And R i ≥T i -10% and w i When the value is ≥0.30, it is judged as medium risk - requiring attention, and it is recommended to be included in the short-term optimization plan;

[0152] When R i <T i And R i ≥T i -10% and 0.15≤w i When <0.30, it is judged as medium risk and mid-term optimization is recommended;

[0153] When R i <T i And w i When <0.15, it is judged as low risk and optimization is recommended according to resource conditions;

[0154] When R i ≥T i , it is judged as risk-free and no optimization is required.

[0155] For example, for scenarios with a medium risk level or above, customized optimization suggestions are output as follows:

[0156] If the scenario is high-risk or high-risk, improve the performance bottleneck indicators (such as CPU usage and task success rate) in that scenario, increase the resource scheduling capabilities of the scenario, and optimize the testing strategy (such as introducing more frequent stress testing and optimizing data processing processes). It is recommended to immediately initiate a special performance optimization project and conduct regular retests.

[0157] If the scenario is medium risk - requiring attention or medium risk, analyze the secondary indicators that lead to performance degradation (such as memory leak rate and response latency), perform local optimization, and recommend inclusion in the next phase of optimization plan. The optimization priority will be determined based on subsequent system load and test results.

[0158] If it is a low-risk scenario, pay attention to the development trend of the scenario and optimize it according to the subsequent testing results.

[0159] Specifically, based on the defined scoring levels, the evaluation report will present the comprehensive performance score and the corresponding performance level in detail in the conclusion section, including:

[0160] When S ≥ 90%, the report outputs: "Comprehensive performance score: XX%, grade: excellent, the vehicle computer automation test equipment has excellent performance, and the overall performance of the system is stable";

[0161] When 80%≤S<90%, the report outputs: "Comprehensive performance score: XX%, grade: good, the performance of the vehicle computer automation test equipment is good, and some scenarios can be further optimized";

[0162] When 70% ≤ S < 80%, the report output is: "Comprehensive performance score: XX%, level: average, there are multiple medium-risk or some high-risk scenarios, and gradual optimization is recommended";

[0163] When S<70%, the report outputs: "Comprehensive performance score: XX%, grade: poor, there are multiple high-risk scenarios, the system performance has obvious deficiencies, and comprehensive optimization is required as soon as possible."

[0164] For example, the output structure of the assessment report is shown in the following table:

[0165] Table 2. Evaluation report structure

[0166]

[0167]

[0168] In an optional embodiment, the evaluation results are generated as an electronic report in PDF, HTML or JSON format, which is automatically pushed to the user interface for display or stored in a local / cloud database for subsequent review and analysis.

[0169] The aforementioned data preprocessing method for collected data can be implemented using existing technologies and will not be described in detail in this embodiment.

[0170] In application of the above embodiments, other aspects disclosed in the embodiments of the present invention further provide a performance evaluation system for vehicle-mounted automated testing equipment, including one or more processors and a memory.

[0171] A memory is used to store instructions that can be operated on, and when these instructions are executed by the one or more processors, they cause the one or more processors to perform operations, which include the processes of the performance evaluation method of the in-vehicle automation test device of the foregoing embodiments, in particular Figure 1 the processes of the method shown.

[0172] Some other aspects disclosed in the embodiments of the present invention also propose a computer-readable medium storing software, and the software includes instructions executable by one or more computers, and these instructions, through such execution, cause the one or more computers to perform operations, which include the processes of the performance evaluation method of the in-vehicle automation test device of the foregoing embodiments, in particular Figure 1 the processes of the method shown.

[0173] It should be recognized that the embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory.

[0174] The method can be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with the computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner.

[0175] Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program can be implemented in assembly or machine language.

[0176] In any case, the language can be a compiled or interpreted language.

[0177] In addition, for this purpose the program is capable of running on a programmed application-specific integrated circuit.

[0178] The processes described herein (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs or one or more applications) commonly executed on one or more processors, by hardware, or a combination thereof. The computer program includes multiple instructions executable by one or more processors.

[0179] Furthermore, the method can be implemented in any type of computing platform operably connected, including but not limited to personal computers, minicomputers, mainframes, workstations, network or distributed computing environments, separate or integrated computer platforms, or communicating with charged particle tools or other imaging devices.

[0180] Aspects of the present invention may be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform such as a hard disk, an optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer and, when read by the storage medium or device, can be used to configure and operate the computer to perform the processes described herein.

[0181] In addition, the machine-readable code, or portions thereof, may be transmitted via a wired or wireless network.

[0182] When such media includes instructions or programs that implement the above-described steps in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media.

[0183] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention may be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A performance evaluation method for a vehicle-mounted computer automated test device, characterized in that Including: Collect the historical test data of the in-vehicle infotainment system automation test equipment within a preset test cycle, construct an original data sample, and perform data preprocessing on it to form a standardized data sequence; Dynamically group the standardized data sequence according to the test scenario type, with each group corresponding to a test scenario, and sort the data in each group in descending order according to the preset performance indicators; Based on the data sorted in descending order, calculate the performance guarantee rate under each test scenario respectively, where the performance guarantee rate is defined as the probability that the in-vehicle infotainment system automation test equipment reaches the preset performance threshold under a specific test scenario; Associate the performance indicators and guarantee rate thresholds under each test scenario, analyze them, and output the performance evaluation results of the in-vehicle infotainment system automation test equipment.

2. The performance evaluation method of the in-vehicle infotainment system automation test equipment according to claim 1, characterized in that The historical test data includes test response time, test success rate, error type code, and system resource occupancy rate; The preset test cycle includes: When the in-vehicle software version update cycle is less than or equal to 30 days, the test cycle is set to 7 days; When the in-vehicle software version update cycle is greater than 30 days and less than 90 days, the test cycle is set to 14 days; When the in-vehicle software version update cycle is greater than or equal to 90 days, the test cycle is set to 30 days.

3. The performance evaluation method of the in-vehicle machine automation test equipment according to claim 1 or 2, characterized in that, Perform data preprocessing on the historical test data to form a standardized data sequence, including: Interpolate and complete the missing values in the collected historical test data; Then perform normalization processing on the completed data; According to the requirements of different test scenarios, use the sliding window method to smooth the normalized data to generate a standardized data sequence.

4. The performance evaluation method of the in-vehicle infotainment system automation test equipment according to claim 3, wherein Dynamically group the standardized data sequence according to the test scenario type, and sort the data in each group in descending order according to the preset performance indicators, including: Classify each group of standardized data according to the preset test scenario type identifier, and perform dynamic grouping according to the identification label corresponding to the scenario; Sort the data in each group from high to low according to the value of the preset performance indicator to obtain a data sequence sorted in descending order.

5. The performance evaluation method of the in-vehicle infotainment system automation test equipment according to claim 4, characterized in that The test scenario types include in-vehicle entertainment system stress test, navigation system multi-path planning test, voice interaction concurrency test, and vehicle networking security verification test; The preset performance indicators include system response time, peak CPU occupancy rate, memory leakage rate, task completion success rate, and abnormal interruption frequency.

6. The performance evaluation method of the in-vehicle machine automation test equipment according to claim 4, characterized in that Based on the data sorted in descending order, calculate the performance guarantee rate under each test scenario respectively, including: For each test scenario group, count the number n of data that meet the performance metric value not lower than the preset performance threshold in the data sorted in descending order within the group p and the total amount n of data within the group t ; Calculate the performance guarantee rate R using the following formula: where R is the performance guarantee rate for the corresponding test scenario, n p is the data sample size meeting the performance threshold requirement, n t is the total sample size for this test scenario; Calculate and record the performance guarantee rates under multiple test scenarios respectively.

7. The performance evaluation method of the in-vehicle infotainment system automation test equipment according to claim 1, characterized in that The associating the performance indicators and guarantee rate thresholds under each test scenario and analyzing them includes: For each test scenario, establish an association relationship table between the performance indicators and the corresponding performance guarantee rates respectively; Preset the performance guarantee rate thresholds for each test scenario; Use statistical analysis methods to identify the performance indicators that do not reach the performance guarantee rate thresholds in some test scenarios and generate risk warnings; Generate a comprehensive performance indicator matrix based on the performance indicators, performance guarantee rates, and thresholds under each test scenario; According to the comprehensive performance indicator matrix, combined with the importance weights of the test scenarios, calculate the comprehensive performance score S, and the score adopts a weighted calculation method: where I is the total number of test scenarios, and R i is the performance guarantee rate of the i-th test scenario, and w i is the importance weight of the i-th scenario; Define the pressure test weight of the in-vehicle entertainment system w1 = 0.25, the multi-path planning test weight of the navigation system w2 = 0.25, the voice interaction concurrency test weight w3 = 0.30, and the vehicle networking security verification test weight w4 = 0.

20.

8. The performance evaluation method of the in-vehicle infotainment system automation test equipment according to claim 7, characterized in that, Output the performance evaluation results of the in-vehicle automation test equipment, including: Generate a charted performance evaluation report based on the comprehensive performance score S and the risk warnings in each test scenario. The evaluation report includes the performance guarantee rate, the compliance of performance indicators, potential performance risks, and the comprehensive score in each test scenario. According to the calculated interval value of the comprehensive performance score S, classify the equipment performance into several levels, where: When S ≥ 90%, it is judged as excellent, that is, the equipment performance is excellent. When 80% ≤ S < 90%, it is judged as good, that is, the equipment performance is good. When 70% ≤ S < 80%, it is judged as average, that is, the equipment performance is medium. When S < 70%, it is judged as poor, that is, there is a large room for optimization in the equipment performance. For each test scenario, when the performance guarantee rate R in this scenario i is less than the performance guarantee rate threshold T preset for the scenario i then this scenario is marked as a performance risk, where: If R i <T i and R i ≥T i - 10%, generate a medium - risk warning; If R i <T i -10%, generate a high-risk prompt; If R i ≥ T i , then risk-free or compliant is generated.

9. A performance evaluation system for a vehicle-mounted computer automated test device, characterized in that, Including: One or more processors; A memory that stores executable instructions. When the instructions are executed by the one or more processors, the one or more processors perform operations, and the operations include the process of a performance evaluation method of an in-vehicle automation test equipment as described in any one of claims 1 to 8.

10. A computer-readable medium storing software, characterized in that: The software includes instructions that can be executed by one or more computers. When the instructions are executed in this way, the one or more computers perform operations, and the operations include the process of a performance evaluation method of an in-vehicle automation test equipment as described in any one of claims 1 to 8.

Images