Application performance prediction methods, devices, equipment, storage media, and program products

Abstract

This disclosure provides a method, apparatus, device, storage medium, and program product for predicting the performance of an application, relating to the application of artificial intelligence technology in the vehicle field. The method includes: acquiring historical driving data of a historical version of a target application that has been enabled; constructing a scenario test set based on the historical driving data; using the target application for active safety functions of the vehicle; running the test version of the target application in a simulation environment based on the scenario test set to obtain test results; and inputting the test results into a preset performance prediction model to determine the performance prediction indicators of the test version under actual operating conditions. This method can acquire historical driving data to construct a scenario test set, test the test version in a simulation environment based on the scenario test set, and then input the test results into a preset performance prediction model, thereby efficiently and accurately predicting the performance of the test version in actual road environments, providing reliable data support for version optimization and deployment decisions.

Description

Technical Field

[0001] This disclosure relates to the application of artificial intelligence technology in the vehicle field, and in particular to a method, apparatus, device, storage medium, and program product for predicting the performance of such applications. Background Technology

[0002] With the development of computer technology and vehicle intelligence technology, active safety functions have become the core of reducing accident risks and ensuring driving safety. The performance of active safety functions is directly related to the safety of users' lives and property. Therefore, performance evaluation of application versions is crucial during the iterative development of related applications for active safety functions.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide an application performance prediction method, apparatus, device, storage medium, and program product.

[0005] According to a first aspect of the present disclosure, a method for predicting the performance of an application is provided, comprising: acquiring historical driving data of a historical version of a target application that has been enabled; constructing a scenario test set based on the historical driving data; the target application being used for active safety functions of a vehicle; running a version of the target application to be tested in a simulation environment based on the scenario test set to obtain test results; and inputting the test results into a preset performance prediction model to determine the performance prediction index of the version to be tested in an actual operating environment.

[0006] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: This disclosure allows for the acquisition of historical driving data to construct scenario test sets, and the testing of the version under test in a simulation environment based on these test sets. The test results are then input into a pre-defined performance prediction model, thereby efficiently and accurately predicting the performance of the version under test in real-world road environments. This effectively overcomes the problems of high cost and limited scenario coverage associated with traditional real-vehicle road testing, and can efficiently evaluate the expected performance of a new version in a real-world environment, providing reliable data support for version optimization and deployment decisions.

[0007] In some implementations, the application performance prediction method further includes: optimizing the software version of the target application based on the performance prediction metrics; and / or periodically monitoring the target application based on the performance prediction metrics to identify abnormal fluctuations in the metrics and trigger alarms.

[0008] In the above implementation, software versions can be optimized or continuous monitoring can be performed based on predictive metrics, which can promptly detect performance anomalies and issue alerts, thereby improving system stability and user experience and reducing deployment risks.

[0009] In some implementations, constructing a scenario test set based on the historical driving data includes: extracting scenario data segments corresponding to preset events from the historical driving data; labeling the scenario data segments with trigger performance categories and / or operating condition categories; and filtering the scenario data segments based on the trigger performance categories and / or the operating condition categories to obtain a scenario test set.

[0010] In the above implementation, scene fragments can be extracted from historical data, labeled with trigger performance and / or operating condition categories, and then filtered to construct a highly targeted scene test set that covers a variety of key situations, effectively simulating real complex scenarios and improving the accuracy and comprehensiveness of performance prediction.

[0011] In some implementations, labeling the scene data fragments with trigger performance categories includes: labeling the scene data fragments with trigger performance categories based on the preset events; wherein, the preset events include at least one of the following: application-triggered events, vehicle collision events, application-triggered complaint events, and risk-free vehicle driving events; the trigger performance categories include at least one of the following: positive triggering, false triggering, missed triggering, and normal non-triggered events.

[0012] In the above implementation, the triggered performance category can be associated with a specific preset event, making the annotation of scene data fragments more accurate and reasonable, and providing a clear classification basis for subsequent screening of scene test sets and accurate performance evaluation.

[0013] In some implementations, the step of filtering the scene data segments based on the trigger performance category and / or the operating condition category to obtain a scene test set includes at least one of the following: including the portion of the scene data segments marked as missed triggers and false triggers that exceeds a certain proportion threshold in the scene test set; performing stratified sampling based on the distribution of operating condition categories for scene data segments marked as positive triggers, and including the sampled scene data segments in the scene test set; and identifying low-frequency and / or abnormal operating condition combinations in the scene data segments through cluster analysis, and including the identified scene data segments in the scene test set.

[0014] In the above implementation, various methods can be used to filter scenario data segments based on trigger performance and operating condition category, thereby comprehensively covering different performance problem scenarios and special operating conditions, making the scenario test set more representative and challenging, and improving the reliability of performance prediction.

[0015] In some implementations, the scene data fragment is labeled with a trigger performance category based on the preset event, including at least one of the following: labeling the scene data fragment as positively triggered or falsely triggered based on the application trigger event and the assessment of scene risk; labeling the scene data fragment as missed trigger based on the vehicle collision event; labeling the scene data fragment as positively triggered or falsely triggered based on the application trigger complaint event and the complaint content; and labeling the scene data fragment as normal and not triggered based on the vehicle risk-free driving event.

[0016] In the above implementation, different methods can be used to label the trigger performance categories for different preset events, accurately reflecting the triggering status of applications in the scene data fragments, and providing accurate data classification support for subsequent analysis of performance changes and prediction indicators.

[0017] In some implementations, labeling the scene data segment with operating condition categories includes at least one of the following: identifying the driving environment category in the scene data segment; identifying the object category of traffic participants in the scene data segment; and identifying the behavioral intent of traffic participants in the scene data segment.

[0018] In the above implementation, the operating condition categories can be labeled from aspects such as driving environment, traffic participants and behavioral intentions, so as to comprehensively and meticulously describe the operating conditions of the scene data segments, and provide rich operating condition information for screening scene test sets and accurately evaluating performance.

[0019] In some implementations, the performance prediction metric includes accident prediction density; wherein, the step of inputting the test results into a preset performance prediction model to determine the performance prediction metric of the version under test in the actual operating environment includes: determining the historical version response results of the historical version under the corresponding scenario data in the scenario test set, and obtaining the historical accident density data corresponding to the historical driving data; inputting the test results, the historical version response results, and the historical accident density data into the performance prediction model, and outputting the accident prediction density of the version under test in the actual operating environment.

[0020] In the above implementation, accident prediction density can be introduced as a core indicator. The performance prediction model integrates the response results of the old and new versions with historical accident data for processing, and outputs the actual accident situation that may be brought about after the deployment of the version under test, providing a quantitative indicator for evaluating the effectiveness of proactive safety functions in preventing accidents.

[0021] In some implementations, the step of inputting the test results, the historical version response results, and the historical incident density data into the performance prediction model and outputting the incident prediction density of the version under test in the actual operating environment includes: comparing and analyzing the test results and the historical version response results through the performance prediction model, identifying the missed trigger optimization scenarios and missed trigger rollback scenarios in the scenario test set, and constructing performance change features to characterize the performance differences between versions based on the identified scenarios; and generating the incident prediction density of the version under test in the actual operating environment by predicting based on the performance change features and the historical incident density data through the performance prediction model.

[0022] In the above implementation, the performance change characteristics can be extracted by comparing the performance of the old and new versions in key scenarios through the performance prediction model, and the accident prediction density that the version under test may bring can be accurately predicted by combining the historical accident density, so as to make the accident prediction more accurate and reasonable.

[0023] In some implementations, the performance prediction metric includes the predicted number of customer complaints; wherein, the step of inputting the test results into a preset performance prediction model to determine the performance prediction metric of the version under test in the actual operating environment includes: obtaining the historical customer complaint rate corresponding to the historical driving data; inputting the test results and the historical customer complaint rate into the performance prediction model, and outputting the predicted number of customer complaints of the version under test in the actual operating environment.

[0024] In the above implementation, the predicted customer complaint volume can be introduced as a core indicator. By combining historical complaint data and test results with a performance prediction model, the customer complaint situation that the version under test may cause in actual operation can be estimated in advance, providing a reference for optimizing the version and improving customer satisfaction.

[0025] In some implementations, the historical customer complaint rate includes: historical false trigger customer complaint rate and historical positive trigger customer complaint rate; wherein, inputting the test results and the historical customer complaint rate into the performance prediction model and outputting the predicted customer complaint volume of the version under test in the actual operating environment includes: identifying false trigger events and positive trigger events in the test results through the performance prediction model, and constructing functional trigger features to characterize the functional triggering mode based on the identified events; and generating the predicted customer complaint volume of the version under test in the actual operating environment by predicting based on the functional trigger features, the historical false trigger customer complaint rate and the historical positive trigger customer complaint rate through the performance prediction model.

[0026] In the above implementation, different triggering events can be identified through performance prediction models to construct functional triggering characteristics. Then, by combining the false triggering and positive triggering customer complaint rates, the number of user complaints that the version under test may bring can be accurately predicted, making the predicted customer complaint volume more in line with the actual situation and providing direction for version improvement.

[0027] According to a second aspect of the present disclosure, an application performance prediction apparatus is provided, comprising: a construction module, configured to acquire historical driving data of a historical version of a target application that has been enabled, and construct a scenario test set based on the historical driving data; the target application is used for active safety functions of a vehicle; a simulation module, configured to run a test version of the target application based on the scenario test set in a simulation environment to obtain test results; and a prediction module, configured to input the test results into a preset performance prediction model to determine the performance prediction index of the test version in an actual operating environment.

[0028] In some implementations, the application performance prediction device further includes an optimization module for: optimizing the software version of the target application based on the performance prediction metrics; and / or periodically monitoring the target application based on the performance prediction metrics to identify abnormal fluctuations in the metrics and trigger alarms.

[0029] In some implementations, the construction module constructs a scenario test set based on the historical driving data, including: extracting scenario data segments corresponding to preset events from the historical driving data; labeling the scenario data segments with trigger performance categories and operating condition categories; and filtering the scenario data segments based on the trigger performance categories and / or the operating condition categories to obtain the scenario test set.

[0030] In some implementations, the construction module labels the scene data fragments with trigger performance categories, including: labeling the scene data fragments with trigger performance categories based on the preset events; wherein, the preset events include at least one of the following: application-triggered events, vehicle collision events, application-triggered complaint events, and risk-free vehicle driving events; the trigger performance categories include at least one of the following: positive triggering, false triggering, missed triggering, and normal non-triggered events.

[0031] In some implementations, the construction module filters the scene data segments based on the trigger performance category and / or the operating condition category to obtain a scene test set, including at least one of the following: including the portion of the scene data segments marked as missed triggers and false triggers that exceeds a proportional threshold in the scene test set; for the scene data segments marked as positive triggers, perform stratified sampling according to the distribution of operating condition categories, and include the sampled scene data segments in the scene test set; identify low-frequency and / or abnormal operating condition combinations in the scene data segments through cluster analysis, and include the identified scene data segments in the scene test set.

[0032] In some implementations, the construction module labels the scene data fragments with trigger performance categories based on the preset events, including at least one of the following: labeling scene data fragments as positively triggered or falsely triggered based on application-triggered events and assessments of scene risks; labeling scene data fragments as missed-triggered based on vehicle collision events; labeling scene data fragments as positively triggered or falsely triggered based on application-triggered complaint events and complaint content; and labeling scene data fragments as normally not triggered based on risk-free vehicle driving events.

[0033] In some implementations, the construction module labels the scene data segments with operating condition categories, including at least one of the following: identifying the driving environment category in the scene data segments; identifying the object category of traffic participants in the scene data segments; and identifying the behavioral intentions of traffic participants in the scene data segments.

[0034] In some implementations, the performance prediction metric includes accident prediction density; wherein, the prediction module inputs the test results into a preset performance prediction model to determine the performance prediction metric of the version under test in the actual operating environment, including: determining the historical version response results of the historical version under the corresponding scenario data in the scenario test set, and obtaining the historical accident density data corresponding to the historical driving data; inputting the test results, the historical version response results, and the historical accident density data into the performance prediction model, and outputting the accident prediction density of the version under test in the actual operating environment.

[0035] In some implementations, the prediction module inputs the test results, the historical version response results, and the historical incident density data into the performance prediction model, and outputs the incident prediction density of the version under test in the actual operating environment. This includes: comparing and analyzing the test results and the historical version response results through the performance prediction model, identifying missed trigger optimization scenarios and missed trigger rollback scenarios in the scenario test set, and constructing performance change features to characterize the performance differences between versions based on the identified scenarios; and generating the incident prediction density of the version under test in the actual operating environment by predicting based on the performance change features and the historical incident density data through the performance prediction model.

[0036] In some implementations, the performance prediction metric includes the predicted number of customer complaints; wherein, the prediction module inputs the test results into a preset performance prediction model to determine the performance prediction metric of the version under test in the actual operating environment, including: obtaining the historical customer complaint rate corresponding to the historical driving data; inputting the test results and the historical customer complaint rate into the performance prediction model, and outputting the predicted number of customer complaints of the version under test in the actual operating environment.

[0037] In some implementations, the historical customer complaint rate includes: historical false trigger customer complaint rate and historical positive trigger customer complaint rate; wherein, the prediction module inputs the test results and the historical customer complaint rate into the performance prediction model, and outputs the predicted customer complaint volume of the version under test in the actual operating environment, including: identifying false trigger events and positive trigger events in the test results through the performance prediction model, and constructing function trigger features to characterize the function triggering mode based on the identified events; and generating the predicted customer complaint volume of the version under test in the actual operating environment by predicting based on the function trigger features, the historical false trigger customer complaint rate and the historical positive trigger customer complaint rate through the performance prediction model.

[0038] According to a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the performance prediction method of the application described above.

[0039] According to a fourth aspect of the present disclosure, a vehicle is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the performance prediction method of the application described above.

[0040] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the performance prediction method of the application described above.

[0041] According to a sixth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the performance prediction method for the application described above.

[0042] According to a seventh aspect of the present disclosure, a chip is provided, the chip including a processor and an interface, the interface being used to input or output signals, and the processor being configured to implement the performance prediction method of the application described above.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0045] Figure 1 This is a flowchart illustrating an application performance prediction method according to some embodiments of the present disclosure.

[0046] Figure 2 This is a flowchart illustrating the construction of a scenario test set in a performance prediction method for an application, according to some embodiments of the present disclosure.

[0047] Figure 3 This is a flowchart illustrating the determination of accident prediction density in an application performance prediction method according to some embodiments of the present disclosure.

[0048] Figure 4 This is a flowchart illustrating the determination of predicted customer complaints in a performance prediction method for an application, according to some embodiments of this disclosure.

[0049] Figure 5 This is a block diagram illustrating a performance prediction apparatus for an application according to some embodiments of the present disclosure.

[0050] Figure 6 This is a block diagram of a device for performance prediction of an application, illustrated according to some embodiments of the present disclosure.

[0051] Figure 7 This is a block diagram of a performance prediction chip 700 for an application according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0052] Some exemplary embodiments of this disclosure will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. Various changes, modifications, and equivalents of the methods, apparatus, and / or devices described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0053] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all content and steps, nor does it necessarily have to be executed in the described order or in the order of the step numbers. For example, some steps can be broken down, while others can be combined or partially combined, and multiple steps can have their order interchanged or be executed simultaneously. Therefore, the actual execution order may change depending on the actual situation.

[0054] The embodiments described below, which are examples of some of the embodiments of this disclosure, do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0055] In some embodiments of this disclosure, the acquisition of data and information, as well as the collection, updating, analysis, processing, use, transmission, and storage of related user personal information, may comply with the laws and regulations of the country where the location is situated.

[0056] In some embodiments of this disclosure, data, information, etc., may be obtained after obtaining the user's consent.

[0057] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0058] Figure 1 This is a flowchart illustrating an application performance prediction method according to some embodiments of the present disclosure, such as... Figure 1 As shown, the performance prediction method can be applied to electronic devices, including but not limited to terminal devices such as smartphones, smart tablets, wearable devices, desktop computers, laptops, and smart speakers. It can also include server-side components such as local servers and cloud servers, which can be deployed on a single computer or a cluster of multiple computers. The performance prediction method can include the following steps.

[0059] In step S110, historical driving data of a historical version of the target application that has been enabled is obtained, and a scenario test set is constructed based on the historical driving data; the target application is used for the vehicle's active safety functions.

[0060] In this embodiment of the disclosure, the target application may be a specific software application related to the vehicle's active safety function being evaluated. The active safety function may be a feature of the vehicle safety system that can prevent accidents through real-time monitoring and intervention. Target applications may include, for example, AEB (Autonomous Emergency Braking) or FCW (Forward Collision Warning).

[0061] Historical driving data can be actual driving data collected during the operation of previous versions of the target application. This data may include vehicle status information, vehicle driving information, environmental perception information, and usage logs of the target application. Driving data from the actual operation of previous versions of the target application can be collected, and a scenario test set can be built based on this data to evaluate the performance of the target application. This test set may include data that simulates real-world driving scenarios.

[0062] In an exemplary embodiment, historical driving data may include multi-source sensor information (such as camera, radar, lidar), vehicle CAN bus information (such as vehicle speed, steering, braking), as well as historical AEB software trigger records and internal decision data.

[0063] In step S120, the test version of the target application is run in a simulation environment based on the scenario test set to obtain test results.

[0064] In this embodiment of the disclosure, a version of the target application can be run in a virtual simulation environment using a constructed scenario test set to simulate actual driving conditions and record the test results output.

[0065] In an exemplary embodiment, the historical driving data files in the scenario test set can be accurately replayed through a pre-set simulation playback module, thereby inputting the original sensor data, vehicle CAN information, etc. recorded in the historical driving data files into the simulation environment in a time sequence.

[0066] The simulation environment can also be configured with a version of the target application to be tested, such as a new version of the AEB software module to be evaluated. The new version of the AEB software module can receive sensor data and vehicle information from the simulation playback module, and execute the perception, prediction, and decision-making logic of the new version of the AEB software module, and output the decision results of the new version of the AEB software. The decision results may include, for example, the AEB trigger signal, the suggested braking intensity or pressure, and the internal risk assessment value.

[0067] In step S130, the test results are input into a preset performance prediction model to determine the performance prediction index of the version under test in the actual operating environment.

[0068] In this embodiment of the disclosure, the performance prediction model can be a preset machine learning model, artificial intelligence model, mathematical model or algorithm, which can predict the performance of the application in a real environment based on the test results.

[0069] In an exemplary embodiment, the performance prediction model can be a pre-trained machine learning model, which can be constructed and trained based on the correlation between historical data and simulation results. Simulation test results can be input into this pre-trained performance prediction model for processing, outputting quantitative metrics to obtain the expected performance of the version under test in a real-world environment.

[0070] As can be seen from the above steps, the performance prediction method provided in this disclosure can acquire historical driving data to construct a scenario test set, test the version under test in a simulation environment based on the scenario test set, and then input the test results into a preset performance prediction model, thereby efficiently and accurately predicting the performance of the version under test in a real road environment. This effectively overcomes the problems of high cost and limited scenario coverage of traditional real-vehicle road testing, and can efficiently evaluate the expected performance of a new version in a real environment, providing reliable data support for version optimization and deployment decisions.

[0071] In some embodiments of this disclosure, the application performance prediction method further includes: optimizing the software version of the target application based on the performance prediction metrics; and / or periodically monitoring the target application based on the performance prediction metrics to identify abnormal fluctuations in the metrics and trigger alarms.

[0072] In this embodiment, the quantitative indicators output by the performance prediction model can be analyzed to identify performance shortcomings or areas for improvement in the target application's tested version. Targeted improvements can then be made to the logic, algorithms, parameters, or code of the tested version to enhance its expected performance in a real-world environment. For example, if the performance prediction indicators show an increase in the number of incidents in a certain scenario (such as a low-light environment), the relevant identification algorithm can be optimized for that scenario to reduce the number of incidents.

[0073] In this embodiment of the disclosure, the performance data of the deployed application under test can also be continuously tracked and compared in actual operation according to performance prediction indicators at certain time periods. By periodically collecting measured data and calculating key indicators, when the actual indicators deviate from the expected range of the performance prediction indicators or show abnormal fluctuation trends, an alarm mechanism can be automatically triggered to remind relevant personnel to pay attention, analyze anomalies in a timely manner, and eliminate potential risks.

[0074] Through the embodiments disclosed herein, software versions can be optimized or continuous monitoring can be performed based on predictive metrics, performance anomalies can be detected and alerts can be issued in a timely manner, system stability and user experience can be improved, and the risk of going live can be reduced.

[0075] Figure 2 This is a flowchart illustrating the construction of a scenario test set in a performance prediction method for an application, according to some embodiments of the present disclosure.

[0076] like Figure 2 As shown, in some embodiments of this disclosure, the process of constructing a scenario test set based on the historical driving data may include the following steps.

[0077] Step S210: Extract scene data segments corresponding to preset events from the historical driving data.

[0078] In this embodiment of the disclosure, key event data can be located in historical driving data by pre-defined event definitions, and data blocks containing the time sequence information before and after the event can be extracted to form a basic test scenario.

[0079] In an exemplary embodiment, the preset event may be a predefined specific driving situation, dangerous condition, or feedback information related to the vehicle's active safety functions, which serves as the trigger condition for data extraction.

[0080] Step S220: Label the scene data fragment with trigger performance category and operating condition category.

[0081] In this embodiment of the disclosure, structured tags can be added to each scene data fragment. Among them, the trigger performance category can be used to identify the correctness of the behavior of active safety functions in the scene, such as true positive, false negative, false positive, etc.; the operating condition category can be used to describe the environment and vehicle operating conditions in which the scene occurs, such as weather, lighting, traffic participant type, road type, traffic density, etc.

[0082] Step S230: Filter the scene data segments based on the trigger performance category and / or the operating condition category to obtain a scene test set.

[0083] In this embodiment of the disclosure, specific categories of scene data fragments can be selected according to the labeled data and testing requirements to construct a representative or targeted scene test set for subsequent simulation evaluation.

[0084] For example, you can select all failure scenarios where "the function should have been triggered but was not" for focused testing, or select scenarios under different weather and lighting conditions in a certain proportion to ensure that the test set can fully cover various key operating conditions and performance boundary situations, and avoid test bias.

[0085] Through the embodiments of this disclosure, scene fragments can be extracted from historical data, labeled with trigger performance and operating condition categories, and then filtered to construct a highly targeted scene test set that covers a variety of key situations, effectively simulating real complex scenarios and improving the accuracy and comprehensiveness of performance prediction.

[0086] In some embodiments of this disclosure, labeling the scene data fragments with trigger performance categories includes: labeling the scene data fragments with trigger performance categories based on the preset events; wherein, the preset events include at least one of the following: application-triggered events, vehicle collision events, application-triggered complaint events, and risk-free vehicle driving events; the trigger performance categories include at least one of the following: positive triggering, false triggering, missed triggering, and normal non-triggered events.

[0087] In this embodiment of the disclosure, the labeling of trigger performance categories can be achieved by comparing the nature of the event with the expected behavior of the function to complete the classification. Specifically, predefined preset events can be used as one of the judgment criteria to classify the extracted scene data fragments and determine their respective trigger performance categories.

[0088] Among them, application-triggered events can be event records where the vehicle's active safety function (i.e., the target application) is activated, issues a warning, or performs intervention during actual historical operation. Vehicle collision events can be event records where the vehicle actually collided with other vehicles, pedestrians, or other objects during historical driving. Complaint events regarding application triggers can be feedback events recorded by the driver or device after the function is triggered, indicating objections to the trigger or that the driver considers it unnecessary. Risk-free driving events can be events or scenarios where the vehicle was in a normal, safe driving state during historical driving, did not encounter any obvious risks, and did not require functional intervention.

[0089] In this context, a positive trigger occurs when a real risk exists or the triggering conditions are met, and the active safety function is correctly activated; this is the expected correct behavior of the function and can also be called a true positive (TP). A false trigger occurs when a real risk does not exist or the triggering conditions are not met, and the active safety function is incorrectly activated; this can also be called a false positive (FP). A missed trigger occurs when a real risk exists or the triggering conditions are met, and the active safety function fails to activate; this can also be called a false negative (FN). A normal non-trigger occurs when a real risk does not exist or the triggering conditions are not met, and the active safety function correctly remains inactive; this is the expected correct behavior of the function and can also be called a true negative (TN).

[0090] In an exemplary embodiment, scene data fragments related to application-triggered events, vehicle collision events, and application-triggered complaint events may be collected during the vehicle's use of a historical version of the target application. Scene data fragments related to risk-free driving events may be generated when the vehicle is driving on a specific risk-free road (such as a dedicated test road), where historical versions of the target application may or may not be enabled.

[0091] Through the embodiments of this disclosure, the triggered performance category can be associated with a specific preset event, making the annotation of scene data fragments more accurate and reasonable, and providing a clear classification basis for subsequent screening of scene test sets and accurate performance evaluation.

[0092] In some embodiments of this disclosure, the step of filtering the scene data segments based on the trigger performance category and / or the operating condition category to obtain a scene test set includes at least one of the following: including the portion of the scene data segments marked as missed triggers and false triggers that exceeds a proportional threshold in the scene test set; performing stratified sampling based on the distribution of operating condition categories for scene data segments marked as positive triggers, and including the sampled scene data segments in the scene test set; identifying low-frequency and / or abnormal operating condition combinations in the scene data segments through cluster analysis, and including the identified scene data segments in the scene test set.

[0093] In this embodiment of the disclosure, failure scenarios can be selectively screened. Specifically, all scenario data segments marked as missed triggers or false triggers can be included in the scenario test set, or other strategies can be used to select scenario data segments exceeding a certain amount for inclusion. For example, when the data volume is extremely large, data segments can be included based on the severity of the problem caused, such as selecting the top 80% of the most severe segments from all missed trigger scenarios.

[0094] The system can perform balanced filtering on the correctly triggered scenario data to ensure that the test set reflects the distribution of various operating conditions in the real world (such as different weather conditions and road types). All correctly triggered scenarios can be stratified according to their operating condition labels (such as "rainy day - highway", "night - city road"), and samples can be taken from each stratum according to its proportion in historical data to construct a statistically representative test set for evaluating the overall performance and generalization ability of the function.

[0095] It can identify and filter long-tail scenarios and edge cases. Among them, cluster analysis is an unsupervised machine learning method that can automatically discover the inherent groups in the data based on the multidimensional features of scene data fragments (such as vehicle speed, relative distance, target object type, etc.), and thus find long-tail scenarios and edge cases with low frequency of occurrence or abnormal combination of working conditions (such as "stationary motorcycles in a construction area in dense fog"), so as to avoid these scenarios being ignored in rule-based or proportion-based screening.

[0096] Through the embodiments of this disclosure, scenario data segments can be filtered based on trigger performance and operating condition category in various ways, thereby comprehensively covering different performance problem scenarios and special operating conditions, making the scenario test set more representative and challenging, and improving the reliability of performance prediction.

[0097] In some embodiments of this disclosure, the scene data fragment is labeled with a trigger performance category based on the preset event, including at least one of the following: the scene data fragment is labeled as positively triggered or falsely triggered based on the application trigger event and the assessment of scene risk; the scene data fragment is labeled as missed trigger based on the vehicle collision event; the scene data fragment is labeled as positively triggered or falsely triggered based on the complaint event triggered by the application and the complaint content; the scene data fragment is labeled as normal and not triggered based on the vehicle's risk-free driving event.

[0098] In this embodiment of the disclosure, for data segments where a function has been identified as triggered, an objective risk independent of the function itself can be introduced to assess whether a real risk exists in the scenario. If the assessment confirms the existence of a risk, it can be marked as a positive trigger; if the assessment confirms no risk, it can be marked as a false trigger.

[0099] For scene data fragments where actual physical collisions have occurred, since the collision event itself can be regarded as clear evidence that the active safety function has not worked effectively, this type of data can be directly classified as a missed trigger.

[0100] For data segments containing complaints, analysis can be based on complaints from users or the system (e.g., a driver's perception that the function suddenly brakes when unnecessary). Furthermore, the objective risks of the corresponding scenario can be considered to qualitatively characterize the function's triggering behavior. For example, if a complaint claims "no risk" but an objective risk actually exists, it can be labeled "positive trigger"; if a complaint claims "risk exists but the user experience is poor" and an objective risk actually exists, it can be labeled "positive trigger"; if a complaint claims "no risk" and no objective risk actually exists, it can be labeled "false trigger".

[0101] For pre-defined or screened normal driving scenarios that have been assessed and confirmed to have no risk, since active safety functions are not activated in these scenarios and meet expectations, they can be uniformly marked as "normal and not triggered." The scenario data marked as "normal and not triggered" can be used to verify the silent and non-intrusive nature of active safety functions under safe conditions.

[0102] Through the embodiments of this disclosure, different methods can be used to label the trigger performance categories for different preset events, accurately reflecting the triggering status of applications in scene data segments, and providing accurate data classification support for subsequent analysis of performance changes and prediction indicators.

[0103] In some embodiments of this disclosure, labeling the scene data segments with operating condition categories includes at least one of the following: identifying the driving environment category in the scene data segments; identifying the object category of traffic participants in the scene data segments; and identifying the behavioral intentions of traffic participants in the scene data segments.

[0104] In this embodiment of the disclosure, the static and objective conditions of the scene can be classified. For example, by analyzing sensor information (such as camera and weather sensor) and map data in data segments, objective environmental attributes that do not change rapidly over time, such as weather, lighting, and road type, can be extracted and categorized.

[0105] It can perform type recognition on dynamic traffic entities appearing in a scene. For example, it can classify other traffic participants around the vehicle based on perception data (such as vision, radar point cloud) to determine the category of traffic participants, such as cars, trucks, pedestrians, cyclists, etc.

[0106] It can also infer the purpose or future trend of traffic participants' dynamic behavior in a scene. For example, by analyzing the historical trajectory and movement status (such as turn signal) of traffic participants, it can be determined whether they are performing or about to perform a maneuver, such as a vehicle next to them intending to change lanes, a pedestrian intending to run a red light, or a vehicle in front braking suddenly.

[0107] Through the embodiments of this disclosure, operating condition categories can be labeled from aspects such as driving environment, traffic participants and behavioral intentions, so as to comprehensively and meticulously describe the operating conditions of scene data segments, providing rich operating condition information for screening scene test sets and accurately evaluating performance.

[0108] In some embodiments of this disclosure, the performance prediction metric includes accident prediction density.

[0109] In this embodiment of the disclosure, the accident prediction density can be a predictive metric that represents the potential accident density after the target application's test version is deployed and launched. Specifically, the accident density can be the number of accidents occurring per unit mileage (e.g., per million kilometers).

[0110] Figure 3 This is a flowchart illustrating the determination of accident prediction density in an application performance prediction method according to some embodiments of the present disclosure.

[0111] like Figure 3 As shown, in some embodiments of this disclosure, the process of inputting the test results into a preset performance prediction model to determine the performance prediction indicators of the version under test in the actual operating environment may include the following steps.

[0112] Step S310: Determine the historical version response result under the corresponding scenario data in the scenario test set, and obtain the historical accident density data corresponding to the historical driving data.

[0113] In this embodiment, historical version response results can be used as a benchmark for performance evaluation, reflecting the behavior of historical versions under specific driving conditions. Historical accident density can be a statistical indicator of historical driving data, representing the number of collisions recorded per million kilometers of actual road driving by the target application's historical version—collisions that the function should have avoided or mitigated but failed to prevent. Historical accident density is an objective measure of real-world safety performance.

[0114] In some embodiments of this disclosure, the historical version response result is extracted from the historical driving data, or obtained by running the historical version in the simulation environment based on the scenario test set.

[0115] In this embodiment of the disclosure, baseline data can be obtained based on actual operation records. The historical driving data itself can contain the original output signals (such as warning signs and braking commands) of the historical version when it was running in a real scenario. By parsing the time-series data aligned with the scenario data fragments, the actual response result of the historical version in that scenario (e.g., triggering AEB) can be directly extracted.

[0116] Alternatively, benchmark data can be obtained through simulation. A pre-built scenario test set can be run in a simulation environment with historical versions of the target application, and the response results (such as whether a trigger occurred) can be recorded, thus obtaining the response results of historical versions in all scenarios of the test set.

[0117] Through the embodiments disclosed herein, the method for obtaining historical version response results can be clearly defined, ensuring the reliability and consistency of data sources, making the comparative analysis of different versions more accurate, and thereby improving the accuracy of performance prediction indicators such as accident prediction density.

[0118] Step S320: Input the test results, the historical version response results, and the historical incident density data into the performance prediction model, and output the incident prediction density of the version under test in the actual operating environment.

[0119] In this embodiment, the performance prediction model can be a pre-trained artificial intelligence model with the following functions: predicting the performance indicators (such as incident prediction density) of the target application version under test in a real-world operating environment based on various input data (such as test results of the version under test, response results of historical versions, historical incident density data, etc.). This model can establish corresponding mapping rules by learning and analyzing the relationships between a large amount of historical data and simulation test data, thereby enabling accurate prediction of the performance of new versions.

[0120] For example, a performance prediction model can analyze the degree of improvement in the response of the version under test relative to historical versions in simulation, and combine it with the historical accident density corresponding to the actual historical versions to predict the accident rate that the version under test may achieve after deployment, i.e., the accident prediction density.

[0121] Through the embodiments disclosed herein, incident prediction density can be introduced as a core indicator. By integrating the response results of new and old versions with historical incident data through a performance prediction model, the actual incident situation that may be brought about after the deployment of the version under test can be output, providing a quantitative indicator for evaluating the effectiveness of proactive safety functions in preventing incidents.

[0122] In some embodiments of this disclosure, the step of inputting the test results, the historical version response results, and the historical incident density data into the performance prediction model and outputting the incident prediction density of the version under test in the actual operating environment includes: comparing and analyzing the test results and the historical version response results through the performance prediction model, identifying the missed trigger optimization scenarios and missed trigger rollback scenarios in the scenario test set, and constructing performance change features to characterize the performance differences between versions based on the identified scenarios; and generating the incident prediction density of the version under test in the actual operating environment by predicting based on the performance change features and the historical incident density data through the performance prediction model.

[0123] In this embodiment of the disclosure, the performance prediction model can compare the test results of the version under test with the response results of historical versions in the same scenario test set. The identified missed trigger optimization scenario can be a scenario where the historical version response result is a missed trigger and the test result is a positive trigger. The identified missed trigger rollback scenario can be a scenario where the historical version response result is a positive trigger and the test result is a missed trigger.

[0124] Performance prediction models can extract key information from these two identified scenarios to construct performance change features, reflecting the performance differences between different versions. These performance change features can be a set of feature vectors or metrics built based on identified missed-trigger optimization and missed-trigger rollback scenarios, used to quantitatively describe the performance changes of the tested version relative to historical versions. For example, performance change features can be constructed based on the number and operating conditions of missed-trigger optimization and rollback scenarios.

[0125] The constructed performance change characteristics and known historical incident density data from previous versions can be used as input to drive the performance prediction model. A well-trained performance prediction model can establish a quantitative relationship between performance changes and safety outcomes, considering the impact of performance improvements or regressions in the tested version on the probability of incidents, and generating the incident prediction density of the tested version running in the real world.

[0126] For example, assuming the historical accident density is 3 times per million kilometers, the model analyzes and finds that the performance change characteristics of the version under test include information such as "the number of missed trigger optimization scenarios increases by x1, the scenario operating condition types include x2, and the number of missed trigger rollback scenarios is 0". Based on the performance change characteristics, predictions can be made, and it is expected that the risk of related types of accidents can be reduced. The final output is that the accident prediction density of the version under test is 1 or 0 times per million kilometers.

[0127] Through the embodiments of this disclosure, performance change features can be extracted by comparing the performance of new and old versions in key scenarios using a performance prediction model, and the accident prediction density that the version under test may bring can be accurately predicted by combining historical accident density, making accident prediction more accurate and reasonable.

[0128] In some embodiments of this disclosure, the performance prediction metrics include predicted customer complaints.

[0129] Predicted customer complaints can be the number of customer complaints that the version under test is expected to generate in a real operating environment, which can provide a reference for evaluating the performance and customer acceptance of the version under test.

[0130] Figure 4 This is a flowchart illustrating the determination of predicted customer complaints in a performance prediction method for an application, according to some embodiments of this disclosure.

[0131] like Figure 4 As shown in some embodiments of this disclosure, the process of inputting the test results into a preset performance prediction model to determine the performance prediction indicators of the version under test in the actual operating environment may include the following steps.

[0132] Step S410: Obtain the historical customer complaint rate corresponding to the historical driving data.

[0133] In this embodiment of the disclosure, the historical customer complaint rate can be a ratio indicator statistically derived from historical driving data and its associated feedback data. The historical customer complaint rate can represent the number of customer complaints caused by each triggering of the active safety function during the actual operation of the historical version. The historical customer complaint rate can comprehensively reflect the rationality of the triggering behavior and user experience of the historical version.

[0134] Step S420: Input the test results and the historical customer complaint rate into the performance prediction model, and output the predicted customer complaint volume of the version under test in the actual operating environment.

[0135] In this embodiment, the performance prediction model can be a pre-trained artificial intelligence model with the following functions: predicting the performance indicators (such as predicting the number of customer complaints) of the target application under test version in the actual operating environment based on the test results of the input version and historical customer complaint rates. This model can establish a mapping relationship between input and output through learning and analysis of a large amount of historical data, thereby enabling accurate prediction of the number of customer complaints for the new version.

[0136] For example, performance prediction models can analyze the performance trend of a version under test relative to a historical benchmark, establish a correlation between performance differences and the probability of user complaints, and thus estimate the number of complaints that the version under test may cause in future real-world use. Performance prediction models can also analyze the types of vehicles involved in false triggering based on the test results of the version under test, and combine this with the proportion of the corresponding vehicle type in the historical customer complaint rate, thereby estimating the number of complaints that the version under test may cause when used on different types of vehicles in future real-world use.

[0137] In this embodiment of the disclosure, the test results obtained from the simulation test of the version under test and the known historical customer complaint rate can be jointly input into the performance prediction model. The performance prediction model can analyze the changing trend of the performance of the version under test relative to the historical benchmark, establish a correlation mapping between performance differences and the probability of user complaints, and thus estimate the number of complaints that the version under test may cause in future real-world use.

[0138] For example, the test results may include data related to false triggers of the version under test (such as the number of false triggers or the rate of reduction of false triggers). Then the performance prediction model can make predictions based on the data related to false triggers and historical customer complaint rate data, and can output results such as "the number of customer complaints is z".

[0139] Through the embodiments disclosed herein, the predicted customer complaint volume can be introduced as a core indicator. By combining historical complaint data and test results with a performance prediction model, the potential customer complaints that the version under test may cause in actual operation can be estimated in advance, providing a reference for optimizing the version and improving customer satisfaction.

[0140] In some embodiments of this disclosure, the historical customer complaint rate includes: historical false trigger customer complaint rate and historical positive trigger customer complaint rate; wherein, inputting the test results and the historical customer complaint rate into the performance prediction model and outputting the predicted customer complaint volume of the version under test in the actual operating environment includes: identifying false trigger events and positive trigger events in the test results through the performance prediction model, and constructing functional trigger features to characterize the functional triggering mode based on the identified events; and generating the predicted customer complaint volume of the version under test in the actual operating environment by predicting based on the functional trigger features, the historical false trigger customer complaint rate and the historical positive trigger customer complaint rate through the performance prediction model.

[0141] Both the proportion of complaints caused by mistakenly triggered behaviors (historical mistakenly triggered complaint rate) and the proportion of complaints caused by correctly triggered behaviors (historical correctly triggered complaint rate) in the historical customer complaint rate can have independent reference value. For example, even if the function responds correctly, if some pre-processes themselves (such as excessive braking force) have poor comfort, user complaints may still be received.

[0142] In this embodiment of the disclosure, the performance prediction model can statistically analyze false and positive triggering events in the test results, and construct a set of quantitative features that can describe the triggering behavior pattern of the version under test based on the statistical data of these two types of events.

[0143] For example, if X3 positive triggers and X4 false triggers occur in the simulation test, the performance prediction model can construct functional trigger features based on these number information and related operating conditions, including information such as "false trigger count data", "false trigger condition distribution", "positive trigger count data", and "positive trigger condition distribution".

[0144] Next, the performance prediction model can combine the function triggering characteristics of the version under test with two types of customer complaint rates accumulated in the actual operation of historical versions (including historical false trigger customer complaint rate and historical positive trigger customer complaint rate). Through the mapping relationship between the "trigger behavior pattern" and "user complaint tendency" established in the model, the model can predict the number of customer complaints that may be caused by false triggering and positive triggering issues, and sum them up to obtain the final predicted number of customer complaints.

[0145] Through the embodiments of this disclosure, different triggering events can be identified through a performance prediction model to construct functional triggering characteristics. Then, by combining the false triggering and positive triggering customer complaint rates, the number of user complaints that the version under test may bring can be accurately predicted, making the predicted customer complaint volume more in line with the actual situation and providing direction for version improvement.

[0146] In an exemplary embodiment, the performance prediction model can also be implemented based on preset statistical rules and calculation logic.

[0147] It is understandable that the method of calculating and predicting based on preset statistical rules provides a causal logic and computational principle for mapping simulation test results to actual performance indicators. The aforementioned machine learning model, based on this logical principle, can automatically learn and optimize this logical principle through data-driven training, discovering complex and non-linear correlations, thereby achieving predictive performance with stronger accuracy and generalization ability.

[0148] In an exemplary embodiment, the performance prediction model can determine the performance prediction indicators of the version under test in a real-world operating environment based on statistical data corresponding to test results and historical driving data. In this way, the performance prediction model can achieve quantitative prediction of the performance of new versions of vehicle active safety applications.

[0149] In an exemplary embodiment, the statistical data of historical driving data can be statistical information extracted from historical driving data, such as indicators used to evaluate vehicle driving safety performance and user experience.

[0150] In an exemplary embodiment, the statistical data corresponding to historical driving data may include historical accident density, which is used to determine the performance prediction index of accident prediction density.

[0151] In an exemplary embodiment, determining the performance prediction index of the version under test in the actual operating environment based on the statistical data corresponding to the test results and historical driving data may include: determining the historical version response result of the historical version under the corresponding scenario data in the scenario test set; comparing the test results with the historical version response result to determine the performance change data of the version under test; and determining the accident prediction density of the version under test in the actual operating environment based on the performance change data and the historical accident density.

[0152] In this embodiment of the disclosure, the test results of the version under test can be compared with the response results of historical versions under the same scenario test set, and the quantitative difference of key indicators (such as the rate of reduction of missed triggers) can be calculated. The quantitative difference is the performance change data.

[0153] In this embodiment of the disclosure, the performance change data can reflect the improvement ratio of functional safety capabilities. By applying the improvement ratio to the accident statistics of historical versions in the real world (historical accident density), the expected accident occurrence rate of the version under test under the same operating scale can be derived, that is, the accident prediction density.

[0154] Through the embodiments of this disclosure, the accident prediction density can be determined by using historical accident density and performance change data, which can intuitively reflect the impact of the version under test on the occurrence of accidents in actual operation, and provide quantitative indicators for evaluating the preventive effect of active safety functions on accidents.

[0155] In an exemplary embodiment, comparing the test results with the historical version response results to determine the performance change data of the version under test may include: identifying a missed trigger optimization scenario where the historical version response result is a missed trigger and the test result is a positive trigger, and determining the missed trigger optimization amount based on the number of missed trigger optimization scenarios; identifying a missed trigger rollback scenario where the historical version response result is a positive trigger and the test result is a missed trigger, and determining the missed trigger rollback amount based on the number of missed trigger rollback scenarios.

[0156] In this embodiment, scenarios can be compared one by one to identify those that were missed triggers in historical versions but have been corrected to triggers in the version under test. The total number of such scenarios or their proportion relative to all historical missed trigger scenarios in the scenario test set can be counted, and the resulting value is the missed trigger optimization amount. The missed trigger optimization amount can quantify the effect of the version under test on fixing security vulnerabilities (functions that should have been triggered but were not) existing in historical versions, thereby directly reflecting the positive improvement in security.

[0157] Furthermore, it allows for scenario-by-scenario comparisons to identify scenarios that were positively triggered in historical versions but degraded into missed triggers in the version under test. The total number of such scenarios or their proportion relative to all historical positively triggered scenarios in the scenario test set can be calculated; the resulting value is the missed trigger rollback amount. The missed trigger rollback amount quantifies the new security risks that the version under test may introduce compared to historical versions (the degradation of original functional capabilities), thus reflecting a negative change in security.

[0158] In an exemplary embodiment, if the historical version response result is the result measured in a simulation environment, then in all missed trigger scenarios in the scenario test set, a first number of missed triggers in the historical version and positive triggers in the version under test, and a second number of positive triggers in the historical version and missed triggers in the version under test, can be determined. The value of the first number minus the second number is used as the missed trigger optimization amount. Furthermore, in all positive trigger scenarios in the scenario test set, a third number of positive triggers in the historical version and missed triggers in the version under test, and a fourth number of missed triggers in the historical version and positive triggers in the version under test, can be determined. The value of the third number minus the fourth number is used as the missed trigger rollback amount.

[0159] Through the embodiments of this disclosure, the corresponding quantity can be determined by identifying missed trigger optimization and rollback scenarios, accurately quantifying the performance changes of the version under test in terms of missed triggers, and providing a data foundation for subsequent calculation of incident optimization quantity and incident prediction density.

[0160] In an exemplary embodiment, determining the incident prediction density of the version under test in the actual operating environment based on the performance change data and the historical incident density may include: determining the incident optimization amount based on the missed trigger optimization amount and the missed trigger rollback amount; determining the incident optimization density based on the incident optimization amount and the historical mileage corresponding to the scenario test set; and determining the incident prediction density based on the historical incident density and the incident optimization density.

[0161] In this embodiment of the disclosure, the amount of missed trigger optimization, which represents an improvement in security, can be subtracted from the amount of missed trigger rollback, which represents a decrease in security, to obtain a net value that reflects the overall improvement in missed trigger scenarios (i.e., the net reduction in the number of missed trigger scenarios), which is the incident optimization amount. The incident optimization amount can quantify the net gain of the tested version compared to the historical version in terms of its potential ability to avoid incidents caused by missed triggers.

[0162] Then, the number of accident optimizations can be divided by the total mileage of the historical driving data on which the test set for this scenario was built, to calculate the expected number of missed trigger events per unit mileage (e.g., per million kilometers), i.e., the accident optimization density. This metric can correlate test results with real-world operational scale, normalizing the absolute number of performance improvements to an improvement rate per unit mileage.

[0163] Next, the accident optimization density (i.e., the predicted accident avoidance rate per unit mile) can be subtracted from the historical accident density, and the difference is the accident prediction density. The accident prediction density can represent the expected accident rate per unit mile of the version under test under the same operating scale, thus realizing accident density prediction based on the predicted value and the historical baseline.

[0164] In an exemplary embodiment, if the historical version response result is the result measured in a simulation environment, the missed trigger optimization amount obtained by subtracting the second quantity from the first quantity can be divided by the driving mileage corresponding to the missed trigger scenario in the scenario test set to obtain the optimization density under the missed trigger scenario; the missed trigger rollback amount obtained by subtracting the fourth quantity from the third quantity can be divided by the driving mileage corresponding to the positive trigger scenario in the scenario test set to obtain the rollback density under the positive trigger scenario; and the accident optimization density can be obtained by subtracting the rollback density from the optimization density.

[0165] Through the embodiments of this disclosure, the accident optimization quantity can be determined based on the missing trigger related quantity, the optimization density can be determined by introducing the historical mileage corresponding to the scenario set, and then the accident prediction density can be obtained. The limited test results can be extrapolated to the expected effect after large-scale deployment, so that the prediction has practical application significance.

[0166] In an exemplary embodiment, the statistical data corresponding to the historical driving data may include the historical customer complaint rate, which is used to determine the performance prediction indicator of the predicted customer complaint volume.

[0167] In an exemplary embodiment, determining the performance prediction index of the version under test in the actual operating environment based on the statistical data corresponding to the test results and historical driving data may include: determining the test trigger quantity of the version under test based on the test results; and determining the predicted customer complaint quantity of the version under test in the actual operating environment based on the historical customer complaint rate and the test trigger quantity.

[0168] In this embodiment of the disclosure, the test results can be analyzed to calculate the number of times the function of the target application version under test is activated during the entire scenario test set operation, and this number is the test trigger quantity.

[0169] In an exemplary embodiment, the test trigger count may include the number of positive triggers and the number of false triggers.

[0170] In this embodiment of the disclosure, the proportion of customer complaints caused by a unit number of triggers can be regarded as stable. Then, by multiplying the historical customer complaint rate obtained from the actual operation of the historical version with the number of test triggers shown by the version under test in the test, the number of complaints that the version under test may cause in actual operation in the same mode can be estimated, that is, the predicted number of customer complaints.

[0171] Through the embodiments disclosed herein, the predicted customer complaint volume can be determined based on test results and historical customer complaint rates, and the potential customer complaints that the version under test may cause in actual operation can be estimated in advance, providing a reference for optimizing the version and improving customer satisfaction.

[0172] In an exemplary embodiment, the test trigger quantity may include: false trigger quantity and positive trigger quantity; wherein, determining the predicted customer complaint quantity of the version under test in the actual operating environment based on the historical customer complaint rate and the test trigger quantity may include: determining the predicted false trigger customer complaint quantity based on the false trigger quantity and the historical false trigger customer complaint rate; determining the predicted positive trigger customer complaint quantity based on the positive trigger quantity and the historical positive trigger customer complaint rate; and determining the predicted customer complaint quantity based on the predicted false trigger customer complaint quantity and the predicted positive trigger customer complaint quantity.

[0173] In this embodiment of the disclosure, the historical false trigger complaint rate is applied to the total number of false triggers generated by the version under test during testing (e.g., multiplying the two) to calculate the predicted number of false trigger-related complaints. Similarly, the historical positive trigger complaint rate is applied to the total number of positive triggers generated by the version under test during testing (e.g., multiplying the two) to calculate the predicted number of positive trigger-related complaints.

[0174] Summing the predicted complaint numbers from two different sources yields a comprehensive predictive metric for the total number of customer complaints triggered by feature-triggered behaviors—the predicted customer complaint volume. This predicted customer complaint volume provides concrete quantitative evidence for assessing the potential impact of the version under test on the overall user experience.

[0175] Through the embodiments of this disclosure, the predicted customer complaint volume can be determined based on the relevant quantities of false triggering and positive triggering, so as to more accurately and meticulously assess the impact of different triggering situations on customer complaints, making the predicted customer complaint volume more in line with the actual situation and providing direction for version improvement.

[0176] It should be noted that the above figures are merely illustrative representations of the processes included in methods according to some embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0177] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0178] Figure 5 This is a block diagram illustrating a performance prediction apparatus for an application according to some embodiments of the present disclosure. (Refer to...) Figure 5 The device includes: a construction module 501, a simulation module 502, a prediction module 503, and an optimization module 504.

[0179] The construction module 501 is used to acquire historical driving data of historical versions of the target application that have been enabled, and to construct a scenario test set based on the historical driving data; the target application is used for the active safety functions of the vehicle; the simulation module 502 is used to run the version under test of the target application in a simulation environment based on the scenario test set to obtain test results; the prediction module 503 is used to input the test results into a preset performance prediction model to determine the performance prediction index of the version under test in the actual operating environment.

[0180] In some embodiments of this disclosure, the optimization module 504 is used to: optimize the software version of the target application based on the performance prediction metrics; and / or to periodically monitor the target application based on the performance prediction metrics to identify abnormal fluctuations in the metrics and trigger alarms.

[0181] In some embodiments of this disclosure, the construction module 501 constructs a scenario test set based on the historical driving data, including: extracting scenario data segments corresponding to preset events from the historical driving data; labeling the scenario data segments with trigger performance categories and operating condition categories; and filtering the scenario data segments based on the trigger performance categories and / or the operating condition categories to obtain the scenario test set.

[0182] In some embodiments of this disclosure, the construction module 501 labels the scene data fragment with trigger performance categories, including: labeling the scene data fragment with trigger performance categories based on the preset events; wherein, the preset events include at least one of the following: application trigger events, vehicle collision events, application-triggered complaint events, and risk-free vehicle driving events; the trigger performance categories include at least one of the following: positive triggering, false triggering, missed triggering, and normal non-triggering.

[0183] In some embodiments of this disclosure, the construction module 501 filters the scene data segments based on the trigger performance category and / or the operating condition category to obtain a scene test set, including at least one of the following: including the portion of the scene data segments marked as missed triggers and false triggers that exceeds a proportion threshold in the scene test set; for the scene data segments marked as positive triggers, perform stratified sampling according to the distribution of operating condition categories, and include the sampled scene data segments in the scene test set; identify low-frequency and / or abnormal operating condition combinations in the scene data segments through cluster analysis, and include the identified scene data segments in the scene test set.

[0184] In some embodiments of this disclosure, the construction module 501 labels the scene data fragments with trigger performance categories based on the preset events, including at least one of the following: labeling scene data fragments as positively triggered or falsely triggered based on application-triggered events and assessments of scene risks; labeling scene data fragments as missed-triggered based on vehicle collision events; labeling scene data fragments as positively triggered or falsely triggered based on application-triggered complaint events and complaint content; and labeling scene data fragments as normally not triggered based on risk-free vehicle driving events.

[0185] In some embodiments of this disclosure, the construction module 501 labels the scene data segment with operating condition categories, including at least one of the following: identifying the driving environment category in the scene data segment; identifying the object category of traffic participants in the scene data segment; and identifying the behavioral intention of traffic participants in the scene data segment.

[0186] In some embodiments of this disclosure, the statistical data corresponding to the historical driving data includes historical accident density, and the performance prediction index includes accident prediction density; wherein, the prediction module 503 inputs the test results into a preset performance prediction model to determine the performance prediction index of the version under test in the actual operating environment, including: determining the historical version response result of the historical version under the corresponding scenario data in the scenario test set, and obtaining the historical accident density data corresponding to the historical driving data; inputting the test results, the historical version response result, and the historical accident density data into the performance prediction model, and outputting the accident prediction density of the version under test in the actual operating environment.

[0187] In some embodiments of this disclosure, the prediction module 503 inputs the test results, the historical version response results, and the historical incident density data into the performance prediction model, and outputs the incident prediction density of the version under test in the actual operating environment. This includes: comparing and analyzing the test results and the historical version response results through the performance prediction model, identifying the missed trigger optimization scenarios and missed trigger rollback scenarios in the scenario test set, and constructing performance change features to characterize the performance differences between versions based on the identified scenarios; and generating the incident prediction density of the version under test in the actual operating environment by predicting based on the performance change features and the historical incident density data through the performance prediction model.

[0188] In some embodiments of this disclosure, the statistical data corresponding to the historical driving data includes the historical customer complaint rate, and the performance prediction index includes the predicted customer complaint volume; wherein, the prediction module 503 inputs the test results into a preset performance prediction model to determine the performance prediction index of the version under test in the actual operating environment, including: obtaining the historical customer complaint rate corresponding to the historical driving data; inputting the test results and the historical customer complaint rate into the performance prediction model, and outputting the predicted customer complaint volume of the version under test in the actual operating environment.

[0189] In some embodiments of this disclosure, the historical customer complaint rate includes: historical false trigger customer complaint rate and historical positive trigger customer complaint rate; wherein, the prediction module 503 inputs the test results and the historical customer complaint rate into the performance prediction model, and outputs the predicted customer complaint volume of the version under test in the actual operating environment, including: identifying false trigger events and positive trigger events in the test results through the performance prediction model, and constructing functional trigger features to characterize the functional triggering mode based on the identified events; and generating the predicted customer complaint volume of the version under test in the actual operating environment by predicting based on the functional trigger features, the historical false trigger customer complaint rate and the historical positive trigger customer complaint rate through the performance prediction model.

[0190] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0191] Figure 6 This is a block diagram illustrating a device 600 for performance prediction of an application according to some embodiments of the present disclosure. For example, device 600 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0192] Reference Figure 6 The device 600 may include one or more of the following components: a processing component 602, a memory 604, a power component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0193] Processing component 602 typically controls the overall operation of device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0194] Memory 604 is configured to store various types of data to support the operation of device 600. Examples of this data include instructions for any application or method operating on device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0195] Power supply component 606 provides power to various components of device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 600.

[0196] Multimedia component 608 includes a screen that provides an output interface between the device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0197] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0198] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0199] Sensor assembly 614 includes one or more sensors for providing status assessments of various aspects of device 600. For example, sensor assembly 614 may detect the on / off state of device 600, the relative positioning of components such as the display and keypad of device 600, changes in the position of device 600 or a component of device 600, the presence or absence of user contact with device 600, the orientation or acceleration / deceleration of device 600, and temperature changes of device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0200] Communication component 616 is configured to facilitate wired or wireless communication between device 600 and other devices. Device 600 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 616 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0201] In some embodiments of this disclosure, device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0202] In some embodiments of this disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions that can be executed by a processor 620 of device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0203] Figure 7 This is a block diagram of a chip 700 for performance prediction of an application according to an exemplary embodiment of the present disclosure. Figure 7 As shown, the chip 700 may include a processor 701 and an interface 702. The interface 702 may be connected to one or more memories 703, and the interface 702 may be used by the memories 703 to receive or acquire data or instructions. Additionally, all or part of the memories 703 may be located outside the chip 700. The processor 701 is configured to execute processing steps of the performance prediction method stored in the above-described application. The specific method executed by the processor has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

[0204] A vehicle may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other type of vehicle. The vehicle may be a driver-assisted vehicle, a semi-driver-assisted vehicle, or a driver-free vehicle.

[0205] Vehicles can include various subsystems, such as infotainment systems, perception systems, decision control systems, drive systems, and computing platforms. A vehicle can also include more or fewer subsystems, and each subsystem can include multiple components. Furthermore, each subsystem and each component of the vehicle can be interconnected via wired or wireless means.

[0206] In some embodiments, an infotainment system may include a communication system, an entertainment system, and a navigation system, etc.

[0207] The perception system may include several types of sensors used to sense information about the environment surrounding the vehicle. For example, the perception system may include a global positioning system, an inertial measurement unit (IMU), lidar, millimeter-wave radar, ultrasonic radar, and camera devices.

[0208] The decision control system may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0209] A drive system may include components that provide powered motion to a vehicle. In one embodiment, a drive system may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0210] Some or all of the vehicle's functions are controlled by a computing platform. The computing platform may include at least one processor and memory, the processor being able to execute instructions stored in the memory.

[0211] The processor can be any conventional processor, such as a commercially available CPU. The processor can also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems-on-chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0212] Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0213] In addition to instructions, memory can also store data, such as road maps, route information, and vehicle position, direction, and speed. The data stored in memory can be used by the computing platform.

[0214] In this embodiment of the disclosure, the processor may execute instructions to complete all or part of the steps of the above-described application performance prediction method.

[0215] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform a performance prediction method for an application.

[0216] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0217] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for predicting the performance of an application, characterized in that, include: Obtain historical driving data for historical versions of the target application that have been enabled, and construct a scenario test set based on the historical driving data; The target application is for the active safety functions of the vehicle; The test version of the target application is run in a simulation environment based on the scenario test set to obtain test results; The test results are input into a preset performance prediction model to determine the performance prediction indicators of the version under test in the actual operating environment.

2. The method according to claim 1, characterized in that, The method further includes: Optimize the software version of the target application based on the performance prediction metrics; and / or, The target application is monitored regularly based on the performance prediction metrics to identify abnormal fluctuations in the metrics and trigger alarms.

3. The method according to claim 1, characterized in that, The construction of the scenario test set based on the historical driving data includes: Extract scene data fragments corresponding to preset events from the historical driving data; The scene data fragments are labeled with trigger performance categories and / or operating condition categories; The scenario data fragments are filtered based on the trigger performance category and / or the operating condition category to obtain a scenario test set.

4. The method according to claim 3, characterized in that, The scene data fragments are labeled with trigger performance categories, including: The scene data fragment is labeled with a trigger performance category based on the preset event; The preset events include at least one of the following: application-triggered events, vehicle collision events, application-triggered complaint events, and risk-free vehicle driving events; the trigger performance categories include at least one of the following: positive triggering, false triggering, missed triggering, and normal non-triggering.

5. The method according to claim 4, characterized in that, The scenario test set is obtained by filtering the scenario data fragments based on the trigger performance category and / or the operating condition category, including at least one of the following: The portion of scene data segments marked as missed triggers and false triggers that exceeds the proportional threshold will be included in the scene test set; For scene data segments marked as positive triggering, stratified sampling is performed based on the distribution of working condition categories, and the sampled scene data segments are included in the scene test set; Low-frequency and / or abnormal operating condition combinations in the scenario data segments are identified through cluster analysis, and the identified scenario data segments are included in the scenario test set.

6. The method according to claim 4, characterized in that, The scene data fragment is labeled with a trigger performance category based on the preset event, including at least one of the following: Based on application-triggered events and assessments of scenario risks, scenario data fragments are labeled as either positively triggered or falsely triggered. Based on vehicle collision events, scene data segments are marked as missed triggers; Based on the complaint events and content triggered by the application, scene data fragments are labeled as either positively triggered or falsely triggered; Based on the risk-free driving events of vehicles, scene data segments are marked as normal and not triggered.

7. The method according to claim 3, characterized in that, The scene data fragments are labeled with working condition categories, including at least one of the following: Identify the driving environment category in the scene data segment; Identify the object categories of traffic participants in the scene data fragment; Identify the behavioral intentions of traffic participants in the scene data segment.

8. The method according to claim 1, characterized in that, The performance prediction metrics include accident prediction density; The step of inputting the test results into a preset performance prediction model to determine the performance prediction indicators of the version under test in the actual operating environment includes: Determine the historical version response results under the corresponding scenario data in the scenario test set, and obtain the historical accident density data corresponding to the historical driving data; The test results, the historical version response results, and the historical incident density data are input into the performance prediction model, and the incident prediction density of the version under test in the actual operating environment is output.

9. The method according to claim 8, characterized in that, The step of inputting the test results, the historical version response results, and the historical incident density data into the performance prediction model, and outputting the incident prediction density of the version under test in the actual operating environment, includes: The performance prediction model is used to compare and analyze the test results with the historical version response results to identify the missed trigger optimization scenarios and missed trigger rollback scenarios in the scenario test set, and to construct performance change features to characterize the performance differences between versions based on the identified scenarios. The performance prediction model uses the performance change characteristics and historical accident density data to predict the accident prediction density of the version under test in the actual operating environment.

10. The method according to claim 1, characterized in that, The performance prediction metrics include the predicted number of customer complaints; The step of inputting the test results into a preset performance prediction model to determine the performance prediction indicators of the version under test in the actual operating environment includes: Obtain the historical customer complaint rate corresponding to the historical driving data; The test results and the historical customer complaint rate are input into the performance prediction model, which outputs the predicted customer complaint volume of the version under test in the actual operating environment.

11. The method according to claim 10, characterized in that, The historical customer complaint rate includes: historical false-trigger customer complaint rate and historical positive-trigger customer complaint rate; wherein, the step of inputting the test results and the historical customer complaint rate into the performance prediction model and outputting the predicted customer complaint volume of the version under test in the actual operating environment includes: The performance prediction model identifies false and positive trigger events in the test results, and constructs functional trigger features to characterize the functional triggering mode based on the identified events. The performance prediction model uses the function triggering characteristics, the historical false triggering complaint rate, and the historical positive triggering complaint rate to predict the number of complaints for the version under test in the actual operating environment.

12. A performance prediction device for an application, characterized in that, include: The module is used to obtain historical driving data of historical versions of the target application that have been enabled, and to build a scenario test set based on the historical driving data; The target application is for the active safety functions of the vehicle; The simulation module is used to run the test version of the target application in a simulation environment based on the scenario test set and obtain test results. The prediction module is used to input the test results into a preset performance prediction model to determine the performance prediction indicators of the version under test in the actual operating environment.

13. The apparatus according to claim 12, characterized in that, The device further includes an optimization module for: Optimize the software version of the target application based on the performance prediction metrics; and / or, The target application is monitored regularly based on the performance prediction metrics to identify abnormal fluctuations in the metrics and trigger alarms.

14. The apparatus according to claim 12, characterized in that, The construction module builds a scenario test set based on the historical driving data, including: Extract scene data fragments corresponding to preset events from the historical driving data; The scene data fragments are labeled with trigger performance categories and / or operating condition categories; The scenario data fragments are filtered based on the trigger performance category and / or the operating condition category to obtain a scenario test set.

15. The apparatus according to claim 12, characterized in that, The performance prediction metrics include accident prediction density; The prediction module inputs the test results into a preset performance prediction model to determine the performance prediction indicators of the version under test in the actual operating environment, including: Determine the historical version response results under the corresponding scenario data in the scenario test set, and obtain the historical accident density data corresponding to the historical driving data; The test results, the historical version response results, and the historical incident density data are input into the performance prediction model, and the incident prediction density of the version under test in the actual operating environment is output.

16. The apparatus according to claim 12, characterized in that, The performance prediction metrics include the predicted number of customer complaints; The prediction module inputs the test results into a preset performance prediction model to determine the performance prediction indicators of the version under test in the actual operating environment, including: Obtain the historical customer complaint rate corresponding to the historical driving data; The test results and the historical customer complaint rate are input into the performance prediction model, which outputs the predicted customer complaint volume of the version under test in the actual operating environment.

17. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the method according to any one of claims 1-11.

18. A vehicle, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the method according to any one of claims 1-11.

19. A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of a mobile terminal, enable the mobile terminal to perform the steps of the method according to any one of claims 1-11.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-11.

21. A chip, characterized in that, The chip includes a processor and an interface for inputting or outputting signals, the processor being configured to implement the steps of the method according to any one of claims 1-11.

Images