Automatic navigation assisted driving system evaluation method, device and storage medium

Through the combination of support vector machine and autoregressive model, the problem of insufficient quantitative and generalization capabilities of the evaluation method of automatic navigation assisted driving system in the existing technology is solved, and a comprehensive quantitative evaluation of system performance and user experience is achieved, and the evaluation is improved. accuracy and adaptability.

CN118227510BActive Publication Date: 2024-11-15CHINA AUTOMOTIVE TECH & RES CENT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410637183.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-22
Publication Date
2024-11-15
Estimated Expiration
2044-05-22

AI Technical Summary

Technical Problem

The existing evaluation methods of automatic navigation assisted driving systems rely on subjective experience and qualitative analysis, lack a quantitative evaluation framework, it is difficult to fully capture the multifaceted characteristics and user experience of system performance, and lack generalization capabilities.

Method used

By integrating the support vector machine model and autoregressive model, evaluation indicators are determined based on the autonomous driving navigation function and operation boundary, related data are collected for risk assessment, efficiency analysis and fault trend prediction, and combined with user feedback information, comprehensively evaluate the safety and efficiency of the system. and reliability, determine safety and performance levels.

Benefits of technology

Quantitative evaluation of the automatic navigation assisted driving system is realized, the scope of application and generalization of the evaluation method is enhanced, and the actual performance and user experience of the system can be accurately reflected, and the evaluation accuracy and adaptability are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118227510B_ABST
    Figure CN118227510B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of data processing technology, and discloses an automatic navigation assisted driving system evaluation method, device and storage medium. The method determines the associated data items of each evaluation index according to the automatic driving navigation function and operation boundary of the system to be evaluated, collects the associated data, and then classifies the operation data based on the support vector machine model to obtain the risk level, determines the safety evaluation value, and performs kernel density estimation on the path optimization rate and the time saved to determine the efficiency evaluation value, and predicts the future failure trend according to the autoregressive model, determines the reliability evaluation value in combination with the model prediction accuracy, and determines the user satisfaction evaluation value according to the user feedback information, and finally determines the safety level and performance level by comprehensively considering safety, efficiency, reliability and user satisfaction, thereby realizing the quantitative evaluation of the system, helping to judge whether the system needs to be optimized, realizing the comprehensive evaluation of the system, and improving the evaluation accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an automatic navigation assisted driving system evaluation method, device and storage medium. Background Art

[0002] The Navigate on Autopilot (NOA) system has broad development prospects and is an important market for autonomous driving technology in the commercial field. The typical application of this technology is to realize autonomous navigation and control of vehicles on roads such as highways, including automatic speed adjustment, automatic lane change and overtaking, automatic entry and exit ramps, etc. It can significantly improve the transportation efficiency and traffic capacity of roads, reduce the probability of traffic accidents, and provide drivers with a more comfortable driving experience.

[0003] The safety of NOA systems is also receiving more and more attention. In autonomous driving, vehicles need to interact with other vehicles and road environments in a complex manner, and the reliability and safety of the system must be ensured to prevent potential risks and accidents.

[0004] Current safety evaluation methods mainly rely on subjective experience and qualitative analysis, and lack a quantitative evaluation framework. Existing evaluation methods lack sufficient generalization capabilities when comprehensively evaluating the performance of NOA systems, and are difficult to adapt to different usage scenarios and operating conditions; at the same time, there are obvious deficiencies in processing multi-dimensional data and its dynamic changes, making it difficult to fully capture and analyze the various aspects of system performance; in addition, existing evaluation methods fail to fully reflect the actual performance of the system and user experience.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides an automatic navigation assisted driving system evaluation method, device and storage medium. By integrating multiple characteristics of system performance, a comprehensive evaluation of the NOA system is achieved, the application scope and generalization ability of the evaluation method are significantly enhanced, and accurate analysis of user satisfaction and efficiency indicators detected in real time is achieved, thereby improving the comprehensiveness and adaptability of the evaluation method, so that the evaluation results can accurately reflect the actual performance of the system and user experience.

[0007] An embodiment of the present invention provides an automatic navigation assisted driving system evaluation method, the method comprising:

[0008] Determine the associated data items of each evaluation indicator based on the autonomous driving navigation function and the operation boundary of the system to be evaluated, and collect the associated data of the system to be evaluated according to the associated data items;

[0009] Classifying all operation data in the associated data based on the trained support vector machine model to obtain a risk level of each operation data, and determining a safety evaluation value based on the risk level of each operation data;

[0010] Performing kernel density estimation on the path optimization rate and the time saving in the associated data to obtain a probability density function, and determining an efficiency evaluation value based on a peak value in the probability density function;

[0011] Predicting historical fault data in the associated data based on the trained autoregressive model to obtain future fault trends, and determining a reliability evaluation value based on the fault trends and the prediction accuracy of the autoregressive model;

[0012] Determine a satisfaction change trend based on the user feedback information in the associated data, and determine a user satisfaction evaluation value based on the satisfaction change trend;

[0013] A comprehensive evaluation result is determined based on the safety evaluation value, the efficiency evaluation value, the reliability evaluation value and the user satisfaction evaluation value, and the safety level and performance level of the system to be evaluated are determined according to the comprehensive evaluation result.

[0014] An embodiment of the present invention provides an electronic device, the electronic device comprising:

[0015] Processor and memory;

[0016] The processor is used to execute the steps of the automatic navigation assisted driving system evaluation method described in any embodiment by calling the program or instruction stored in the memory.

[0017] An embodiment of the present invention provides a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute the steps of the automatic navigation assisted driving system evaluation method described in any embodiment.

[0018] The embodiments of the present invention have the following technical effects:

[0019] First, according to the autonomous driving navigation function and operation boundary of the system to be evaluated, the associated data items of each evaluation index are determined, and the associated data are collected through the associated data items. Then, all the operation data are classified based on the trained support vector machine model to obtain the risk level of each operation data, determine the safety evaluation value, and perform kernel density estimation on the path optimization rate and time saving to obtain the probability density function. The efficiency evaluation value is determined in combination with the peak value, and the historical fault data is predicted according to the trained autoregressive model to obtain the future fault trend. The reliability evaluation value is determined in combination with the prediction accuracy of the autoregressive model, and the satisfaction change trend is determined according to the user feedback information to obtain the user satisfaction evaluation value. Finally, the comprehensive safety evaluation value, The efficiency evaluation value, reliability evaluation value and user satisfaction evaluation value determine the safety level and performance level of the system to be evaluated, realize the quantitative evaluation of the NOA system, and help judge whether the NOA system needs to be optimized. This method realizes the comprehensive evaluation of the NOA system by integrating multiple characteristics of system performance, significantly enhancing the scope of application and generalization ability of the evaluation method. Moreover, by accurately analyzing the user satisfaction and efficiency indicators detected in real time, the comprehensiveness and adaptability of the evaluation method are improved, so that the evaluation results can accurately reflect the actual performance of the system and user experience. In addition, this method uses support vector machines and autoregressive models to conduct in-depth analysis and prediction of key performance indicators, which can further improve the evaluation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a flow chart of an automatic navigation assisted driving system evaluation method provided by an embodiment of the present invention;

[0022] Figure 2 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0024] The automatic navigation assisted driving system evaluation method provided in the embodiment of the present invention is mainly applicable to evaluating the automatic navigation assisted driving system to determine the safety level and performance level of the system to be evaluated, thereby facilitating the judgment of whether the system needs to be optimized. The automatic navigation assisted driving system evaluation method provided in the embodiment of the present invention can be executed by an electronic device such as a computer or a vehicle controller.

[0025] Figure 1 is a flow chart of an automatic navigation assisted driving system evaluation method provided by an embodiment of the present invention. Figure 1 , the automatic navigation assisted driving system evaluation method specifically includes:

[0026] S110. Based on the autonomous driving navigation function and operation boundary of the system to be evaluated, determine the associated data items of each evaluation indicator, and collect the associated data of the system to be evaluated according to the associated data items.

[0027] The system to be evaluated may refer to the NOA system to be evaluated. Specifically, the autonomous driving navigation function and the operational boundary (Operational Design Domain, ODD) of the system to be evaluated may be determined first.

[0028] Among them, the automatic driving navigation function of the system to be evaluated can be determined according to the operating function definition of the system to be evaluated, and the functions under the information assistance category can be: driver state detection, lane departure warning, etc., the functions under the control assistance category can be: automatic emergency braking, automatic emergency steering, lane keeping assistance, intelligent speed limit control, adaptive cruise control, etc., and the functions under other categories can be: driver takeover, minimum risk strategy, etc. As shown in Table 1, the various automatic driving navigation functions of the NOA system and the operating mechanism and description of each function are exemplarily shown.

[0029] Table 1. NOA system’s autonomous driving navigation functions and descriptions

[0030]

[0031] In addition, the operating boundary of the system to be evaluated can also be determined. The operating boundary (i.e., the operating design domain) is the set of environmental conditions, road types, traffic flow, weather conditions, etc., in which the autonomous vehicle is designed and expected to operate safely. The operating boundary defines the boundary within which the autonomous vehicle can operate safely and is a prerequisite for the autonomous driving navigation function and performance of the system to be evaluated.

[0032] Specifically, the overall conditions of the operating domain of the system to be evaluated, that is, the operating boundary, can be clarified by integrating the operating mechanisms corresponding to the various autonomous driving navigation functions of the system to be evaluated. As shown in Table 2, the operating domain of the NOA system is exemplified.

[0033] Table 2 NOA system operation domain

[0034]

[0035] In the embodiment of the present invention, the evaluation indicators may be defined as security, efficiency, reliability and user satisfaction, so as to evaluate the system to be evaluated with respect to security, efficiency, reliability and user satisfaction respectively.

[0036] Furthermore, the expected functions, operating scope and application scenarios of the system to be evaluated can be studied through the autonomous driving navigation function and operating boundaries of the system to be evaluated, and the associated data items of each evaluation indicator can be determined.

[0037] For example, the associated data items of safety include reaction time, braking distance, accident rate, etc., the associated data items of efficiency include path optimization, time saving, fuel economy, etc., the associated data items of reliability include system stability, failure rate, maintenance cycle, etc., and user satisfaction includes user feedback information obtained through questionnaire surveys, etc. As shown in Table 3, the evaluation indicators of the NOA system are shown as an example.

[0038] Table 3 Evaluation indicators of NOA system

[0039]

[0040] Furthermore, actual driving data of the system to be evaluated may be collected. When collecting actual driving data, experimental data, user feedback, system logs and other related data generated during the driving process of the system to be evaluated may be collected according to the associated data items to obtain associated data corresponding to the associated data items. The associated data may include operation data, path optimization rate, time saved, historical fault data and user feedback information.

[0041] After collecting the relevant data of the system to be evaluated, the collected relevant data can be preprocessed. Specifically, assuming Represents a collection of linked data, where each It is a multidimensional data point representing different performance indicators.

[0042] First, for all associated data , you can clean the data and remove outliers. Define the outlier identification function ,if is an outlier, then ,otherwise . The threshold T can be set based on the Z-Score method to identify outliers:

[0043] ;

[0044] in, is the i-th data, for The corresponding Z-Score, and are the mean and standard deviation of the associated data respectively. The threshold T represents the degree of deviation of the data, which is usually 2 or 3. At this time, the cleaned associated data is:

[0045] ;

[0046] Secondly, the cleaned data can also be normalized. Specifically, the data can be normalized to the minimum and maximum values:

[0047] ;

[0048] in, and are the minimum and maximum values ​​in the associated data, respectively, and then normalized by Z-Score:

[0049] ;

[0050] in, and They are the cleaned associated data Through the above process, the collected related data After cleaning and normalization, it is transformed into data suitable for further analysis and modeling This process can ensure the quality and consistency of the associated data and provide data guarantee for subsequent performance evaluation.

[0051] S120, classifying all operation data in the associated data based on the trained support vector machine model to obtain the risk level of each operation data, and determining a safety evaluation value based on the risk level of each operation data.

[0052] The support vector machine model can be pre-trained using a sample data set. Specifically, the support vector machine model can find a segmentation plane in the feature space and maximize the margin between data points of different categories to identify high-risk and low-risk operation modes. The support vector machine model can be defined as:

[0053] ;

[0054] in, is the weight vector, b is the bias term, is the input feature vector (such as reaction time, braking distance, etc.). The objective function can be designed as:

[0055] ;

[0056] in, is a slack variable, representing the data point The degree of violation of the distance margin, is a regularization parameter used to control the trade-off between classification interval and classification error. Considering that driving data is generally nonlinearly separable data, that is, positive examples and negative examples cannot be completely separated by a hyperplane, it is necessary to use a nonlinear classifier or map the data to a high-dimensional space.

[0057] Therefore, in the embodiment of the present invention, when processing nonlinearly separable operation data, the data is mapped to a high-dimensional space through the kernel trick, and the inner product between two samples is calculated in the low-dimensional space, thereby avoiding explicit calculations in the high-dimensional space. The kernel function is defined as the radial basis function (RBF) kernel:

[0058] ;

[0059] in, is the parameter of the kernel function. For the new input feature vector x, the decision function of the SVM model is defined as:

[0060] ;

[0061] in, It is a sign function that determines the classification based on the positive or negative value of f(x) (“+1” represents low risk and “-1” represents high risk).

[0062] Specifically, the trained support vector machine model can be used to classify all the operation data (including reaction time and braking distance) of the system to be evaluated, so as to identify the risk level of each operation data, wherein the risk level can reflect the safety of the operation data.

[0063] Furthermore, the safety evaluation value of the system to be evaluated can be calculated according to the risk level of each operation data. Exemplarily, the safety evaluation value can be determined according to the frequency of occurrence of low-risk operation data or high-risk operation data in all operation data, for example, the higher the frequency of occurrence of high-risk operation data, the lower the safety evaluation value.

[0064] In a specific implementation, determining a safety evaluation value based on the risk level of each operation data includes the following steps:

[0065] Step 11: for each operation data at a low risk level, determine a score of the operation data based on the distance between the operation data and the classification boundary, so as to reduce the score of the operation data when the distance is close;

[0066] Step 12: for each operation data with a high risk level, determine the score of the operation data based on the distance between the operation data and the classification boundary, so as to reduce the score of the operation data when the distance is far;

[0067] Step 13: Determine the sum of low-risk scores based on the scores of all low-risk operation data and the weights corresponding to each operation data, and determine the sum of high-risk scores based on the scores of all high-risk operation data and the weights corresponding to each operation data;

[0068] Step 14: Determine a safety evaluation value based on the sum of the low risk scores and the sum of the high risk scores.

[0069] In the above step 11, for each operation data of the low risk level, the corresponding score can be determined based on the distance between the operation data and the classification boundary. The smaller the distance between the operation data of the low risk level and the classification boundary, the closer the operation data is to the high risk level, and the lower score can be assigned to the operation data. For example, a positive correlation function between the boundary distance and the score can be pre-fitted, and the positive correlation function is used to reduce the score of the operation data when the boundary distance is close, and then the boundary distance is substituted into the positive correlation function to obtain the corresponding score.

[0070] In the above step 12, for each operation data of the high risk level, the corresponding score can be determined based on the distance between the operation data and the classification boundary. The smaller the distance between the operation data of the high risk level and the classification boundary, the closer the operation data is to the low risk level, and the higher score can be assigned to the operation data. For example, a negative correlation function between the boundary distance and the score can be pre-fitted, and the negative correlation function is used to improve the score of the operation data when the boundary distance is close, and then the boundary distance is substituted into the negative correlation function to obtain the corresponding score.

[0071] Furthermore, in step 13, the sum of low risk scores can be calculated according to the scores of all low risk level operation data and the weights corresponding to each operation data, as shown in the following formula:

[0072] ;

[0073] Where L is the sum of low risk scores, n is the number of operation data at the low risk level, is the score corresponding to the i-th low-risk level operation data, is the weight corresponding to the i-th low-risk level operation data, and the weight corresponding to each operation data may be preset.

[0074] Furthermore, in step 13, the sum of high-risk scores may be calculated according to the scores of all high-risk operation data and the weights corresponding to each operation data, as shown in the following formula:

[0075] ;

[0076] In the formula, H is the total number of high-risk scores, m is the number of high-risk level operation data, is the score corresponding to the j-th high-risk level operation data, is the weight corresponding to the j-th high-risk level operation data, and the weight corresponding to each operation data may be preset.

[0077] Furthermore, in step 14, the sum of the low risk scores and the sum of the high risk scores may be added to obtain a safety evaluation value:

[0078] ;

[0079] In the formula, is the safety evaluation value.

[0080] In the above steps 11 to 14, the score of the operation data is determined by the distance between the operation data and the classification boundary, so as to lower the score of the operation data for low-risk operation data when the distance is close, and increase the score of the operation data for high-risk operation data when the distance is close. This realizes a safety assessment based on the risk level of the operation data, and can accurately assess the safety of the system in combination with the risk level of the reaction time and braking distance of the system to be evaluated, thereby effectively improving the accuracy of the safety assessment.

[0081] S130 , performing kernel density estimation on the path optimization rate and the saved time in the associated data to obtain a probability density function, and determining an efficiency evaluation value based on a peak value in the probability density function.

[0082] Among them, the path optimization rate can refer to the probability that the system to be evaluated optimizes the vehicle's driving path. For example, it can be determined whether the system to be evaluated optimizes the vehicle's driving path by judging whether the vehicle's initial planned path is the same as the actual driving path, and then the path optimization rate can be determined by the frequency of the system to be evaluated optimizing the vehicle's driving path.

[0083] The saved time may refer to the travel time saved by the evaluated system by optimizing the vehicle travel path. For example, the saved time may be calculated based on the estimated travel time corresponding to the initial planned path and the actual travel time corresponding to the optimized travel path.

[0084] In the embodiment of the present invention, the probability density distribution of the path optimization rate can be determined using kernel density estimation to obtain the probability density function of the path optimization rate, and the probability density distribution of time saving can be determined using kernel density estimation to obtain the probability density function of time saving. By analyzing the probability density function, the distribution characteristics of the system efficiency index and the possible optimization direction can be understood.

[0085] Given a set of independent and identically distributed sample data , the kernel density estimation can be expressed as:

[0086] ;

[0087] In the formula, is the kernel function, n is the number of samples, is the bandwidth parameter, which is used to control the smoothness of the estimate. The choice of has a great influence on the result of kernel density estimation. Too small bandwidth will lead to overfitting, while too large bandwidth will lead to underfitting. Usually the bandwidth can be automatically selected by methods such as cross-validation. Since the Gaussian kernel is smooth and convenient in mathematical processing, the kernel function here chooses the Gaussian kernel:

[0088] ;

[0089] In the formula, Represents the mean.

[0090] In a specific implementation, performing kernel density estimation on the path optimization rate and the time saving in the associated data to obtain a probability density function includes the following steps:

[0091] Step 21: Based on the data volume of the path optimization rate in the associated data, the corresponding bandwidth parameter and the preset kernel function, kernel density estimation is performed on the path optimization rate in the associated data to obtain a probability density function of the path optimization rate;

[0092] Step 22: Based on the amount of time-saving data in the associated data, the corresponding bandwidth parameter, and the preset kernel function, kernel density estimation is performed on the time-saving in the associated data to obtain a probability density function of the time-saving.

[0093] In step 21, when determining the probability density function of the path optimization rate using kernel density estimation, it can be assumed that Represents the optimization rate of each path in the associated data, and applies kernel density estimation to the path optimization rate:

[0094] ;

[0095] In the formula, represents the probability density function of the path optimization rate, is the number of path optimization rates, The bandwidth of the path optimization rate.

[0096] In step 22, when determining the time-saving probability density function using kernel density estimation, it can be assumed that Represent the individual time savings in the association data and apply kernel density estimation to the time savings:

[0097] ;

[0098] In the formula, represents the probability density function of time saving, To save time, To save time and bandwidth.

[0099] By analyzing and , the distribution characteristics of path optimization rate and time saving can be understood. The analysis results can indicate the potential optimization direction of efficiency indicators. If the distribution of time saving has a long tail characteristic, it may indicate the existence of significant time saving in extreme cases. Through this process, kernel density estimation provides an intuitive and effective way to analyze and understand the distribution characteristics of NOA system efficiency indicators, thereby providing valuable information for further system optimization and decision-making.

[0100] After obtaining the probability density function of the path optimization rate and the time saved, the efficiency evaluation value can be further determined according to the peak value in the probability density function. For example, if the peak value in the distribution of the path optimization rate or the time saved is in the preset high efficiency interval, a high score is given; if the peak value in the distribution is in the medium efficiency interval, a medium score is given; if the peak value is in the low efficiency interval, a low score is given.

[0101] In a specific implementation, determining the efficiency evaluation value based on the peak value in the probability density function includes the following steps:

[0102] Step 31: In each preset efficiency interval, determine the first efficiency interval where the peak value of the probability density function of the path optimization rate is located, and determine the second efficiency interval where the peak value of the probability density function of the time saving is located;

[0103] Step 32: Determine an efficiency evaluation value according to the efficiency score corresponding to the first efficiency interval and the efficiency score corresponding to the second efficiency interval.

[0104] That is, multiple efficiency intervals can be preset, and corresponding efficiency scores can be set for each preset efficiency interval. Further, the interval where the peak value of the probability density function of the path optimization rate is located can be queried in all preset efficiency intervals to obtain the first efficiency interval, and the interval where the peak value of the probability density function of the time saving is located can be queried in all preset efficiency intervals to obtain the second efficiency interval.

[0105] Furthermore, the efficiency score corresponding to the first efficiency interval and the efficiency score corresponding to the second efficiency interval may be combined to obtain an efficiency evaluation value. For example, the efficiency evaluation value is:

[0106] ;

[0107] In the formula, is the efficiency evaluation value, , They are the weights corresponding to the path optimization rate and the weights corresponding to the time saved, , They are respectively the efficiency score corresponding to the first efficiency interval (i.e., the score corresponding to the path optimization rate analyzed based on the kernel density estimation) and the efficiency score corresponding to the second efficiency interval (i.e., the score corresponding to the time saved analyzed based on the kernel density estimation).

[0108] Through the above implementation, it is possible to determine the efficiency evaluation value based on the peak value in the probability density function, thereby ensuring the accuracy of the efficiency evaluation of the NOA system.

[0109] S140, predicting historical fault data in the associated data based on the trained autoregressive model to obtain future fault trends, and determining a reliability evaluation value based on the fault trends and the prediction accuracy of the autoregressive model.

[0110] The autoregressive model can be obtained by training a sample data set in advance. The autoregressive model is a model in time series analysis, which is used to describe the relationship between a variable and its own past values. In the embodiment of the present invention, it can be used to predict future failure rates.

[0111] Specifically, for a time series , an autoregressive model of order p can be expressed as:

[0112] ;

[0113] in, are model parameters, is the white noise term. The parameters of the autoregressive model are The least squares method can be used here to estimate by minimizing the prediction error:

[0114] ;

[0115] Where n is the total length of the time series. Once the parameters of the autoregressive model are estimated, the autoregressive model can be used to predict future failure rates. For the next time point t+1, the failure prediction can be expressed as:

[0116] ;

[0117] Where p is the number of historical time points used in predicting future failure trends. is the failure rate at the next moment (i.e. the future).

[0118] After obtaining the future failure rate through the autoregressive model, the historical failure data and the future failure rate can be combined to determine the future failure trend, that is, the failure rate is increasing, the failure rate is decreasing, or the failure rate is stable.

[0119] Furthermore, the reliability evaluation value can be determined based on the future failure trend and the prediction accuracy of the autoregressive model. For example, if the prediction accuracy is high and the future failure trend is a stable failure rate or a declining failure rate, the reliability evaluation value is increased; if the prediction accuracy is low or the future failure trend is an increasing failure rate, the reliability evaluation value is decreased.

[0120] In a specific implementation, determining a reliability evaluation value based on the prediction accuracy of the failure trend and the autoregressive model includes the following steps:

[0121] Step 41: If the fault trend is that the fault rate is stable or the fault rate is decreasing, the preset first score is determined as the fault score; otherwise, the preset second score is determined as the fault score, wherein the preset first score is higher than the preset second score;

[0122] Step 42: Determine a reliability evaluation value based on the prediction mean square error and the prediction root mean square error of the autoregressive model and the fault score.

[0123] In the above step 41, if the fault trend is that the fault rate is stable or the fault rate is decreasing, the preset first score is used as the fault score, otherwise, the preset second score lower than the preset first score is used as the fault score. It should be noted that the fault score can also be determined in combination with the change range of the fault rate. For example, if the fault trend is that the fault rate is decreasing, the greater the change range of the fault rate, the higher the fault score.

[0124] Furthermore, in the above step 42, the prediction accuracy of the autoregressive model can be described by the prediction mean square error and the prediction root mean square error of the autoregressive model, and the reliability score value can be obtained by combining the prediction accuracy and the fault score:

[0125] ;

[0126] In the formula, is the reliability score, MSE is the prediction mean square error, RMSE is the prediction root mean square error, and TrendScore is the score given based on the positive and negative and magnitude of the fault trend. , , is a weight factor that can be set according to the importance of prediction accuracy and fault trend.

[0127] Through the above steps 41 and 42, the reliability of the system to be evaluated can be evaluated in combination with the failure trend of the system to be evaluated and the accuracy of predicting the failure trend, and the stability of the system to be evaluated can be accurately measured to ensure the accuracy of the evaluation of the NOA system.

[0128] S150: Determine a satisfaction change trend based on user feedback information in the associated data, and determine a user satisfaction evaluation value based on the satisfaction change trend.

[0129] The user feedback information may be data collected through online survey or application feedback mechanism, including user satisfaction scores. In the embodiment of the present invention, the long-term changes of user satisfaction may be tracked through time series analysis method.

[0130] set up represents the user satisfaction score (i.e., user feedback information) at time point t. The user satisfaction dataset can be obtained through online surveys or in-app feedback mechanisms and can be expressed as , where n is the total number of feedbacks.

[0131] For the collected user feedback information, the satisfaction change trend can be determined by moving average method or exponential smoothing method.

[0132] In a specific implementation, determining the satisfaction change trend based on user feedback information in the associated data includes the following steps:

[0133] Step 51: for each user feedback information at a time point in the associated data, determine the average satisfaction level at the time point based on the user feedback information at the time point and the user feedback information before the time point;

[0134] Step 52: Determine the satisfaction change trend based on the average satisfaction at each time point.

[0135] In the above step 51, a moving average may be applied to the user feedback information:

[0136] ;

[0137] In the formula, is the average satisfaction at time point t, m is the time window size of the moving average, which means considering the average satisfaction at the past m time points, The user feedback information at the i-th time point before (including) time point t.

[0138] Furthermore, in the above step 52, the average satisfaction at adjacent time points may be compared to obtain a satisfaction change trend, that is, a change trend of user satisfaction over time, such as a stable increase in satisfaction, a stable decrease in satisfaction, or fluctuation in satisfaction.

[0139] Through the above steps 51 and 52, user satisfaction analysis based on moving average is realized, which can accurately analyze the dynamically changing user satisfaction. Combined with long-term data, the changes in user satisfaction with the NOA system are comprehensively analyzed, ensuring the accuracy of the evaluation of user satisfaction with the NOA system.

[0140] In another specific implementation, determining the satisfaction change trend based on user feedback information in the associated data includes the following steps:

[0141] Step 61: for each time point in the associated data, determine the smoothed satisfaction value of the time point based on the user feedback information at the time point, the weight corresponding to the time point, the smoothed satisfaction value at the previous time point, and the weight corresponding to the previous time point;

[0142] Step 62: Determine the satisfaction change trend based on the satisfaction smoothing value at each time point.

[0143] In the above step 61, exponential smoothing may be applied to the user feedback information:

[0144] ;

[0145] In the formula, is the smoothed value of satisfaction at time point t, is the user feedback information at time point t, is the smoothed value of satisfaction at time point t-1, is the smoothing parameter.

[0146] Furthermore, in the above step 62, the satisfaction change trend can be obtained by comparing the smoothed satisfaction values ​​at adjacent time points, that is, the change trend of user satisfaction over time, such as stable increase in satisfaction, stable decrease in satisfaction, or fluctuation in satisfaction.

[0147] Through the above steps 61 and 62, user satisfaction analysis based on exponential smoothing is realized, which can give higher weight to recent data, accurately analyze dynamically changing user satisfaction, and comprehensively analyze changes in user satisfaction with the NOA system in combination with long-term data, thereby ensuring the accuracy of the evaluation of user satisfaction with the NOA system.

[0148] After determining the satisfaction change trend, the user satisfaction evaluation value can be further determined according to the satisfaction change trend. For example, if the satisfaction change trend is a steady increase in satisfaction, the first preset value is used as the user satisfaction evaluation value, and if the satisfaction change trend is a steady decrease in satisfaction or a fluctuation in satisfaction, the second preset value is used as the user satisfaction evaluation value, wherein the second preset value is lower than the first preset value.

[0149] Alternatively, abnormal fluctuations can be identified and scored. Large fluctuations may indicate that the system to be evaluated has systemic problems or event impacts, which require special attention. Therefore, trends and fluctuations can be considered comprehensively, and trend scores and stability scores can be determined based on the satisfaction change trend. Then, the user satisfaction evaluation value can be obtained based on the trend score and stability score, such as:

[0150] ;

[0151] In the formula, is the user satisfaction evaluation value, , is a weight factor, which can be divided according to the trend and stability The relative importance of the trend is set. The greater the downward trend, the lower the trend score, the greater the upward trend, the higher the trend score, the greater the volatility, the lower the stability score, and the smaller the volatility, the higher the stability score.

[0152] S160, determining a comprehensive evaluation result based on the safety evaluation value, the efficiency evaluation value, the reliability evaluation value and the user satisfaction evaluation value, and determining the safety level and performance level of the system to be evaluated according to the comprehensive evaluation result.

[0153] After obtaining the safety evaluation value, efficiency evaluation value, reliability evaluation value and user satisfaction evaluation value, safety, efficiency, reliability and user satisfaction can be combined to obtain a comprehensive evaluation result.

[0154] In the embodiment of the present invention, the security evaluation value, efficiency evaluation value, reliability evaluation value and user satisfaction evaluation value can be weighted by the weights corresponding to security, efficiency, reliability and user satisfaction respectively to obtain a comprehensive evaluation result.

[0155] For example, suppose They represent the weights of safety, efficiency, reliability and user satisfaction respectively. The weights can be set by the expert group according to the importance of each indicator and the system goal. .

[0156] For example, the following method can be used to obtain the weights of security, efficiency, reliability and user satisfaction: First, the expert group evaluates each indicator based on their domain knowledge and experience and gives its contribution to the system goal. This process can be done through expert discussion, expert survey or expert scoring. In addition, by analyzing the overall goals of the system, determine the importance of each indicator for achieving these goals, including considering factors such as the key requirements of the system, user expectations, market competition and organizational strategy. It is also necessary to communicate and discuss with stakeholders of the system (such as users, management, technical personnel, etc.) to understand their views and priorities on each indicator, so as to determine the importance of the indicator and the system goals.

[0157] Alternatively, the weights of safety, efficiency, reliability, and user satisfaction can be determined by mathematical decision-making methods: by using decision-making methods such as the Analytic Hierarchy Process (AHP) or the weight assignment method to help determine the relative importance of indicators. AHP is used to determine the relative importance of multiple indicators. This method compares indicators pairwise, constructs a hierarchy, and then uses expert judgment to score the relative importance of each pair of indicators.

[0158] In the embodiment of the present invention, the comprehensive evaluation result is calculated based on the weights of security, efficiency, reliability and user satisfaction, as well as the security evaluation value, efficiency evaluation value, reliability evaluation value and user satisfaction evaluation value.

[0159] After obtaining the comprehensive evaluation results of the system to be evaluated, the safety level and performance level of the system to be evaluated can be determined based on the comprehensive evaluation results.

[0160] For example, a high security level threshold and a medium security level threshold may be preset, and a high performance level threshold and a medium performance level threshold may be preset.

[0161] If the comprehensive evaluation result of the system to be evaluated is greater than the high security level threshold, the security level of the system to be evaluated is determined to be high. If the comprehensive evaluation result of the system to be evaluated is greater than the high performance level threshold, the performance level of the system to be evaluated is determined to be high.

[0162] If the comprehensive evaluation result of the system to be evaluated is between the high security level threshold and the medium security level threshold, the security level of the system to be evaluated is determined to be medium. If the comprehensive evaluation result of the system to be evaluated is between the high performance level threshold and the medium performance level threshold, the performance level of the system to be evaluated is determined to be medium.

[0163] If the comprehensive evaluation result of the system to be evaluated is less than the medium security level threshold, the security level of the system to be evaluated is determined to be low. If the comprehensive evaluation result of the system to be evaluated is less than the medium performance level threshold, the performance level of the system to be evaluated is determined to be low.

[0164] After obtaining the security level and performance level of the system to be evaluated, it is also possible to determine whether the system to be evaluated needs to be optimized based on the security level and performance level. For example, if the security level does not exceed the medium level, or the performance level is a low level, it is determined that the system to be evaluated needs to be optimized. Furthermore, if the system to be evaluated needs to be optimized, optimization prompt information can be generated and sent to the user terminal associated with the system to be evaluated.

[0165] In addition, batches of NOA systems can be evaluated to determine the security level and performance level of each NOA system, and then based on the security level and performance level of each NOA system, the NOA systems to be optimized can be screened out. Furthermore, for each screened NOA system to be optimized, the optimization priority of each system can be determined based on the security level and performance level.

[0166] In the embodiment of the present invention, considering that in the process of conducting experiments on the system to be evaluated, the associated data of the system to be evaluated can be continuously collected, therefore, the support vector machine model and the autoregressive model can also be continuously optimized in combination with the data collected in real time.

[0167] Optionally, the method provided by the embodiment of the present invention further includes:

[0168] New associated data are collected; the support vector machine model is updated based on the operation data in the new associated data, and the autoregressive model is updated based on the historical fault data in the new associated data.

[0169] That is, the support vector machine model can be updated and trained based on the operation data in the new associated data, and the autoregressive model can be updated and trained based on the historical fault data in the new associated data, so as to continuously optimize the support vector machine model and the autoregressive model and ensure their adaptability and effectiveness.

[0170] In addition, new related data can be combined to continuously optimize the weights involved in the entire automatic navigation assisted driving system evaluation method.

[0171] The present invention has the following technical effects: first, according to the automatic driving navigation function and operation boundary of the system to be evaluated, the associated data items of each evaluation index are determined, and the associated data are collected through the associated data items, and then all the operation data therein are classified based on the trained support vector machine model to obtain the risk level of each operation data, determine the safety evaluation value, and perform kernel density estimation on the path optimization rate and the time saved to obtain the probability density function, and determine the efficiency evaluation value in combination with the peak value thereof, and predict the historical fault data according to the trained autoregressive model to obtain the future fault trend, and determine the reliability evaluation value in combination with the prediction accuracy of the autoregressive model, and determine the satisfaction change trend according to the user feedback information to obtain the user satisfaction evaluation value, and finally comprehensively evaluate the safety. The safety evaluation value, efficiency evaluation value, reliability evaluation value and user satisfaction evaluation value are used to determine the safety level and performance level of the system to be evaluated, realizing the quantitative evaluation of the NOA system, which can help determine whether the NOA system needs to be optimized. This method realizes the comprehensive evaluation of the NOA system by integrating multiple characteristics of system performance, significantly enhancing the scope of application and generalization ability of the evaluation method, and by accurately analyzing the user satisfaction and efficiency indicators detected in real time, the comprehensiveness and adaptability of the evaluation method are improved, so that the evaluation results can accurately reflect the actual performance of the system and user experience. In addition, this method uses support vector machines and autoregressive models to conduct in-depth analysis and prediction of key performance indicators, which can further improve the evaluation accuracy.

[0172] The method provided by the embodiment of the present invention has the following beneficial effects:

[0173] (1) In view of the inadequacy and inaccuracy of the NOA system safety performance and reliability evaluation, this method proposes to use statistical and machine learning techniques such as support vector machines and autoregressive models to achieve in-depth analysis and prediction of these key performance indicators, thereby improving the accuracy and foresight of the evaluation;

[0174] (2) In view of the limitations of user satisfaction monitoring and efficiency index analysis, this method introduces kernel density estimation and dynamic risk assessment methods to achieve real-time monitoring of user satisfaction and accurate analysis of efficiency indicators, thus improving the comprehensiveness and adaptability of the evaluation method.

[0175] (3) In order to address the problem of insufficient generalization ability of existing NOA system evaluation methods, this method optimizes the comprehensive scoring algorithm and achieves a comprehensive evaluation of the NOA system by integrating multiple analysis results, significantly enhancing the scope of application and practical value of the evaluation method.

[0176] Figure 2 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 2As shown, the electronic device 400 includes one or more processors 401 and a memory 402 .

[0177] The processor 401 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0178] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may run the program instructions to implement the automatic navigation assisted driving system evaluation method of any embodiment of the present invention described above and / or other desired functions. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0179] In one example, the electronic device 400 may further include: an input device 403 and an output device 404, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 403 may include, for example, a keyboard, a mouse, etc. The output device 404 may output various information to the outside, including early warning prompt information, braking force, etc. The output device 404 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0180] Of course, to simplify, Figure 2 Only some of the components related to the present invention in the electronic device 400 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 400 may also include any other appropriate components.

[0181] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the automatic navigation assisted driving system evaluation method provided by any embodiment of the present invention.

[0182] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0183] In addition, an embodiment of the present invention may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the automatic navigation assisted driving system evaluation method provided by any embodiment of the present invention.

[0184] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0185] It should be noted that the terms used in the present invention are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0186] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. An automatic navigation assisted driving system evaluation method, characterized in that: include: Determine the associated data items of each evaluation indicator based on the autonomous driving navigation function and the operation boundary of the system to be evaluated, and collect the associated data of the system to be evaluated according to the associated data items; Classifying all the operation data in the associated data based on the trained support vector machine model to obtain a risk level of each operation data, and determining a safety evaluation value based on the risk level of each operation data, wherein the operation data includes reaction time and braking distance; Performing kernel density estimation on the path optimization rate and the time saving in the associated data to obtain a probability density function, and determining an efficiency evaluation value based on a peak value in the probability density function; Predicting historical fault data in the associated data based on the trained autoregressive model to obtain future fault trends, and determining a reliability evaluation value based on the fault trends and the prediction accuracy of the autoregressive model; Determine a satisfaction change trend based on the user feedback information in the associated data, and determine a user satisfaction evaluation value based on the satisfaction change trend; Determine a comprehensive evaluation result based on the safety evaluation value, the efficiency evaluation value, the reliability evaluation value and the user satisfaction evaluation value, and determine the safety level and performance level of the system to be evaluated according to the comprehensive evaluation result; Perform kernel density estimation on the path optimization rate and the time saving in the associated data to obtain a probability density function, including: Based on the data volume of the path optimization rate in the associated data, the corresponding bandwidth parameter and the preset kernel function, kernel density estimation is performed on the path optimization rate in the associated data to obtain a probability density function of the path optimization rate; Based on the amount of time-saving data in the associated data, the corresponding bandwidth parameter and a preset kernel function, performing kernel density estimation on the time saved in the associated data to obtain a probability density function of the time saved; The efficiency evaluation value is determined based on the peak value in the probability density function, including: In each preset efficiency interval, determine a first efficiency interval where a peak value in a probability density function of a path optimization rate is located, and determine a second efficiency interval where a peak value in a probability density function of time saving is located; Determining an efficiency evaluation value according to an efficiency score corresponding to the first efficiency interval and an efficiency score corresponding to the second efficiency interval; Determining a reliability evaluation value based on the failure trend and the prediction accuracy of the autoregressive model includes: If the fault trend is that the fault rate is stable or the fault rate is decreasing, the preset first score is determined as the fault score, otherwise, the preset second score is determined as the fault score, wherein the preset first score is higher than the preset second score; A reliability evaluation value is determined based on the prediction mean square error and the prediction root mean square error of the autoregressive model and the fault score.

2. The method according to claim 1, characterized in that The safety evaluation value is determined based on the risk level of each operation data, including: For each operation data of a low risk level, a score of the operation data is determined based on the distance between the operation data and the classification boundary, so as to reduce the score of the operation data when the distance is close; For each operation data of a high risk level, a score of the operation data is determined based on a distance between the operation data and a classification boundary, so as to reduce the score of the operation data when the distance is far; Determine the sum of low-risk scores based on the scores of all low-risk operation data and the weights corresponding to each operation data, and determine the sum of high-risk scores based on the scores of all high-risk operation data and the weights corresponding to each operation data; A safety rating value is determined based on the sum of the low risk scores and the sum of the high risk scores.

3. The method according to claim 1, characterized in that Determining a satisfaction change trend based on the user feedback information in the associated data includes: For each user feedback information at a time point in the associated data, based on the user feedback information at the time point and user feedback information before the time point, determine an average satisfaction level at the time point; Determine the satisfaction trend based on the average satisfaction at each time point.

4. The method according to claim 1, characterized in that: Determining a satisfaction change trend based on the user feedback information in the associated data includes: For each time point of the user feedback information in the associated data, determining a smoothed satisfaction value at the time point based on the user feedback information at the time point, the weight corresponding to the time point, the smoothed satisfaction value at the previous time point, and the weight corresponding to the previous time point; The satisfaction change trend is determined based on the smoothed satisfaction value at each time point.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Collect new contextual data; The support vector machine model is updated based on the operation data in the new associated data, and the autoregressive model is updated based on the historical fault data in the new associated data.

6. An electronic device, characterized in that: The electronic device comprises: Processor and memory; The processor is used to execute the steps of the automatic navigation assisted driving system evaluation method as described in any one of claims 1 to 5 by calling the program or instructions stored in the memory.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the automatic navigation assisted driving system evaluation method as described in any one of claims 1 to 5.