Method and device for evaluating vehicle function and computer readable storage medium

Abstract

The embodiment of the invention provides a method and device for evaluating vehicle functions and a computer readable storage medium. The method may include receiving a test data set associated with a function of the vehicle, where the test data set is obtained prior to a durability test for the function of the vehicle; processing the test data set using a machine learning model to be set in the vehicle to perform the function to generate a test result set; the test result set is used for determining out-of-distribution data in the test data set, and the out-of-distribution data and test results corresponding to the out-of-distribution data are used for evaluating the performance of the machine learning model for executing the function before the durability test.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicles, and in particular, to methods, devices, and computer-readable storage media for evaluating vehicle functions. Background Art

[0002] To ensure the safe and reliable use of a vehicle after it enters the market, it is usually necessary to conduct long-term or long-mileage tests on the vehicle on the road before the vehicle enters the market, so that the vehicle can travel multiple times under various conditions as much as possible, in order to effectively evaluate the performance of the vehicle function. However, this may consume a large amount of manpower and material resources, be inefficient, and also extend the cycle for the vehicle to enter the market. Summary of the Invention

[0003] In view of the need to improve the prior art, embodiments of the present disclosure provide methods, devices, and computer-readable storage media for evaluating vehicle functions.

[0004] On the one hand, embodiments of the present disclosure provide a method for evaluating a vehicle function, including: receiving a test data set associated with the function of the vehicle, where the test data set is obtained before a durability test for the function of the vehicle; processing the test data set using a machine learning model to generate a test result set, where the machine learning model will be set in the vehicle to perform the function, the test result set is used to determine out-of-distribution data in the test data set, and the out-of-distribution data and the test results corresponding to the out-of-distribution data are used to evaluate the performance of the machine learning model in performing the function before the durability test.

[0005] On the other hand, embodiments of the present disclosure provide a device for evaluating the performance of a vehicle function, including: a receiving unit configured to receive a test data set associated with the function of the vehicle, where the test data set is obtained before a durability test for the function of the vehicle; a model processing unit configured to process the test data set using a machine learning model to generate a test result set, where the machine learning model will be set in the vehicle to perform the function, the test result set is used to determine out-of-distribution data in the test data set, and the out-of-distribution data and the test results corresponding to the out-of-distribution data are used to evaluate the performance of the machine learning model in performing the function before the durability test.

[0006] On the other hand, embodiments of the present disclosure provide an apparatus for evaluating the execution performance of vehicle functions, including: at least one processor; a memory communicatively coupled to the at least one processor and storing executable code, which, when executed by the at least one processor, causes the at least one processor to execute the above method.

[0007] On the other hand, embodiments of the present disclosure provide a computer-readable storage medium storing executable code, which, when executed, implements the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other objects, features and advantages of the embodiments of the present disclosure will become more apparent from the following more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings, in which like reference numerals generally represent like elements in the various drawings.

[0009] Figure 1 is a schematic flowchart of a method for evaluating vehicle functions according to some embodiments.

[0010] Figure 2 is a schematic flowchart of a process for evaluating vehicle functions according to some embodiments.

[0011] Figure 3 is a schematic block diagram of an apparatus for evaluating vehicle functions according to some embodiments.

[0012] Figure 4 is a schematic block diagram of an apparatus for evaluating vehicle functions according to some embodiments. DETAILED DESCRIPTION

[0013] The subject matter described herein will now be discussed with reference to various embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not intended to limit the scope of protection, applicability, or examples set forth in the claims.

[0014] In today's increasingly competitive market, various manufacturers or suppliers related to vehicles need to adapt to the development trends of new technologies and the market at an increasingly rapid pace. This means that manufacturers or suppliers need to shorten the development or mass production cycle of vehicles. On the other hand, in order to ensure the safety and reliability of vehicle use, before mass-producing vehicles, it is usually necessary to conduct long-term or long-distance tests on the road, such as endurance runs. However, such tests often consume a large amount of manpower and material resources and also extend the development or mass production cycle of vehicles.

[0015] In view of this, embodiments of the present disclosure provide a technical solution for evaluating vehicle functions. The following will be described in conjunction with specific embodiments.

[0016] Figure 1 is a schematic flowchart of a method for evaluating vehicle functions according to some embodiments.

[0017] As Figure 1 shown, in step 102, a test data set associated with the function of the vehicle can be received. The test data set can be obtained before a durability test for the function of the vehicle. For example, the test data set can be obtained by at least one sensor on a road test vehicle during an early road test of the vehicle.

[0018] In step 104, the test data set can be processed using a machine learning model to generate a test result set. The machine learning model will be set in the vehicle to perform the function. That is, the vehicle will be equipped with such a machine learning model to perform the function.

[0019] The test result set can be used to determine out-of-distribution (OOD) data in the test data set. The OOD data and the test results corresponding to the OOD data can be used to evaluate the performance of the machine learning model in performing the function before the durability test.

[0020] OOD data generally refers to data having a distribution different from or completely unknown to the distribution of the training data of the machine learning model. The ability of the machine learning model to detect OOD data is closely related to the stability and security of the machine learning model. Therefore, the OOD data and its corresponding test results will be able to serve as an important basis for evaluating the performance of the machine learning model in performing vehicle functions.

[0021] It can be seen that in the embodiments of the present disclosure, a test data set can be obtained before the durability test of the vehicle. Based on the test data set, the machine learning model that will be set in the vehicle to perform the function is tested to obtain a test result set, and the test result set can be used to determine the OOD data in the test data set. Thus, before the durability test, the performance of the machine learning model in performing the function can be evaluated based on the OOD data and its corresponding test results. In this way, the execution performance of vehicle functions can be efficiently evaluated, which can serve as a supplement to traditional durability tests, thereby reducing the duration or mileage of durability tests, and further reducing the overall vehicle test cost, and also ensuring the effectiveness of the overall test.

[0022] In embodiments of the present disclosure, the machine learning model may include a deep neural network (DNN) or other applicable models, which is not limited herein.

[0023] In embodiments of the present disclosure, the machine learning model may be a trained model.

[0024] In embodiments of the present disclosure, the vehicle may include an intelligent vehicle equipped with a machine learning model. For example, the vehicle may be an autonomous vehicle.

[0025] In some embodiments, the mentioned vehicle function may include a lane detection function for Highway Assist (HWA).

[0026] For example, more and more vehicles are equipped with the HWA function nowadays. The HWA function is mainly for the driving scenario on highways. When the vehicle is driving on a highway, the HWA function can automatically achieve lane keeping of the vehicle in the longitudinal and lateral directions, and the driver can be in a Hand Free (HF) state. If a situation that the HWA function is not designed to handle is encountered, the driver may be requested to take over the control of the vehicle to ensure safety.

[0027] It can be seen that in the HWA HF function, lane detection performance is one of the very important links. As mentioned above, currently, the evaluation of lane detection performance may mainly rely on durability tests. For example, the vehicle is made to drive on a predetermined route multiple times under various conditions (such as different weather, traffic flow, lanes, etc.). Although ideally the vehicle's driving should experience various situations, it is sometimes difficult to comprehensively cover all situations in actual tests, which results in the recorded evaluation data may not be sufficient. In addition, in order to evaluate lane detection performance, more accurate Ground Truth is usually required, and there are often a large number of errors in the data of durability tests, which need to be manually checked and corrected, consuming a large amount of cost.

[0028] Nowadays, in order to achieve better lane detection performance, the lane detection function has gradually adopted a machine learning model to implement. Since the machine learning model often needs to be pre-trained with a large amount of data, a part of the data can be separated from such data for testing the machine learning model, and reducing the dependence on subsequent durability tests will be able to greatly shorten the duration or mileage of subsequent durability tests.

[0029] Typically, in order to train a machine learning model, a large amount of data associated with vehicle functions is collected in the early stage of vehicle development. Since the typical training process for a machine learning model usually involves a training data set, a validation data set, and a test data set, the collected data can usually be divided into these three data sets accordingly. The training data set is usually used to train the machine learning model for multiple rounds, generally including about 70% of the collected data. The validation data set is usually used to optimize the machine learning model (such as fine-tuning the hyperparameters of the model), and generally includes about 10%-15% of the collected data. The test data set is usually used for performance evaluation of the machine learning model, and generally includes about 10%-15% of the collected data.

[0030] In the embodiments of this article, the test data set may include image data, radar data, and / or high-definition map data. Of course, in other implementation manners, the test data set may also include other forms of data, which are not limited herein. In some cases, if the test data set includes two or more forms of data, these data can be fused to form the test data set. For example, various applicable data fusion methods can be adopted.

[0031] The test data set in the embodiments of the present disclosure can be obtained through various appropriate methods. For example, in some implementation manners, the original data collected for vehicle functions can be obtained, and then the original data can be compressed to obtain the test data set.

[0032] For example, the original data can be collected by using at least one sensor associated with vehicle functions. For example, such sensors can include cameras (such as installed on the vehicle or outside the vehicle), radars (such as lidar), etc. The collection of the original data generally occurs before the durability test. For example, the original data can be collected by at least one sensor on a dedicated road test vehicle during the early road test of the vehicle.

[0033] The original data can include various forms of data. For example, the original data can include original image data, original radar data, and / or original high-definition map data. The original image data can be collected by a camera. The original radar data can be collected by a radar (such as installed on the vehicle or outside the vehicle). The original high-definition map data can come from map suppliers, etc.

[0034] For example, the original image data, original radar data, and / or original high-definition map data can be compressed or further sampled to obtain the test data set. Accordingly, the test data set can include any one or any combination of image data, radar data, and high-definition map data.

[0035] For example, for the original image data, assuming the original image data is 10 to 15 frames per second (i.e., the frame rate is 10 - 15 frames / second), the image data in the test dataset can be 1 frame per second (i.e., the frame rate is 1 frame / second). For the testing of machine learning models, a high frame rate is usually unnecessary; in addition, using the test dataset to evaluate the machine learning model is equivalent to open-loop perception simulation; for time-related or multi-frame-induced systematic failures, they can be compensated for by subsequent durability tests; therefore, low-frame-rate image data can meet the test requirements. The same is true for other forms of data, and the test dataset can be obtained by compression or further sampling.

[0036] Since the test dataset is usually collected in the early stage of vehicle development, through such a test dataset, a rough evaluation of the performance of the machine learning model's execution function can be obtained in the early stage of vehicle development, thereby effectively promoting the subsequent development or mass production process of the vehicle.

[0037] In some embodiments, the test dataset may include a series of time-related data. The ratio between the total duration corresponding to this series of data and the reference duration defined for the durability test may be greater than or equal to a preset ratio.

[0038] In this case, the durability test can reduce the test duration by at least this preset ratio. For example, assuming the reference duration defined for the durability test is 12,000 hours. The preset ratio can be 30%. The test dataset can include at least a series of data collected within 3,600 hours. If the test dataset includes image data and the frame rate of the image data is 1 frame / second, then the test dataset can include at least approximately 13 million frames of images. In this case, the durability test can reduce the test by at least 3,600 hours. This can also be understood as, for the entire test of this function of the vehicle, the performance evaluation achieved by the technical solution in this article can account for at least 30%, while the subsequent durability test can account for at most 70%. It can be seen that in this way, the cost of the durability test can be significantly reduced.

[0039] It should be understood that the reference duration can be different for different vehicles or different application scenarios, and only examples are given here. The preset ratio can be determined according to actual business requirements, application scenarios, etc., and this article does not limit it.

[0040] In some embodiments, Figure 1The method may further include: determining OOD data based on the test result set. In such an embodiment, generating the test result set using a machine learning model and determining OOD data based on the test result set can be accomplished by the same device. In this case, the device typically may have a large computing power. For example, the device may be a specially designed machine, or may be a computing device in the cloud (e.g., a dedicated central processing unit (CPU) in the cloud), etc.

[0041] In some embodiments, it may be assumed that Figure 1 the method is executed by a first device. In this case, Figure 1 the method may further include: outputting the test result set so that a second device different from the first device can obtain the test result set and determine OOD data based on the test result set. In such an embodiment, generating the test result set using a machine learning model and determining OOD data based on the test result set can be accomplished by different devices. For example, the first device may generate the test result set using a machine learning model, and the second device may determine OOD data based on the test result set. In this case, the first device may have a large computing power, while the second device may have a small computing power. For example, the first device may be a specially designed machine, or may be a computing device in the cloud, etc. The second device may be another device with a small computing power, etc.

[0042] OOD data can be determined in various suitable ways. In some embodiments, test results with a deviation from the expected result greater than a preset deviation threshold may be identified in the test result set. Then, the test data in the test data set corresponding to such test results can be determined as OOD data. For example, the expected result may be a label pre-added to the test data in the test data set. For instance, the test data set can be labeled in various applicable ways (e.g., manually or other automatic ways), that is, annotated. The label can serve as the ground truth. The preset deviation threshold can be determined according to actual business requirements, application scenarios, etc., and is not limited herein.

[0043] In some embodiments, if the security risk indicated by the test result corresponding to the OOD data is within a preset security risk range, the performance of the machine learning model in performing its function is acceptable. If the security risk indicated by the test result corresponding to the OOD data exceeds the preset security risk range, the performance of the machine learning model in performing its function is unacceptable. In this case, the OOD data can be used to update the machine learning model or the software module associated with the function performed by the machine learning model. For example, the OOD data can be used as further training data to update the machine learning model. In this case, only the OOD data needs to be labeled, which can also reduce the cost of labeling the training data for the machine learning model. Generally speaking, based on practical experience, the OOD data accounts for about 5% in the test dataset, and as the machine learning model becomes more and more maturely trained, the OOD data may further decrease.

[0044] The preset security risk range can be determined according to actual business requirements and application scenarios, and this is not limited in this article. For example, for the lane detection function, the preset security risk range can be characterized by the lane detection accuracy.

[0045] To facilitate understanding of the technical solutions of the present disclosure, specific examples will be described below. It should be understood that the following examples do not impose any limitation on the scope of the present disclosure.

[0046] Figure 2 is a schematic flowchart of a process for evaluating a vehicle function according to some embodiments. It should be understood that although the process of evaluating the vehicle function is generally shown in Figure 2 , each stage or step therein may be executed at the same or different devices, and this is not limited in this article.

[0047] In Figure 2 's example, the original image data is taken as an example for illustration. Additionally, assume that the vehicle function includes a lane detection function for HWAHF. Furthermore, assume that the original image data 202 is collected at 15 frames per second. To reduce the storage space of the test dataset and improve the processing efficiency, the original image data 202 can be compressed or sampled to obtain the test dataset 204. In Figure 2 's example, assume that the test dataset 204 can include image frames 1 to image frame 6. That is to say, the test dataset 204 has a frame rate of 1 frame per second. In this example, the image frames 1 to image frame 6 in the test dataset 204 can be labeled, that is, a label is added to each image frame as the ground truth for comparison with the test result.

[0048] The machine learning module 210 can be utilized to process the test data set 204 to generate a test result set 206. The test result set 206 can include test result 1 to test result 6.

[0049] Assume that the deviations between test results 2 and 3 corresponding to image frames 2 and 3 and the corresponding ground truths are greater than a preset deviation threshold (such as 15%). Then, image frames 2 and 3 can be considered as OOD data. Thus, in stage 220, based on image frames 2 and 3 and their corresponding test results 2 and 3, the performance of the machine learning model 210 in performing the lane detection function can be evaluated.

[0050] For example, if the safety risks indicated by test results 2 and 3 are within a preset safety risk range, it indicates that the performance of the machine learning model 210 in performing the lane detection function is acceptable.

[0051] If the safety risks indicated by test results 2 and 3 exceed the preset safety risk range, it indicates that the performance of the machine learning model 210 in performing the lane detection function is unacceptable. In this case, the machine learning model 210 can be updated or other relevant software modules can be updated. For example, the machine learning model 210 can be further trained using image frames 2 and 3 and the corresponding labels (i.e., ground truths), etc.

[0052] It can be seen that the evaluation method provided by the embodiments of the present disclosure can effectively serve as a supplement to the traditional durability test, thereby generally reducing the cost of the durability test and ensuring the reliability of the performance evaluation of vehicle functions.

[0053] Figure 3 is a schematic block diagram of a device for evaluating vehicle functions according to some embodiments.

[0054] As Figure 3 shown, the device 300 can include a receiving unit 302 and a model processing unit 304.

[0055] The receiving unit 302 can receive a test data set associated with the function of the vehicle. The test data set can be obtained before the durability test for the function of the vehicle.

[0056] The model processing unit 304 can utilize a machine learning model to process the test data set to generate a test result set. The machine learning model will be set in the vehicle to perform the function.

[0057] The test result set can be used to determine the OOD data in the test data set. The OOD data and the test results corresponding to the OOD data are used to evaluate the performance of the machine learning model in performing the function before the durability test.

[0058] In some embodiments, the apparatus 300 may further include a determination unit that can determine OOD data based on a set of test results.

[0059] In some embodiments, the model processing unit 304 may output a set of test results such that another apparatus different from the apparatus 300 can obtain the set of test results and generate OOD data based on the set of test results.

[0060] Each unit of the apparatus 300 may perform the specific processes described above with respect to the method embodiments. Therefore, for the sake of brevity of description, the specific operations and functions of each unit of the apparatus 300 are not described herein again.

[0061] Figure 4 is a schematic block diagram of an apparatus for evaluating vehicle functions according to some embodiments.

[0062] As Figure 4 shown, the apparatus 400 may include a processor 402, a memory 404, an input interface 406, and an output interface 408, and these modules may be coupled together via a bus 410. However, it should be understood that Figure 4 is only an example and does not limit the scope of the present disclosure. For example, in different application scenarios, the apparatus 400 may include more or fewer modules, which are not limited herein.

[0063] The memory 404 may be used to store various information related to the functions or operations of the apparatus 400 (such as the test data set, the set of test results, etc. mentioned above), executable instructions, or code, etc. For example, the memory 404 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, programmable ROM (PROM), erasable programmable ROM (EPROM), registers, hard disks, and so on.

[0064] The processor 402 may be used to execute or implement various functions or operations of the apparatus 400, such as the various operations described herein. For example, the processor 402 may execute the executable instructions or code stored in the memory 404, thereby implementing the various processes described above with respect to the method embodiments. The processor 402 may include various applicable processors, for example, a general-purpose processor (such as a CPU), a dedicated processor (such as a digital signal processor, an application-specific integrated circuit, etc.).

[0065] The input interface 406 may receive various forms of data, etc., such as the above-mentioned test data set. The output interface 408 may output various forms of data, etc., such as the set of test results, OOD data, etc.

[0066] Embodiments of the present disclosure also provide a computer-readable storage medium. The computer-readable storage medium may store executable code, and when the executable code is executed, the specific processes described above with respect to the method embodiments are implemented.

[0067] For example, the computer-readable storage medium may include, but is not limited to, RAM, ROM, electrically-erasable programmable read-only memory (EEPROM), static random access memory (SRAM), hard disk, flash memory, and the like.

[0068] Specific embodiments of the present disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0069] Not all steps and units in the above-mentioned processes and system structure diagrams are necessary, and some steps or units may be omitted according to actual needs. The device structures described in the above embodiments may be physical structures or logical structures, that is, some units may be implemented by the same physical entity, some units may be implemented separately by multiple physical entities, or may be jointly implemented by some components in multiple independent devices.

[0070] The optional embodiments of the embodiments of the present disclosure have been described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present disclosure are not limited to the specific details in the above embodiments. Within the technical concept scope of the embodiments of the present disclosure, various modifications can be made to the technical solutions of the embodiments of the present disclosure, and these modifications all fall within the protection scope of the embodiments of the present disclosure.

Claims

1. A method for evaluating vehicle functions, comprising: receiving a test data set associated with a function of a vehicle, wherein the test data set is obtained before a durability test for the function of the vehicle; processing the test data set using a machine learning model to generate a test result set, wherein the machine learning model will be set in the vehicle to perform the function, the test result set is used to determine out-of-distribution data in the test data set, and the out-of-distribution data and the test results corresponding to the out-of-distribution data are used to evaluate the performance of the machine learning model in performing the function before the durability test.

2. The method according to claim 1, wherein, the test data set includes a series of time-related data, and the ratio between the total duration corresponding to the series of data and a reference duration defined for the durability test is greater than or equal to a preset ratio.

3. The method according to claim 1, wherein, the test data set includes image data, radar data, and / or high-definition map data.

4. The method according to claim 1, wherein, the test data set is obtained by: acquiring original data collected for the function; compressing the original data to obtain the test data set.

5. The method according to claim 4, wherein, the original data is collected using at least one sensor associated with the function.

6. The method according to claim 1, wherein, the function includes a lane detection function for highway assisted driving.

7. The method according to claim 1, further comprising: determining the out-of-distribution data based on the test result set.

8. The method according to claim 1, wherein, the method is executed by a first device; the method further comprises: outputting the test result set so that a second device different from the first device can obtain the test result set and determine the out-of-distribution data based on the test result set.

9. The method according to claim 1, wherein, the out-of-distribution data is determined by: identifying test results in the test result set with a deviation from the expected result greater than a preset deviation threshold; determining the test data in the test data set corresponding to the test results as the out-of-distribution data.

10. The method according to claim 9, wherein, the expected result is a label pre-added to the test data in the test data set.

11. The method according to claim 1, wherein, if the safety risk indicated by the test result corresponding to the out-of-distribution data is within a preset risk range, the performance of the machine learning model in performing the function is acceptable; if the test result corresponding to the out-of-distribution data exceeds the preset risk range, the performance of the machine learning model in performing the function is unacceptable, and the out-of-distribution data is used to update the machine learning model or a software module associated with the machine learning model in performing the function.

12. An apparatus for evaluating the execution performance of vehicle functions, comprising: A receiving unit configured to receive a test data set associated with a function of a vehicle, wherein the test data set is obtained before a durability test for the function of the vehicle; A model processing unit configured to process the test data set using a machine learning model to generate a test result set, wherein the machine learning model will be set in the vehicle to perform the function, the test result set is used to determine out-of-distribution data in the test data set, and the out-of-distribution data and the test results corresponding to the out-of-distribution data are used to evaluate the performance of the machine learning model in performing the function before the durability test.

13. The apparatus according to claim 12, further comprising: A determining unit configured to determine the out-of-distribution data based on the test result set.

14. The apparatus according to claim 12, wherein, the model processing unit is further configured to output the test result set so that another apparatus different from the apparatus can obtain the test result set and determine the out-of-distribution data based on the test result set.

15. An apparatus for evaluating the execution performance of a vehicle function, comprising: At least one processor; A memory in communication with the at least one processor, having executable code stored thereon, the executable code, when executed by the at least one processor, causes the at least one processor to execute the method according to any one of claims 1 to 11.

16. A computer-readable storage medium storing executable code that, when executed, implements the method according to any one of claims 1 to 11.

Images