Method for verifying and permitting predictive program, in particular on-board predictive program, for analyzing vehicle data

Abstract

The invention relates to a method for verifying and permitting a predictive program, in particular an on-board predictive program, for vehicle data analysis of a new generation of vehicles, comprising the following steps: creating a predictive program for creating a predictive on the basis of vehicle data; providing a first data record having real vehicle data acquired by one or more new generation vehicles; providing a second data record having real vehicle data acquired by a plurality of old generation vehicles; providing a third data record having simulated vehicle data generated by a computer simulation of a new generation of vehicle; creating a prediction with respect to the specific event based on the first data record by means of the prediction program; creating a prediction with respect to the specific event on the basis of the second data record by means of the prediction program; creating a prediction with respect to the particular event on the basis of the third data record by means of the prediction program; comparing a prediction based on the first data record with a prediction based on the second data record and the third data record; when the prediction based on the first data record varies within a predetermined limit around the prediction based on the second data record and the third data record, the validity of the prediction program is confirmed and the prediction program is permitted.

Description

Technical Field

[0001] This invention relates to a method for verifying and licensing a predictive program, particularly an in-vehicle predictive program, for vehicle data analysis used in next-generation vehicles. Background Technology

[0002] Modern vehicles are equipped with high-performance onboard computers that can perform a variety of completely different assistance functions. On the one hand, this makes driving not only more comfortable and convenient, but also significantly safer. On the other hand, it makes the maintenance process more efficient and environmentally friendly, for example, by not simply performing maintenance according to a specific mileage, but only when it seems reasonable based on the actual vehicle condition derived, especially from the user's personalized driving behavior.

[0003] The corresponding auxiliary functions typically use so-called "machine learning models," hereinafter referred to as "predictors" based on their function. These predictors can analyze vehicle data—often supplemented by other data—recorded by the vehicle itself using corresponding sensors, particularly on-board, but also elsewhere, such as in the cloud, to predict specific events, such as the wear and tear of specific components. These predictions can be relevant at entirely different levels, such as for vehicle operation or maintenance.

[0004] The predictive programs of this type are generated using so-called "machine learning algorithms" and corresponding training data, and the more real vehicle data provided for training, the better the predictive program generally performs, i.e., the more accurate its predictions. Because modern vehicles typically collect large amounts of vehicle data and transmit it to the vehicle manufacturer, with customer permission, wirelessly or via wired means (e.g., during maintenance visits), the manufacturer can collect vast amounts of data about specific generations of vehicles and their behavior (so-called fleet data), which enables the creation of highly accurate predictive programs.

[0005] Before such predictive programs can be implemented, they must be validated—that is, their predictive reliability must be examined—and their use must be approved. This is because, depending on the type, these predictive programs not only involve safety-related aspects but can also significantly contribute to customer satisfaction and brand loyalty. When, for example, a program that operates accurately prompts a customer to bring their vehicle in for repairs, and those repairs are subsequently proven to be justified, the customer is usually very impressed with the reliability of their vehicle. Summary of the Invention

[0006] When a new generation of vehicles is launched into the market, the following problems arise: on the one hand, there is a lack of fleet data, i.e., a large record of actual daily behavior of the new vehicles, because only a small number of new-generation vehicles are put into trial operation; on the other hand, customers should be provided with as many assistance functions as possible. For assistance functions that use software with few changes (such as radio control), while corresponding licensing processes have been established, this is not the case for predictive programs that are frequently optimized and retrained based on new data.

[0007] Based on this, the objective of this invention is to propose a method for verifying and licensing prediction programs, particularly in-vehicle prediction programs, for vehicle data analysis of next-generation vehicles. This method can verify the effectiveness of prediction programs in a standardized, understandable, and uniform manner, and, where possible, license the prediction programs, even in the absence of comprehensive fleet data.

[0008] This task is solved by the method having the features of claim 1. Advantageous designs and improvements are the subject of the dependent claims. Co-claim 13 relates to a computer program product for performing specific steps of the method according to the invention.

[0009] This task is addressed in particular by a method for validating and licensing a predictive program, especially an in-vehicle predictive program, for vehicle data analysis of the first group of vehicles, particularly the next generation of vehicles, wherein the method includes the following steps:

[0010] Create a forecasting program to generate predictions based on vehicle data;

[0011] Provide a first data record, which has real vehicle data collected by one or more next-generation vehicles;

[0012] Provide a second data record containing real vehicle data collected from multiple second groups of vehicles, especially older generation vehicles;

[0013] Provide a third data record containing simulated vehicle data generated by computer simulation of the first group of vehicles;

[0014] The prediction program creates a prediction about the specific event based on the first data record.

[0015] The prediction program creates a prediction about the specific event based on the second data record.

[0016] The prediction program creates a prediction about the specific event based on the third data record.

[0017] The prediction based on the first data record is compared with the prediction based on the second data record and the third data record;

[0018] When a prediction based on the first data record varies within predetermined limits around a prediction based on the second and third data records, the validity of the prediction program is confirmed and the prediction program is authorized.

[0019] This invention offers the advantage of enabling reliable verification and licensing of predictive programs that perform vehicle data analysis, specifically targeting vehicles whose daily behavior is relatively unknown due to its novelty. Here, it is noted that the term "predictive program" is understood to refer to all types of machine learning models that allow for inference at the vehicle's onboard location—that is, predictions of specific events based on data ultimately collected from the vehicle side—often supplemented by other pre-stored data within the vehicle.

[0020] Here, predictions and events can be entirely different types and are used for both comfort and safety. For example, anticipated wear and tear on specific vehicle components, such as the starter battery, can be predicted, prompting the corresponding customer to bring the vehicle in for maintenance within a specific time or mileage range. With the customer's permission, the vehicle can then communicate directly with the repair shop and / or manufacturer to, for example, announce a scheduled maintenance visit, reserve a specific part, or check its availability. This invention can be advantageously used herein for entirely different machine learning models.

[0021] In the method according to the invention for verifying and licensing a prediction program, particularly an in-vehicle prediction program, for analyzing vehicle data for next-generation vehicles, a prediction program is first created to generate predictions based on vehicle data, more specifically, typically using a learning algorithm and training data records. This practice is well known to those skilled in the art of artificial intelligence and self-learning programs. The specific learning algorithm used is irrelevant to this invention.

[0022] Three data records are then provided: a first data record with real vehicle data from the test fleet (collected by one or more new-generation vehicles); a second data record with real vehicle data collected from multiple older-generation vehicles (e.g., diagnostic data collected during factory maintenance); and a third data record with simulated vehicle data generated by computer simulation of the new-generation vehicles. Here, the terms "new-generation vehicle" and "older-generation vehicle" are understood in a context-specific manner, rather than in their traditional sense, where "new-generation" refers to an improved or at least visually altered version of an existing model. More precisely, "new-generation vehicle" can be understood as all vehicles, regardless of their form—e.g., modified or retrofitted, or entirely newly manufactured—that are not yet in large fleets and for which there is no substantial data record of their behavior. Older-generation vehicles can, in some cases, be entirely different types of vehicles for which fleet data is available and which share specific commonalities with the new-generation vehicles, making their expected behavior similar in certain aspects.

[0023] The prediction program then creates a prediction about the specific event, more precisely, once based on a first data record, once based on a second data record, and once based on a third data record. Here, the term "event" may be defined specifically for the prediction program. If, for example, a maintenance issue is involved, an event could be the condition of a specific component under specific mileage, specific driving behavior, specific (past) environmental conditions, specific material aging, etc.

[0024] The prediction based on the first data record is then compared with the prediction based on the second and third data records. This can advantageously be done by calculating a performance metric for each prediction and comparing these metrics directly or indirectly. An indirect comparison can be performed, for example, by first determining a comparison value from the performance metrics of the predictions based on the second and third data records, and then comparing the performance metric of the prediction based on the first data record with that comparison value.

[0025] The prediction program is permitted when the prediction based on the first data record varies within predetermined limits around the predictions based on the second and third data records. If the prediction program is not permitted, an error notification can be generated indicating that the prediction program still needs further optimization, such as training with new or supplementary training data.

[0026] Licensing can also be carried out as follows: first, the prediction program is licensed only partially, i.e., only for a predetermined portion of the new generation of vehicles, which is then delivered and additional data can be collected during routine operation. This real-vehicle data can then advantageously supplement the first data record and be used to validate and license other versions of the prediction program.

[0027] Further details and advantages of the invention will become apparent from the following purely exemplary and non-limiting description of the embodiments in conjunction with the accompanying drawings. Attached Figure Description

[0028] Figure 1 A flowchart of the method according to the present invention is shown schematically. Detailed Implementation

[0029] exist Figure 1 The diagram schematically illustrates a process for a predictive procedure for analyzing vehicle data for next-generation vehicles, and hereby describes a method for validating and licensing the in-vehicle predictive procedure. Here, a machine learning model, referred to herein as predictive procedure 12, is first created using training data and machine learning algorithm 10.

[0030] Three data records 14, 16, and 18 are provided to the prediction program 12. The first data record 14 contains real vehicle data collected by one or more next-generation vehicles (a so-called test fleet), the second data record 16 contains real vehicle data collected by multiple older-generation vehicles, and the third data record contains simulated vehicle data generated by computer simulation of the next-generation vehicles. These data records can each contain substantial amounts of data about the vehicles and their corresponding driving histories, which the prediction program can use in whole or in part. Some data may be irrelevant for making predictions about a specific event. In practice, the amount of data in the second data record regarding the specific event can be between 10¹ and 10⁻⁶. 5 The multiple between these values ​​is greater than the amount of data about the existence of the specific event in the first data record, and the amount of data about the existence of the specific event in the third data record can be between 10¹ and 10¹. 5 The multiple between them is greater than the amount of data in the second data record regarding the specific event.

[0031] The prediction program 12 then creates three predictions about the specific event: prediction 20 based on a first data record, prediction 22 based on a second data record, and prediction 24 based on a third data record. Technically, these predictions involve inference, specifically, inferences drawn by a machine learning model from given data records. Figure 1In the example shown, the creation of predictions based on the second and third data records is performed outside the vehicle, while the creation of predictions based on the first data record is performed on-board in the next-generation vehicle. Predictions created on-board can be wirelessly transmitted to an external computing center, where they are automatically compared with predictions based on the second and third data records.

[0032] In the illustrated embodiment, performance metrics 26, 28, and 30, in the form of so-called Key Performance Indicators (KPIs), are calculated for each prediction 20, 22, and 24. These performance metrics are used to evaluate the quality of the corresponding predictions 20, 22, and 24, and the basic calculations of such performance metrics 26, 28, and 30 are well known to those skilled in the art. Which performance metric is used in a particular case depends on the type of event. For example, if traffic signs in an image need to be identified, an metric could be the percentage of signs correctly identified. This invention advantageously allows those skilled in the art to select optimized performance metrics for the corresponding application.

[0033] In the illustrated embodiment, in the next step, performance metrics 28 and 30 based on predictions 22 and 24 of the second data record 16 and the third data record 18 are compared with each other, and an average value, also referred to as a “benchmark”, is formed, hereinafter referred to as comparison value 32. In step 34, it is checked whether performance metrics 28 and 30 deviate from each other by more than a predetermined application-related and user-defined amount, such as 10%, or whether, according to the implementation, the performance metrics deviate from the formed comparison value 32 by more than a predetermined application-related and user-defined amount, such as 10%. If so, the prediction program 12 is provided to the machine learning algorithm 10 for further optimization (retraining), as indicated by arrow 36. If the deviation varies within a predetermined limit, in step 38, the comparison value 32 formed by performance metrics 28 and 30 is compared with performance metric 26 based on the first data record 14. Based on this comparison, the prediction program 12 is subsequently evaluated as valid and granted permission (arrow 42) in step 40, or provided to the machine learning algorithm 10 for retraining, as indicated by arrow 44. The criteria used for permission also depend on the specific application. In the example of traffic sign recognition, it can be specified that permission is granted only when there is only a small difference between the average value and performance index 26, while higher tolerance limits can be set in other cases. Permission can be granted automatically through a parent program instance or manually through a human decision-maker.

[0034] The approved prediction program 12 can then be loaded initially only onto a predetermined portion of the next-generation vehicles, as shown in step 46, which can collect additional data during routine operation. This real-world vehicle data can advantageously supplement the first data record, as shown by arrow 48, and be used to validate and approve other versions of the prediction program.

[0035] A concrete application example is the monitoring of worn parts in a vehicle. Based on data points from driving behavior / use, external conditions, and expected material aging, the prediction program 12 can estimate how high the wear level is for an individual component. For example, if the starter battery is involved, a notification is generated at a specific wear level indicating that the battery should be replaced, and this is displayed, for example, via a user app on a mobile phone and / or within the vehicle itself.

[0036] When a fleet of vehicles equipped with prediction program 12 is present, prediction program 12 can estimate the state of the starter battery based on different vehicle data (prediction 20). The same prediction program 12 is applied to both test data and diagnostic data, determined for example in a factory repair scenario (prediction 22), and to simulation data from a laboratory (prediction 24). The data types and questions are the same in all three inferences. Performance metrics 26, 28, and 30 calculated by the predictions are compared to each other as described above. If performance metrics 28 and 30 deviate from each other by more than a predetermined first limit, for example, 10%, prediction program 12 is retrained. If performance metrics 28 and 30 deviate from each other equal to or less than the first limit, i.e., 10%, the average value formed by the two performance metrics 28 and 30 is compared to performance metric 26. If the deviation is greater than a second predetermined limit, which may be equal to the first limit, i.e., 10%, prediction program 12 is retrained. If the deviation is equal to or less than the second limit, the prediction procedure is permitted, or more precisely, preferably initially only for the larger test fleet. This process is then repeated iteratively until prediction procedure 12 is deployed to the entire vehicle fleet. In each iteration, the first and second limits for the deviation can be redefined, typically smaller (i.e., 5% instead of 10% in the second iteration).

[0037] This invention has been described using the verification and licensing of an in-vehicle predictive program for performing vehicle data analysis as an example. However, this invention can also be used to verify and license predictive programs for performing vehicle data analysis, where the predictive program does not run on a vehicle, but rather, for example, on a cloud-based basis.

Claims

1. A method for validating and licensing a predictive program, particularly an in-vehicle predictive program, for vehicle data analysis used in next-generation vehicles, the method comprising the following steps: Create a prediction program (12) to create predictions based on vehicle data; Provide a first data record (14) having real vehicle data collected by one or more next-generation vehicles; Provide a second data record (16) containing real vehicle data collected from multiple older generation vehicles; Provide a third data record (18) containing simulated vehicle data generated by computer simulation of a new generation of vehicles; Using the prediction program, a prediction about the specific event is created based on the first data record (20); The prediction program is used to create a prediction about the specific event based on the second data record (22); The prediction program is used to create a prediction about the specific event based on the third data record (24). The prediction (20) based on the first data record is compared with the predictions (22, 24) based on the second data record and the third data record; When the prediction (20) based on the first data record changes within a predetermined limit around the predictions (22, 24) based on the second data record and the third data record, the validity of the prediction procedure (12) is confirmed and the prediction procedure is authorized (40).

2. The method according to claim 1, characterized in that, To compare the predictions (20, 22, 24) with each other, a performance metric (26, 28, 30) is calculated for each prediction, and the performance metrics are compared with each other directly or indirectly.

3. The method according to claim 2, characterized in that, To compare the predictions (20, 22, 24) with each other, a comparison value (32) is determined by the performance metrics (28, 30) of the predictions (22, 24) based on the second data record and the third data record (16, 18), and the performance metric (26) of the prediction based on the first data record is compared with the comparison value (32).

4. The method according to any one of claims 1 to 3, characterized in that, The amount of data in the second data record (16) regarding the existence of the specific event is between 10¹ and 10¹. 5 The multiple between them is greater than the amount of data about the specific event in the first data record (14).

5. The method according to any one of claims 1 to 4, characterized in that, The amount of data in the third data record (18) regarding the existence of the specific event is between 10¹ and 10¹. 5 The multiple between them is greater than the amount of data about the specific event in the second data record (16).

6. The method according to any one of claims 1 to 5, characterized in that, Create predictions (22, 24) based on the second and third data records outside the vehicle.

7. The method according to any one of claims 1 to 6, characterized in that, Create predictions based on the first data record on the vehicle of the next generation of vehicles (20).

8. The method according to claim 7, characterized in that, The prediction (20) created on the vehicle’s in-vehicle device is wirelessly transmitted to a computing center outside the vehicle and automatically compared with the prediction (22, 24) based on the second data record and the third data record at the computing center.

9. The method according to any one of claims 1 to 8, further comprising the following step: An error notification is generated when the prediction (20) based on the first data record (14) does not change within a predetermined limit around the prediction (22, 24) based on the second data record and the third data record (16, 18).

10. The method according to any one of claims 1 to 9, wherein, The prediction program (12) is created using a learning algorithm and training data records.

11. The method according to claim 10, wherein, If the validity of the prediction procedure (12) is not confirmed and is not permitted, the prediction procedure is re-inputted into the learning algorithm.

12. The method according to any one of claims 1 to 11, wherein, The licensing (42) of the prediction program includes the step of partially licensing the prediction program (12) only for a predetermined portion of the next generation of vehicles.

13. The method according to claim 12, wherein, After partially licensing the prediction program (12), the first data record (14) is supplemented with real vehicle data of a new generation of vehicles using the partially licensed prediction program (12).

14. A computer program product for performing at least the following steps in the method according to any one of claims 1 to 13: The prediction (20) based on the first data record (14) is compared with the predictions (22, 24) based on the second data record and the third data records (16, 18); and Confirm the effectiveness of the prediction procedure (12).

Images