A method for testing a new software version of an electronic component of a motor vehicle by an electronic computing device, computer program product,

Abstract

The invention relates to a method for testing a new software version (12) of an electronic component of a motor vehicle by an electronic computing device (10), comprising the steps of: providing the

Description

[0001] The present invention relates to the field of automobiles. More specifically, the present invention relates to a method for testing a new software version of an electronic component of a motor vehicle by an electronic computing device according to the pending claim 1. Furthermore, the present invention relates to a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as to a corresponding electronic computing device. BACKGROUND INFORMATION

[0002] The so-called “canary release” pattern is a deployment approach that releases a new version of some software, and routes a very small amount of traffic to it prior to promoting that release fully to production. In solutions already in practice, that traffic can be a randomly selected one percentage, or it can be chosen based on attributes of the request such as what region it is originating from, or furthermore.

[0003] Canary releases either just hope for that small one percent to cover enough cases to find problems, or require teams to do work guessing that attributes are relevant for their changes. SUMMARY OF THE INVENTION

[0004] It is an object of the present invention to provide a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding electronic computing device, by which a test of a new software version can be provided in an efficient way.

[0005] This object is solved by a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding electronic computing device according to the independent claims. Advantageous embodiments are presented in the dependent claims.

[0006] One aspect of the invention relates to a method for testing a new software version of an electronic component of a motor vehicle by an electronic computing device. The new software version is provided by the electronic computing device. Different errors, which occurred in the older software versions, are identified by the electronic computing device. The errors are clustered and at least one attribute is identified, which is associated with the identified errors, by the electronic computing device. Data, which is associated with the at least one attribute is provided by the electronic computing device. The new software version is tested depending on the provided data by the electronic computing device.

[0007] In particular, the provided method replaces a manual attribute selection with for example a machine learning algorithm, that uses the data in the monitoring system and correlates all the possible attributes of the request with past alerts and incidents / errors. It may also correlate attributes with an analysis of the code changes involved in the release.

[0008] Once the algorithm has determined the relevant attributes, it dynamically adjusts the incoming traffic so that the minimum amount of requests to cover those attributes are sent to the new software version.

[0009] According to an embodiment the errors are identified by using a database of a plurality of motor vehicles.

[0010] In another embodiment a machine learning algorithm is used for providing the data.

[0011] In another embodiment testing the new software version is provided as a canary release test.

[0012] In another embodiment one percentage of data, which can be potentially tested, is provided as the data.

[0013] In another embodiment a nearest neighbor algorithm is used for clustering the errors.

[0014] In particular, the method is a computer-implemented method. Therefore, another aspect of the invention relates to a computer program product comprising program code means for performing a method according to the preceding aspect.

[0015] Furthermore, the present invention relates to a non-transitory computer-readable storage medium comprising at least the computer program product according to the preceding aspect.

[0016] Furthermore, the present invention relates to an electronic computing device for testing a new software version of an electronic component of a motor vehicle, wherein the electronic computing device is configured for performing a method according to the preceding aspect. In particular, the method is performed by the electronic computing device.

[0017] A computing unit / electronic computing device may in particular be understood as a data processing device, which comprises processing circuitry. The computing unit can therefore in particular process data to perform computing operations. This may also include operations to perform indexed accesses to a data structure, for example a look-up table, LUT.

[0018] In particular, the computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units.

[0019] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0020] A memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a non-volatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0021] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figures and / or shown in the figures alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0023] The drawings show in:

[0024] Fig. 1 a schematic flow chart according to an embodiment of a method; and

[0025] Fig. 2 another schematic flow chart according to another embodiment of the method.

[0026] In the figures the same elements or elements having the same function are indicated by the same reference signs. DETAILED DESCRIPTION

[0027] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0028] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0029] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0030] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0031] Fig. 1 shows a schematic flow chart according to an embodiment of a method. In particular the method is performed by an electronic computing device 10 (Fig. 2). According to the shown embodiment of Fig. 1, a method for testing a new software version 12 (Fig. 2) of an electronic component of a motor vehicle is provided by the electronic computing device 10.

[0032] In a first step S1 the new software version 12 is provided by the electronic computing device 10. In a second step S2 different errors, which occurred in an old version 14 of the software version, are provided by the electronic computing device 10. In a third step S3 the errors are clustered and in a fourth step S4 identifying of at least one attribute, which is associated to the identified and clustered errors, is provided by the electronic computing device. Data 16 (Fig. 2) which is associated to the at least one attribute, is provided by the electronic computing device in a fifth step S5. In a sixth step S6 the new software version 12 is tested depending on the data 16 provided by the electronic computing device 10.

[0033] Fig. 2 shows another schematic flow chart according to an embodiment of the method. In particular Fig. 2 shows a data gathering step 18 from previous releases as well as a behavior testing 20 with the new software version 12.

[0034] Fig. 2 shows a plurality of motor vehicles 22, which are relevant for the data gathering 18 as well as the behavior after the release 20. Furthermore, telemetry applications 24 are shown. The telemetry applications 24 are transmitted to car service monitors 26 and to a car attribute database 28. From the car service monitors 26 at least one behavior 30 is determined and from the car attribute database 28 at least one attribute 32 is determined. Furthermore, behaviors to trigger for new release 34 are shown. The behaviors 30, the attributes 32, as well as the behaviors to trigger for a new release 34 are used by a release analyzer 36. The release analyzer 36 determines attributes 32, behaviors 30 and a combination based on a group of motor vehicles, which is shown with a block 38. According to this block 38 the data 16 is provided as input data for a so-called big bird release controller 40. The big bird release controller 40 provides requests with the data 16 to the existing version 14 as well as to the new software version 12.

[0035] According to the shown embodiment the electronic computing device 10 may replace the manual attribute selection with a machine learning algorithm that uses the data 16 in the monitoring system and correlates all the possible attributes 32 of the request with part alerts and incidents. It may also correlate attributes 32 with an analysis of the code changes involved in the release.

[0036] Once the algorithm has determined the relevant attributes 32, it dynamically adjusts the incoming traffic so that the minimum amount of requests to cover those attributes are sent to the new software version 12.

[0037] For example, the new software version 12 for an in-motor-vehicle backend server is released. The algorithm discovers that of the dozens of ways to “divide up” the plurality of motor vehicles 22, there have been incidents relating to about ten values: Model, year, in-car component version, and country of origin. It may identify that there are three values for model, five different years, four different in-car component versions, and about 15 relevant countries of origin. Combined together (15 multiplied by 4 multiplied by 5 multiplied by 3), this defines 900 unique requests.

[0038] The cluster component then uses this data to give preference to a higher amount of requests to the new software version 12 that are unique within the 900 requests. Rather than random one percentage, including unusual cases, such as a new model with a rare component in for example South Korea is provided. Rather than defining these 900 options manually, it is done automatically based on past data.

[0039] These attributes 32 are available by just looking at the request data itself. For example by headers, payload data, payload size, and originated IP address. These elements hide a lot of complexity. For example, headers for a map application may include two user-agent (browser) client OS end version, and furthermore.

[0040] Payload data is intended to mean the literal (unencrypted) payload. This means, if a post request has associated payload, each one can also be considered. The goal of the release analyzer 36 may be to find a correlation between one of these elements and errors in the past. For example, if requests that include “overwrite subject” have often failed with a 500 error in the past due to some bug, then the analyzer's goal is to find the request automatically, and include some of these requests to the big bird release controller 40.

[0041] The release analyzer 36 is not limited to the attributes 32 that arrive over the network request. The implementation may also map some of those attributes 32 that are identifiers to the additional data they represent. For example, a car identification (VIN) implies a lot of additional data such as model number, component IDs, for example head unit version, wireless radio version, and furthermore. The release analyzer 36 has access to this data as well, for example via some ITL process, stream processing, and furthermore, in particular the mechanism is more of an implementation detail.

[0042] These kinds of releases define a set of desired attributes 32 to cover in a release, and rely on the electronic computing device 10 to select these attributes 32 at runtime. These examples do not make use of the behavior results.

[0043] For example, a release wants to target model, year, in-car component version and country of origin. The electronic computing device 10 identifies that there are three values for model, five different years, four different in-car component versions, and about 15 relevant countries of origin. Therefore, this defines 900 unique requests.

[0044] Instead of routing a random one percentage, the electronic computing device 10 routes requests to “fill up” these 900 unique requests. As a particular set of attributes 32 is seen, and routed to the cluster, further requests of that are no longer routed.

[0045] For example, the controller maintains a table of all possible attributes 32, and a threshold. If this setup is too large, approaches such as All Pairs testing could be used to limit the number of desired, unique requests.

[0046] Instead of manually choosing the attributes 32 for release, the electronic computing device 10 can use an algorithm such as nearest neighbor clustering to identify clusters of requests, and choose attributes 32 of request that provide broad coverage across the neighbors respective of frequency.

[0047] For example, a system with the same attributes as above is covered but instead of choosing a constant set of requests across those attributes 32, the nearest neighbor analysis is used to choose requests whose attributes placed them in distant clusters. Essentially this version “checks off” clusters instead of individual sets of attributes 32.

[0048] These releases define a target set of behaviors 30, and rely on the electronic computing device 10 to choose the smallest set of attributes 32 that may trigger that behavior 30. These behaviors 30 may be failures that the team may want to be sure that does not happen or they may be success cases that are otherwise difficult to test.

[0049] For example, a team is releasing a new version of the “get nearest gas stations” end point, and wants to be sure that request that are likely to trigger 500 errors are routed to the canary, even if they are very uncommon. The electronic computing device 10 reviews a data set of past calls, their attributes 32 and their results. Instead of routing a random one percentage of traffic, the electronic computing device 10 just requests with attributes 32 that are correlated with target behavior 30.

[0050] The gas station team may realize that some failures are correlated with details implied by the request but not specified in it. For example, certain car models seem to generate more 500 errors. These requests include a car identifier but not a model or other data.

[0051] The electronic computing device 10 may be configured to integrate with third-party data sources, recording these attributes on request, and using them to route future requests during a release.

[0052] In this example, the electronic computing device 10 is integrated with a vehicle database that can take the vehicle identification in the request and turn it into other attributes 32, including car model. During regular operation, these attributes 32 are recorded alongside the others. At release time, these attributes 32 are also analyzed for correlations to the target behaviors 30. Also at release time, this integration is a leverage to determine if an incoming request is a candidate for the new software version 12. Signs electronic computing device new software version old software version data data gathering behavior release plurality of motor vehicles telemetry applications car service monitors car attributed database behaviors attributes behaviors to trigger for new release release analyzer block big bird release controller requests steps of the method

Claims

1. A method for testing a new software version (12) of an electronic component of a motor vehicle by an electronic computing device (10), comprising the steps of: - providing the new software version (12) by the electronic computing device (10); (S1)- identifying different errors, which occurred in an old software version (14), by the electronic computing device (10); (S2)- clustering the errors and identifying at least one attribute (32), which is associated with the identified and clustered errors, by the electronic computing device (10); (S3, S4)- providing data (16), which is associated with the at least one attribute (32), by the electronic computing device (10); (S5) and- testing the new software version (12) depending on the data (16) provided by the electronic computing device (10). (S6)2. The method according to claim 1, characterized in thatthe errors are identified by using a database of a plurality of motor vehicles (22).

3. The method according to claim 1 or 2, characterized in thata machine learning algorithm is used for providing the data (16).

4. The method according to any one of claims 1 to 3, characterized in thattesting the new software version (12) is provided as a canary release test.

5. The method according to claim 4, characterized in thatone percentage of data, which can be potentially tested, is provided as the data (16).

6. The method according to any one of claims 1 to 5, characterized in thata nearest neighbor algorithm is used for clustering the errors.

7. A computer program product comprising program code means for performing a method according to any one of claim 1 to 6.

8. A non-transitory computer-readable storage medium comprising at least the computer program product according to claim 7.

9. An electronic computing device (10) for testing a new software version (12) of an electronic component of a motor vehicle, wherein the electronic computing device (10) is configured for performing a method according to any one of claims 1 to 6.

Images