COMPUTER IMPLEMENTED METHOD, COMPUTER PROGRAM, AND APPARATUS FOR SENSOR GENERATING DATA-BASED MODEL COPIES - Patent application

Abstract

An apparatus, computer program, and computer-implemented method for generating a model copy based on data at a first sensor, the method comprising: converting (202) predetermined raw data (204) from a first sensor into data (206) representative of the raw data of a second sensor; determining (208) a first outcome (210) using the predetermined raw data and a first model designed to predict an outcome based on the raw data from the first sensor; and converting (208) the predetermined raw data into data (206) representative of the raw data of the second sensor and a first model designed to predict an outcome based on the raw data from the second sensor. and determining (212) a second outcome (214) using a predetermined second model designed to estimate the first outcome (210) and determining (216) whether the first outcome (210) differs from the second outcome (214), wherein if the first outcome differs from the second outcome, the method comprises determining (218) training data points comprising the predetermined raw data (204) and the second outcome (214) and training (220) the first model using training data comprising the training data points.

Description

[Technical field]

[0001] The present invention relates to a computer-implemented method, computer program product, and apparatus for sensor-based generation of model copies based on data. [Background technology]

[0002] Models are used in sensors, particularly for processing sensor data. Training a model based on data is very complex and requires a large amount of training data and training iterations. A model trained for a particular sensor cannot necessarily be used for another sensor. Summary of the Invention [Problem to be solved by the invention]

[0003] It is therefore desirable to automatically generate a copy of a model based on data from one sensor to use with another sensor for which a trained model already exists. [Means for solving the problem]

[0004] This is made possible by a computer implemented method, a computer program and an apparatus as set out in the independent claims. A computer-implemented method for generating a data-based model copy includes converting predetermined raw data from a first sensor into data representative of raw data of a second sensor, determining a first result using the predetermined raw data and a first model designed to predict an outcome based on the raw data from the first sensor, determining a second result using data representative of the raw data of the second sensor and a predetermined second model designed to predict an outcome based on the raw data from the second sensor, and determining whether the first result differs from the second result, and if the first result differs from the second result, the method includes determining training data points including the predetermined raw data and the second result, and training the first model using training data including the training data points. The method is capable of recognizing relevant data in the operation itself based on a discrepancy in the results between the first new model and the old second model, and using the relevant data in the data-based model copy.

[0005] The raw data preferably represents at least one spectrum of at least one time domain signal, in particular at least one position of a radar, LiDAR, ultrasonic, infrared or acoustic sensor, or filtered data or a transform thereof.

[0006] The first result and / or the second result preferably characterize an estimate for the object type, or dimensions of the object, or indicate whether a blind sensor, clustering, or object has been recognized.

[0007] It may be contemplated that the first model is trained in multiple iterations, and the values ​​of the parameters defining the first model are initialized with random values ​​before the first iteration, thereby providing a starting point for training that is less resource-intensive.

[0008] It may be contemplated that the training is performed in multiple iterations, and values ​​of the parameters defining the first model are or have been determined prior to the first iteration by training with raw data measured by the first sensor, thereby providing a pre-trained first model that is further refined by the training.

[0009] It may be contemplated that the training is performed in such a way that a large number of training data points are determined in multiple iterations without performing a training step, and then a training step is performed, in particular on the first sensor or on a computing device external to the first sensor, in which the parameters defining the first model are determined using some training data points from the large number of training data points or using training data points from the large number of training data points. The training data points are, for example, collected in memory. This allows the training to scale particularly well in terms of available memory.

[0010] For initialization, the method includes a step of determining a structure of the first model depending on a predetermined structure of the second model. It may be provided that the determination of the structure of the first model includes an architecture search using a machine learning system, in which the structure of the first model is determined according to a predetermined structure of the second model, which allows the first model to be automatically adapted to the respective first sensor and / or second sensor.

[0011] It may be contemplated that determining the structure of the first model comprises copying at least a part of the predetermined structure of the second model into the structure of the first model and / or copying at least some values ​​of predetermined parameters of the second model into values ​​for parameters of the first model, such that the first model for the first sensor is particularly well adapted to the second model for the second sensor when these sensors have only minor differences.

[0012] Preferably, the first model is transmitted to a computing unit of the first sensor after at least one training step during which parameters defining the first model are determined, the computing unit being designed to predict results using the first model with respect to raw data measured by the first sensor, thereby providing a first model at the first sensor after training.

[0013] The method may be arranged such that, at a specifiable or specified time point, preferably at regular time intervals, the first model is transmitted from the calculation unit to a computing device outside the first sensor, a third model is determined according to the first model and at least one other model, and the first model at the first sensor is replaced by the third model. For example, the local first model of the first sensor is fused with the local models of various sensors at a global level and distributed. The advantage of this approach is that each local model is significantly smaller than the sum of the training data, so that communication requirements are significantly reduced.

[0014] It may be envisaged that if the first result differs from the second result, the method comprises the steps of transmitting the training data points to a computing device external to the first sensor, determining a third result for the training data points using another model specifically designed to predict the third result for the training data points, determining modified training data points by replacing the second result for the training data points with the third result, transmitting the modified training data points to the first sensor and training the first model with the modified training data points, whereby the first model is further trained with the modified training data points determined external to the first sensor.

[0015] The method may be performed such that the second result on the training data points is checked whether it is correct or incorrect, and if the second result is incorrect, modified training data points are determined and used for training the first model, and otherwise modified training data points are not determined and / or are not used for training the first model. This allows misjudgments of the second model to be recognized and corrected. Thus, the first model is trained using correct results and not incorrect results.

[0016] Further advantageous embodiments will become apparent from the following description and the drawings. [Brief description of the drawings]

[0017] [Figure 1] FIG. 2 is a schematic diagram of a portion of an apparatus for generating a model copy based on data. [Diagram 2] FIG. 2 illustrates steps in a method for generating a model copy based on data. [Diagram 3] FIG. 13 illustrates a further step in the method for generating a model copy based on data. [Figure 4] FIG. 13 illustrates a further step in the method for generating a model copy based on data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] An apparatus 100 for generating a model copy based on data is shown diagrammatically in Fig. 1. The apparatus 100 comprises at least one processor 102 and at least one memory 104, hereinafter referred to as computational units.

[0019] 1 shows a first sensor 106. The first sensor 106 includes a computation unit in this example. The at least one memory 104 stores a first model 108 and a predefined second model 110. The first model 108 is designed to predict an outcome based on raw data from a first sensor 106. The second model 110 is designed to predict an outcome based on raw data from a second sensor. The second model 110 is known in this example and is specifically designed for this purpose by pre-training. The second model 110 is adapted to the second sensor in this example. The second model 110 is not suitable in this example for directly processing the raw data of the first sensor 106, i.e. not suitable for processing without converting it into a suitable data type or a suitable format. The first model 108 is not yet trained or not yet fully trained in this example. The first model 108 is designed to process the raw data of the first sensor 106.

[0020] The first model 108 includes a first classifier in this example. The second model 110 includes a second classifier in this example. The first classifier is a convolutional neural network (CNN) in this example. The second classifier is a convolutional neural network (CNN) in this example. Instead of a CNN, an artificial neural network with a different structure can also be provided. A classifier can also be provided that solves the usual classification problem in a different way.

[0021] The computational unit is designed in this example to predict results using a first model 108 for raw data measured by a first sensor 106 and to predict results using a second model 110 for raw data whose data type and format correspond specifically to the raw data of the second sensor.

[0022] Optionally, the first sensor 106 includes an interface 112 to a computing device 114 located outside the sensor 106. The computing device 114 may be a central controller of the vehicle or one or more servers in an internet infrastructure.

[0023] The computing unit is provided with computer readable instructions, which, when executed by the computing unit, perform steps in the method described below. It may be contemplated that the computing unit and the computing device 114 are designed to respectively perform some of the steps and to mutually exchange the necessary data therefor via the interface 112.

[0024] FIG. 2 shows diagrammatically the steps in a method for generating a model copy based on data. Defining the network architecture of the first model 108 is one technical challenge. Assuming that the network architecture is known, the remaining task is to train the network. Training the first model 108 using a supervised classification problem requires a large amount of labeled data. This expensive operation was performed in this example with respect to the network architecture and training of the second model 110. This expensive operation is avoided or minimized in this example with respect to the first model 108 by model copying based on the data.

[0025] The method is used, for example, for the next generation of radar sensors, which have extended technical capabilities compared to the previous generation of radar sensors. These can affect, for example, the characteristics of the radar sensor, such as the range, resolution, and aperture angle, increased. The method can also be used for fundamental changes in radar modulation and signal evaluation.

[0026] In this method, knowledge is transferred when using the data-based model in the next generation. Knowledge from the classical model can be transferred to the data-based model. The data-based model of the previous generation can be inherited. In the classical model, the challenge is to integrate the domain knowledge into the data-based model. In the data-based model, problems arise when the sensor signals from different generations are significantly different and beyond the generalization range of the second model 110.

[0027] The method may contemplate generating an algorithm copy of an existing algorithm used in a previous generation radar sensor based on input data in a format different from the next generation. This allows for application of the existing algorithm in the next generation radar sensor, for example, for generating ground truth or identifying relevant training data, particularly for a new algorithm based on the existing algorithm. When the algorithm is executed, a model is used to process the sensor data.

[0028] Instead of a radar sensor, other sensors can also be used, in particular LiDAR, ultrasonic, infrared or acoustic sensors. In step 202, given raw data 204 is converted into data 206 representative of the raw data of a second sensor. The data 206 is for example generated by a conversion unit which converts the raw data 204 into the data 206.

[0029] A transformation unit may include transformation rules or a model based on data. In this example, raw data 204 from the first sensor 106 is transformed into data 206 using transformation rules, which may be, for example, mathematically written or trained based on the data.

[0030] Alternatively, it may be contemplated to convert the raw data 204 of the first sensor 106 with a data-based model. Data for supervised training of the data-based model is provided, for example, by mounting the first sensor 106 and the second sensor on a test vehicle and recording data streams of raw data of both sensors for a representative test range. The data-based model to be searched transfers the raw data 204 of the first sensor 106 to the raw data of the second sensor. For example, this is achieved by supervised training of a neural network or other model. It may be contemplated to adaptively improve the data-based model during training, for example by automatic architecture search.

[0031] A transformation can also provide a sampling of the raw data. The use of this transformation allows copying based on the first sensor 106. In this example, both models and algorithms for the transformation are executed, for example in parallel or selectively, on the first sensor 106. The second sensor is not required to train the first model 108.

[0032] The method begins, for example, when raw data 204 of the first sensor 106 is provided. The raw data 204 can be measured by the first sensor 106 or can be read, for example, from a memory of a computing unit.

[0033] The raw data 204 of the first sensor 106 and the raw data of the second sensor can characterize a time domain signal. The raw data 204 of the first sensor 106, and the data 206 may characterize the spectrum of a radar, LiDAR, ultrasonic, infrared, or acoustic sensor, among others.

[0034] The raw data 204 and the data 206 of the first sensor 106 may characterize a position. The raw data 204 of the first sensor 106, and the data 206 may be filtered data or a transform of the data that characterizes a time domain signal, a spectrum, or a position.

[0035] In step 208, a first result 210 is determined using the given raw data 204 and the first model 108. In this example, steps 202 and 208 are performed sequentially, but may also be performed at least partially in parallel in time.

[0036] In step 212, a second result 214 is determined using the data 206 representing the raw data of the second sensor and the second predetermined model 110. Step 212 follows step 202 in this example.

[0037] It may be contemplated that the first result 210 and / or the second result 214 characterize an object type. The first result 210 and / or the second result 214 may be intended to characterize an estimate regarding a dimension of the object.

[0038] It may be contemplated that the first result 210 and / or the second result 214 may indicate whether a blind sensor, clustering, or object was recognized. Step 216 is then executed.

[0039] In step 216 it is determined whether the first result 210 differs from the second result 214 . If the first result 210 is different from the second result 214, then step 218 is performed. Otherwise, in this example, other raw data 204 is specified and step 202 is performed.

[0040] At step 218, training data points are determined that include the given raw data 204 and the second result 214. Step 220 is then executed.

[0041] At step 220, the first model 108 is trained using training data that includes the training data points. It may be contemplated that the training of the first model 108 is performed in multiple iterations.

[0042] It may be contemplated that the values ​​of the parameters 222 defining the first model 108 are initialized with random values ​​before the first iteration. It may also be contemplated that values ​​of the parameters 222 defining the first model 108 are determined or have been determined prior to the first iteration by training using raw data measured by the first sensor 106.

[0043] It is contemplated that the training may be performed such that a large number of training data points are determined in a large number of iterations without performing a training step, and then a training step may be performed in which the parameters 222 defining the first model 108 are determined.

[0044] It may be contemplated that the parameters 222 are determined using a portion of training data points from a number of training data points. It may also be contemplated that the parameters 222 are determined using specifically all of the training data points from a number of training data points.

[0045] For example, training occurs when the memory 104 contains a sufficient number of entries. This is done incrementally from the beginning, on a portion of the training data points, or on the entire batch containing all the training data points. Incremental training allows the memory 104 to be configured to be quite small. A sufficient amount of new training data points can typically range from one new training data point to several thousand new training data points. The fewer the number of iterations without training steps, the fewer redundant training data points are collected between training steps, with corresponding benefits in terms of memory size and dataset balance.

[0046] Step 220 is performed in this example at the first sensor 106. Alternatively, it is contemplated that step 220 may be performed at a computing device 114 external to the first sensor 106.

[0047] When the first model 108 is trained on the computing device 114, it is transmitted to the computational unit of the first sensor 106 after at least one training step. The first model 108 is transmitted to the first sensor 106, for example, via Firmware-Over-The-Air (FOTA) or a wired firmware update.

[0048] The newly trained first model 108 is updated, for example, by updating the coefficients in the computation unit of the first sensor 106. The periodic time interval can be, for example, once a day, once a month, etc.

[0049] Optionally, the method includes a step 224. In step 224, a structure of the first model 108 is determined according to a predetermined structure of the second model 110. Step 224 is preferably performed before the first iteration.

[0050] Step 224 includes, in this example, an architecture search using a machine learning system, in which the structure of the first model 108 is determined according to a predetermined structure of the second model 110 .

[0051] Alternatively, it may be contemplated that step 224 may include copying at least a portion of a given structure of the second model 110 into a structure of the first model 108 . Alternatively or additionally, it may be contemplated that step 224 may include copying values ​​of at least some of the predetermined parameters of the second model 110 to values ​​for the parameters 222 of the first model 108 .

[0052] The model type of the first model 108 may, for example, correspond to the model type of the second model 110 having an input adapted to the first sensor 106. The first model 108 may also have an arbitrarily selected structure different from the second model 110. The first model 108 may also be adapted to the respective data situation using a neural architecture search, for example AutoML.

[0053] The method may further provide the following steps, as illustrated diagrammatically in FIG. In this example, step 302 is executed at regular time intervals. A time point for this can be specified or it may be contemplated that such a time point is specified.

[0054] In step 302 , the first model 108 is transmitted from the calculation unit to a computing device 114 external to the first sensor 106 . Step 302 is then executed.

[0055] In step 304, a third model is determined as a function of the first model 108 and at least one other model. For this purpose, methods such as associative learning can be used. Step 306 is then executed.

[0056] In step 306, the first model 108 in the first sensor 106 is replaced with a third model, which in this example is a globally fused model, and is transmitted to the first sensor 106, for example, via firmware-over-the-air or wired firmware update.

[0057] The method may further provide the following steps, as illustrated diagrammatically in FIG. In step 402, it is checked whether the first result 210 is different from the second result 214. If the first result 210 is different from the second result, step 404 is executed. If not, step 404 is not executed in this example. Step 402 can also be implemented as part of step 216.

[0058] In step 404, the second result 214 with respect to the training data points 218 is checked to see if it is correct or incorrect. If the second result 210 is false, then step 406 is performed. Otherwise, step 406 is not performed in this example.

[0059] At step 406 , the training data points 218 are transmitted to a computing device 114 external to the first sensor 106 . Then, step 408 is executed.

[0060] At step 408, a third outcome is determined for the training data points 218. The third outcome, in this example, is determined using another model that is designed to predict the third outcome for the training data points 218. In this example, such another model is a trained model.

[0061] Then, step 410 is executed. In step 410, a modified training data point is determined by replacing the second result in the training data point 218 with the third result.

[0062] Step 412 is then executed. In step 412 , the modified training data points are transmitted to the first sensor 106 .

[0063] Step 414 is then executed. In step 414, the first model 108 is trained using the modified training data points.

[0064] The present invention can also be used to enable existing models of the same generation of sensors to be used in different mounting locations. For this purpose, two or more sensors are mounted on the test vehicle at different mounting positions, and for a representative test range of several sensors, the corresponding data streams are processed as described above.

Claims

1. 1. A computer-implemented method for generating a model copy based on data at a first sensor (106), comprising: Transforming (202) predetermined raw data (204) from a first sensor (106) into data (206) representative of the raw data of a second sensor; determining (208) a first outcome (210) using predetermined raw data (204) from the first sensor (106) and a first model (108) designed to predict an outcome based on the raw data from the first sensor (106), where parameters (222) define the first model (108); determining (212) a second outcome (214) using data (206) representative of the raw data of the second sensor and a second predetermined model (110) designed to predict an outcome based on the raw data from the second sensor; determining (216) whether the first result (210) is different from the second result (214); If the first result differs from the second result, determining (218) training data points comprising predetermined raw data (204) from the first sensor (106) and the second results (214); training (220) the first model (108) using training data that includes the training data points; The method according to claim 1, further comprising:

2. 2. The method of claim 1, characterized in that the predetermined raw data (204) from the first sensor (106) represents at least one spectrum, in particular of at least one time domain signal, in particular of a radar, LiDAR, ultrasonic, infrared or acoustic sensor, or at least one position, or filtered data thereof or a transform thereof.

3. 2. The method of claim 1, wherein the first result (210) and / or the second result (214) characterize an estimate for an object type or object dimensions, or the first result (210) and / or the second result (214) indicate blind sensor, clustering, or whether an object has been recognized.

4. 2. The method of claim 1, wherein the step of training (220) the first model (108) is performed in multiple iterations, and values ​​of parameters (222) defining the first model (108) are initialized with random values ​​before a first iteration of the step of training the first model (108).

5. 2. The method of claim 1, wherein the step of training (220) the first model (108) is performed in multiple iterations, and values ​​of the parameters (222) defining the first model (108) are determined or have been determined before a first iteration of the step of training the first model (108) by training using raw data measured by the first sensor (106).

6. 6. The method according to claim 4 or 5, characterized in that the step of training (220) of the first model (108) is performed in such a way that a large number of training data points are determined in multiple iterations without performing a training step, and then a training step is performed in particular on the first sensor (106) or on a computing device (114) external to the first sensor (106), and in which parameters (222) defining the first model (108) are determined using a portion of the large number of training data points or using the training data points from the large number of training data points.

7. 2. The method of claim 1, further comprising determining (224) a structure of the first model (108) in response to a predetermined structure of the second model (110).

8. 8. The method of claim 7, wherein the step of determining (224) a structure of the first model (108) includes an architecture search using a machine learning system, wherein in the architecture search, the structure of the first model (108) is determined in response to a predetermined structure of the second model (110).

9. 8. The method of claim 7, wherein the step of determining (224) the structure of the first model (108) comprises copying at least a portion of a predetermined structure of the second model (110) to a structure of the first model (108) and / or copying values ​​of at least a portion of predetermined parameters of the second model (110) to values ​​for parameters (222) of the first model (108).

10. 2. The method of claim 1, wherein the first model (108) is transmitted to a computing unit (102, 104) of the first sensor (106) after at least one training step in which parameters (222) defining the first model (108) are determined, such that the computing unit (102, 104) of the first sensor (106) is designed to predict results using the first model (108) with respect to raw data measured by the first sensor (106).

11. 11. The method according to claim 10, characterized in that, at specifiable or specified time points, preferably at regular time intervals, the first model (108) is transmitted (302) from the calculation unit (102, 104) to a computing device (114) external to the first sensor (106), a third model is determined (304) depending on the first model (108) and at least one further model, and the first model (108) at the first sensor (106) is replaced (306) by the third model.

12. If the first result (210) is different (402) from the second result (214), transmitting (406) the training data points (218) to a computing device (114) external to the first sensor (106); determining (408) the third outcome for the training data points (218) using another model that is specifically designed to predict a third outcome for the training data points (218); determining (410) modified training data points by replacing the second results with the third results for the training data points (218); transmitting (412) the modified training data points to the first sensor (106); training (414) the first model (108) using the modified training data points; 2. The method of claim 1, comprising:

13. 13. The method of claim 12, characterized in that it is checked (404) whether the second result (214) on the training data points (218) is correct or incorrect, and if the second result (214) is incorrect, the modified training data points are determined (412) and used (414) for the training (220) of the first model (108), and if not, the modified training data points are not determined and / or are not used for the training (220) of the first model.

14. A device (100) for generating a model copy based on data from a first sensor, characterized in that the device (100) is designed to carry out the method according to claim 1.

15. A program for executing the method according to claim 1.

Images