Computer-implemented method for comparing outputs of machine-learning detection models processing data acquired from at least one onboard sensor of an automotive vehicle

Abstract

The invention relates to a computer-implemented method for comparing outputs of machine-learning detection models processing data acquired from at least one onboard sensor (11, 12, 13) of an automotive vehicle, comprising the steps of: a) receiving input data representative of an environment exterior to the vehicle delivered by the at least one onboard sensor, b) predicting at least a first, respectively second, object by inputting said input data into a first, second, machine-learning detection model (21, 22), c) determining a standardized first, respectively second, score of the first, second, object based on the raw score of the first, second, object and on a first, second, mapping function (31, 32) associated to the first, second, machine-learning detection model, d) comparing the first standardized score and the second standardized score.

Description

DescriptionTitle of Invention: Computer-implemented method for comparing outputs of machine-learning detection models processing data acquired from at least one onboard sensor of an automotive vehicle Technical field of the invention

[0001] The invention relates to the domain of object recognition in automotive vehicles.

[0002] More precisely the invention relates to a device and a method for comparing outputs of machine-learning detection models processing data acquired from at least one onboard sensor of an automotive vehicle.

[0003] The invention also relates to a computer implementing said method, such as an onboard computer the vehicle or an offline computer.

[0004] The method for comparing outputs of detection models according to the invention allows in particular implementing a method for analysing redundant objects to be used in an advanced driver assistance system.Background

[0005] Advanced driver assistance systems (ADAS) are becoming more sophisticated and prevalent as the automotive industry strives to achieve higher levels of automation and safety. Object detection is one of the most critical tasks as it allows becoming aware of the relevant elements and agents in the near environment of the automotive vehicle.

[0006] Recent advancements in machine-learning, in particular in deep learning, have significantly improved the accuracy of detection models such as neuronal networks. But despite these advances, detection models are still far from infallible. The quality of the outcomes they produce is usually highly dependent on the characteristics of the input data, and the training and inference processes are influenced by diverse stochastic variables that are difficult to predict and control.

[0007] To improve object detection, in particular for automotive vehicles with higher levels of automation, onboard computers may implement ensemble methods which combine different detection models to obtain better predictive performance. In particular, within the scope of onboard perception of the environment of the vehicle, data from different sensors (including cameras, radars, and lidar scanners) may be respectively treated by different detection models.

[0008] When the different detection models predict redundant objects that may be representative of a same physical entity exterior to the vehicle at the same moment (e.g. the camera and the radar detect two pedestrians partially at the same place), the onboard computers implement a merging step which for instance may consist in selecting one of those redundant objects to be used in the driver assistance system.

[0009] Merging may be based on the scores associated to the objects, said scores characterizing the uncertainty of the predictions. Classically, the object with the higher score is preferentially selected over the others.

[0010] Although the scores are commonly normalized through softmax-like functions to comply with the definition of probability distribution, there is no guarantee that they can be straightforwardly interpreted as a probability. Indeed, some “over-optimistic” models will often be wrong (which means that they will produce many false positives), whereas some “over-pessimistic” models will almost always be correct (which means that they will produce false positives). The “optimism” of a model depends on the training data and learning procedure and is extremely difficult to control.

[0011] Therefore, there is a need for better merging redundant objects coming from different detection models.

[0012] There is also a need to provide a method to efficiently compare different prediction models, in order to select the one providing the best performances to be implemented in the vehicle.Summary of the invention

[0013] In this context, one object of the invention is to provide computer-implemented method for comparing outputs of machine-learning detection models processing data acquired from at least one onboard sensor of an automotive vehicle, the method comprising the steps of: a) receiving input data delivered by the at least one onboard sensor of the automotive vehicle, said input data being representative of an environment exterior to the automotive vehicle, b) predicting at least a first object by inputting all or part of said input data into a first machine-learning detection model and a second object by inputting all or part of said input data into a second machine-learning detection model, each object being characterized by a spatial representation and a raw score representative of a certainty in prediction, the first object and the second object having overlapping spatial representations, c) determining a standardized first score of the first object based on the raw score of the first object and on a first mapping function associated to the first machine-learning detection model and a standardized second score of the second object based on the raw score of the second object and on a second mapping function associated to the second machine-learning detection model, d) comparing the first standardized score and the second standardized score.

[0014] Thanks to the mapping functions, the scores of the objects predicted by the different models can be accurately compared. The mapping is specific to each detection modeland is based on precomputed parameters. Put differently, standardized scores are comparable to each other as they are adjusted to provide the same interpretation of the uncertainty of the predictions.

[0015] The standardized scores for instance allow merging effectively predictions from different models. Typically, the most probable detection among the redundant objects can then be used in the driver assistance system.

[0016] The standardized scores also allow comparing the models and selecting the one(s) providing the best performances to be implemented in the vehicle.

[0017] In addition, the standardized score of the selected object that can be further used in the driver assistance system, for example to ignore low-score detections that are unlikely to be correct. Indeed, the standardized scores are better representative of the uncertainty of the predictions than the raw scores.

[0018] Advantageously, the standardized scores do not require any retraining or modification of the internal parameters of the models involved.

[0019] Other advantageous and non-limiting features of the method according to the invention are:- step d) further comprises merging the spatial representation of the first object and the spatial representation of the second object based on the comparison between the first standardized score and the second standardized score;- in step d), merging comprises one of the following: selecting the object, among the first object and the second object, with the higher standardized score; removing the part of the spatial representation characterizing the object, among the first object and the second object, with the lowest standardized score which overlaps the spatial presentation of the other object; performing a weighted sum of the spatial representations of the objects;- said input data comprise first input data delivered by a first sensor of the automotive vehicle and second input data delivered by a second sensor of the automotive vehicle, the method comprising: predicting the first object by inputting the first input data into the first machine-learning detection model; predicting the second object by inputting the second input data into the second machine-learning detection model;- said input data are of one or more of the following types: radar points, lidar points, images, ultrasound data;- the mapping function of each machine-learning detection model is determined by means of the following steps: e) initializing said mapping function, f) providing a calibration dataset, distinct from a dataset on which said machinelearning detection model has been trained, comprising calibration data and pre-labelled objects,g) predicting calibration objects by inputting the calibration data into said machinelearning detection model, h) determining standardized scores of the calibration objects based on the raw scores of the calibration objects and on said mapping function, i) determining efficiency data based on comparisons between spatial representations of the pre-labelled objects and spatial representations of the calibration objects, j) adjusting said mapping function by minimizing a loss function between the standardized scores of the calibration objects and the efficiency data;- the calibration data are of the same type than those inputted in said machine-learning detection models in step b);- said comparisons between the spatial representations of the pre-labelled objects and the spatial representations of the calibration objects are based on ratios of intersection over union between the spatial representations of the calibration objects and the spatial representations the pre-labelled objects;- within the efficiency data, each calibration object is considered as a true positive if a ratio of intersection over union between the spatial representation of said calibration object and a spatial representation of one of the pre-labelled objects is superior to a predetermined threshold and is considered as a false negative if not.

[0020] The invention also relates to an onboard computer of an automotive vehicle comprising:H a memory unit storing:- a first machine-learning detection model and a second machine-learning detection model,- a first mapping function associated to the first machine-learning detection model and a second mapping function associated to the second machine-learning detection model,H a processing unit programmed to implement the following steps: a) receiving input data delivered by at least one onboard sensor of an automotive vehicle, said input data being representative of an environment exterior to the automotive vehicle, b) predicting at least a first object by inputting all or part of said input data into the first machine-learning detection model and a second object by inputting all or part of said input data into the second object machine-learning detection model, each object being characterized by a spatial representation and a raw score representative of a certainty in prediction, the first object and the second object having overlapping spatial representations, c) determining a standardized first score of the first object based on the raw score of the first object and on the first mapping function and a standardized second scoreof the second object based on the raw score of the second object and on the second mapping function, d) comparing the first standardized score and the second standardized score.

[0021] The invention also relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out a method as described above.Detailed description

[0022] The following description, enriched with joint drawings that should be taken as non- limitative examples, will help understand the invention and figure out how it can be realized.

[0023] In the accompanying drawings:

[0024] - [Fig.l] is schematical representation of an onboard computer of an automotive vehicle according to one embodiment of the invention,

[0025] - [Fig.2] is a bloc diagram representing the steps allowing to determine mapping functions stored in the onboard computer of [Fig.l],

[0026] - [Fig.3] is a schematical representation of the architecture of the onboard computer of [Fig.l] implementing a method for comparing outputs of machine-learning detection models.

[0027] An onboard computer 1 of an automotive vehicle, such as a car, is represented in [Fig.l]. The onboard computer 1 is a data-processing device configured, that is to say programmed, to implement a method for comparing outputs of machine-learning detection models in order to analyse redundant objects to be used in a driver assistance system of an automotive vehicle.

[0028] Alternately, the computer can be a computer outside of an automobile vehicle, configured to implement a method for comparing, in an offline manner, outputs of machine-learning detection models acquired from one or more sensors of an automotive vehicle in order to select one detection model.

[0029] As shown in [Fig.l], the onboard computer 1 comprises a processing unit 2, for instance a processor, and a memory 3, which is here non-transitory. The memory 3, which forms a computer-readable storage medium, comprises instructions which, when executed by the processing unit 2, allows implementing the method. The data stored in the memory 3 form a computer program product, comprising instructions which, when the program is executed by a processing unit 2, cause the onboard computer 1 to carry the method.

[0030] The data stored in the memory 3 comprises in particular machine-learning detection models. As represented in [Fig.3], the memory 3 stores N machine-learning detection models 21, 22, 23, N being an integer greater or equal than two. The machine-learningdetection models 21, 22, 23 are here neural networks. However, the machine-learning detection models may present different structures and may for instance be supportvector machines. Hereinafter, the machine-learning detection models are more simply referred to as “models”.

[0031] Before their storing on the memory 3, the models 21, 22, 23 have been trained to predict objects based on input data representative of the environment of the vehicle. Objects are numerical representation which are typically representative of physical entities, that is to say real-life objects, exterior to the vehicle. Nevertheless, an object may not correspond to a physical entity, in particular when said object is a false positive (i.e. a wrong detection). By “predicting”, it is meant that each model is designed to detect or recognize objects by processing the input data. In practice, predicting is thus a computing step performed by the onboard computer 1 by means of the model.

[0032] Each object is more specifically characterized by a spatial representation, here in a frame of reference (2D or 3D) linked to the vehicle, and at least one score representative of a confidence in the object’s prediction. The spatial representation and the one or more scores of each object are assigned by the model predicting said object and depend on the training of said model.

[0033] The spatial representation of each object describes the dimensions and the localisation of said object in the frame of reference. As it appears later, the spatial representations of the different objects are comparable to each other. Here, the spatial representation of each object is a bounding box, for instance an arbitrary oriented bounding box. When a model is designed to process 2D images, the bounding box of an object predicted by this model is typically defined as the smallest rectangle encompassing the object. Here, each object is thus described in the frame of reference by a bounding box composed of values expressing the location of its center, its dimensions and its orientation.

[0034] Here, each object is also characterized by a category, also assigned by the model predicting it. The category is the result of a classification performed by the model. The different categories into which said object may be classified are predetermined. The categories correspond to objects that may be present on or near a road. For instance, the categories are the following: car, bus, truck, motorcycle, bicycle, pedestrian, road sign.

[0035] Each object is here characterized by a plurality of scores. The scores are delivered by the models when asked to perform classification tasks. Each score is representative of a certainty, that is to say a confidence (or conversely an uncertainty), of the prediction. This certainty is expressed by a numerical value, for instance a floating number between 0 and 1. It is thus comparable, but not strictly egal, to a probability.

[0036] The different scores of an object are relative to its category and to its spatial representation, i.e. to the values composing its bounding box. Here, the different scores of an object are thus representative of a confidence in category classification, a confidence in dimension estimation, a confidence in center localization estimation and a confidence in orientation estimation.

[0037] As a variant, each object is characterized by only one score, which is for instance relative to the category of the object (i.e. representative of a confidence in category classification).

[0038] The models 21, 22, 23 stored in the memory 3 are different from each other. This means, in particular, that the models 21, 22, 23 may differ in terms of type of data they process (for instance images or radar points), in terms of structure (for instance different number of neurons or of layers), in terms of training datasets or in terms of learning protocols (for instance supervised or unsupervised). Therefore, even processing the same inputs, two models may give different outputs.

[0039] As illustrated in the embodiment of [Fig.l], the onboard computer 1 also comprises communication means 3 which especially allows the onboard computer 1 to receive the input data. The input data are delivered, i.e. provided, by at least one onboard sensor of the vehicle. In the example represented in [Fig.3], the onboard computer 1 receives the input data from three distinct sensors 11, 12, 13, for instance a camera, a radar and a lidar scanner. One of the sensors may also be an ultrasonic sensor.

[0040] The onboard computer 1 is also programmed to implement an advanced driver assistance system (ADAS), based the objects detected by the models. The advanced driver assistance system is a system which helps the driver pilot or control the vehicle for increasing safety, such as: intelligent speed adaptation, lane departure warning, adaptative cruise control or even autonomous driving.

[0041] The data stored in the memory 3 also comprise mapping functions 31, 32, 33 associated to the models. They more particularly comprise one mapping function per model and per category said model uses for classification, that is to say per category into which said model is designed to classify objects.

[0042] As described afterwards, the role of each mapping function it to transform the scores of an object directly assigned by its model, hereinafter referred to as raw scores, into a standardized score.

[0043] Here, each mapping function is a temperature scaling-like function of the following form:

[0045] Where CsSmiis the standardized score of the object, C;for i=l,..., M are the M raw scores of the object provided by its associated model (representing differentconfidences, for instance the confidence in category classification, the confidence in size estimation, etc.), andare predefined calibration parameters. As detailed below, these calibration parameters are predefined in the sense that they are determined previously to their storing on the memory 3.

[0046] As a variant, when the model outputs a single score (Craw) for the object (e.g. representing the confidence in category classification), the previously introduced formula takes the following form:

[0048] Each mapping function is thus defined by its calibration parameters, here by at least two calibration parameters.

[0049] The number of calibration parameters to defined for all the mapping functions is upper bounded by NxM’xT with N the number of models, M ’ the maximum number of scores provided by a model, and T the number of predefined categories that can be detected.

[0050] At this stage, a method for determining the mapping functions is described in reference to [Fig.2] . This determination is an offline procedure, performed before storing the mapping functions on the memory 3 of the onboard computer 1. This method may be performed by processing means distinct from the onboard computer 1. The determination of the different mapping functions being identical, the determination of one of the mapping functions (referred to as the mapping function) of one of the models (referred to as the model) for one category (referred to as the category) is described below.

[0051] Determining the mapping function means determining its calibration parameters.

[0052] The principle of this determination is to compare the output of the model with a faithful representation of the reality, commonly known as ground-truth. The mapping function is adjusted, i.e. optimized, to increase the accuracy of the model, the accuracy being its capacity to predict objects strictly corresponding to real-life objects.

[0053] As shown in [Fig.2], the method for determining the mapping function starts with a step e) of initializing the mapping function.

[0054] In step e), the calibration parameters are set to initial values. The initial values are for instance all set to one.

[0055] In the next steps of the method, the objective is to determine the calibration parameters by modifying the values of the calibration parameters.

[0056] The method also comprises a step f) of providing a calibration dataset comprising calibration data and pre-labelled objects.

[0057] The calibration dataset comprises data suitable for object prediction. The calibration dataset is distinct from a dataset on which the model has been trained.

[0058] The calibration data are of the same type than those inputted in the model when used by the onboard computer 1 (at step b) below). For instance, when said model is designed to process 2D images from a camera, the calibrations data are also 2D images. Preferentially, the calibration dataset is close to the domain of application of the onboard computer 1, for instance in the sense that the dataset is obtained from sensors and perspectives similar to those of the vehicle, so that the standardized scores provided by the mapping function are more reliable.

[0059] Pre-labelled objects correspond to physical entities represented in the calibration data and that the model should be able to detect. Those objects are “pre-labelled” in the sense that they have been previously assigned a spatial representation and a category. In practice, the pre-labelled objects are manually labelled by an operator which adjusts bounding boxes to the objects, and which selects their category.

[0060] In other terms, the calibration dataset is a labelled dataset wherein the pre-labelled objects correspond to the ground-truth associated with the calibration data.

[0061] Preferentially, the same calibration dataset is used for determining all the mapping functions. In this case, when the models are inputted with different type of data, the calibration dataset also comprises different type of data.

[0062] The method then continues with a step g) of predicting calibration objects by inputting the calibration data into the model.

[0063] Only calibration objects of the category (associated with the mapping function) are predicted. Alternatively, objects of all the categories of the model may be predicted but only the ones of the category are considered for the subsequent steps.

[0064] The method continues with a step h) of determining standardized scores of the calibration objects. A standardized score is more particularly determined for each calibration object.

[0065] The standardized score of each calibration object is determined based on the raw scores of said calibration object (obtained from step g)) and on the mapping function (obtained from step e) or step j)). The standardized score is computed according to the above-mentioned formula based on the raw scores of the calibration object and on the calibration parameters of the mapping function.

[0066] The method then comprises a step i) of determining efficiency data based on comparisons between the spatial representations, here the bounding-boxes, of the pre-labelled objects, and the spatial representations, here the bounding-boxes, of the calibration objects.

[0067] As shown in [Fig.2], the pre-labelled objects are obtained from step f) as they represent the ground-truth of the calibration objects.

[0068] Efficiency data are representative of the performance of the model in predicting objects of the category associated with the mapping function. To do so, the calibrationobjects of the category are compared with the pre-labeled objects, that is to say with the ground-truth, of the category.

[0069] By way of an example, the efficiency data are more specifically determined as follows.

[0070] For each calibration object, ratios of intersection over union, i.e. Jaccard indexes, between its bounding box and the bounding-boxes of the pre-labelled objects (of the category) are computed. As a variant, only one ratio of intersection over union between the bounding box of said calibration object and the bounding-box of the pre-labelled object which is spatially the closest from said calibration object is computed.

[0071] Said calibration object is then considered as a true positive if a ratio of intersection over union between its bounding-box and a bounding-box of one of the pre-labelled objects is superior to a predetermined threshold. Otherwise (when this condition is not satisfied) said calibration object is considered as a false negative. The predetermined threshold is for instance comprised between 0,5 and 0,7.

[0072] The efficiency data classify each calibration object as a true positive, in this case the value one may be associated to this calibration object, or as a false negative, in this case the value zero may be associated to this calibration object.

[0073] The efficiency data thus comprises a plurality of values, for instance zeros and ones, which may take the form a vector. They more specifically comprise one value per calibration object.

[0074] The method finally comprises a step j) of adjusting the mapping function.

[0075] In step j), a loss function is firstly defined between the standardized scores of the calibration objects and the efficiency data. This loss function takes into account all the calibration objects (of the category).

[0076] The loss function is here a logistic loss function. As a variant, the loss function is a square loss function or an exponential loss function.

[0077] The calibration parameters of the mapping function are then determined by minimizing the loss function, for instance by means of the Nelder-Mead method. This minimizing is an iterative process in which the values of the calibration parameters are repeatedly modified, and the standardised scores are correspondingly repeatedly recalculated, such that the loss function is minimized. In [Fig.2] , this iteration is illustrated by a feed-back branch from step j) to step h).

[0078] The general idea is to adjust the mapping function such that true positive calibration objects have a standardized score close to one (as they effectively correspond to a real physical entity) and that false negative calibration objects have a standardized score close to zero (as they do not correspond to a real physical entity).

[0079] The method for comparing outputs of machine-learning detection models processing data acquired from at least one onboard sensor of the automotive vehicle is now described in reference to [Fig.3] .

[0080] This method is here described in the context of an online procedure implemented by the processing unit 2 of the onboard computer 1 for analyzing redundant objects to be used by a driver assistance system.

[0081] However, the method may also be used as an offline procedure, typically to estimate the performances of various detection models. The models and mapping functions are stored in the memory 3 of the onboard computer 1.

[0082] As shown in [Fig.3], this method starts with a step a) of receiving input data. In the example illustrated in [Fig.3], the input data are provided by the three onboard sensors 11, 12, 13 of the vehicle.

[0083] The input data may for instance comprise first input data delivered by a first sensor 11 of the vehicle and second input data delivered by a second sensor 12 of the vehicle.

[0084] In a remarkable manner, the method is particularly efficient when different sensors are involved since the input data contain rich and varied information.

[0085] The method continues with a step b) of detecting objects.

[0086] In step b), each model 21, 22, 23 predicts objects based on the input data. Put differently, the input data are used as inputs of the models 21, 22, 23 to predict objects.

[0087] Each model 21, 22, 23 may use input data coming from one or several sensors 11, 12, 13.

[0088] By way of an example, as illustrated in [Fig.3], a first model 21 predicts objects based on the first input data provided by the first sensor 11 and a second model 22 predicts objects based on the second input data provided by the second sensor 12. As shown in [Fig.3], a third model 23 also uses input data coming from several sensors 12, 13.

[0089] In step b), the raw scores (that is to say the score directly assigned by the models) of each object are normalized between zero and one, here by the models themselves.

[0090] At least two objects, predicted by distinct models, are redundant. Here, this generally means that their spatial representations overlap (i.e. are superposed) in the frame of reference. This means that the intersection of their spatial representations is different from zero. In other words, their bounding boxes have a common portion. In practice, it means that the two objects are representing the same real- world physical entity.

[0091] Preferentially, two objects are considered redundant when the intersection over union of their bounding boxes is above a predetermined threshold, for instance comprises between 50% and 70%.

[0092] By way of an example, a first object predicted by the first model 21 and a second object predicted by the second object 22 have overlapping bounding boxes. In thisexample, the first object is categorized as a car with only one first raw score of 0,7 and the second object is also categorized a car but with only one second raw score of 0,8.

[0093] More generally, each model predicts a set of objects, the sets (N sets in [Fig.3], corresponding to the N models) being redundant as they comprise objects having overlapping spatial representations.

[0094] The method then comprises a step c) of determining standardized scores of the objects predicted in step b).

[0095] In step c), each one of the N set goes through score standardization, which comprises a mapping over the raw scores of the predicted objects. More specifically, a standardized score is computed for each object. As represented in [Fig.3], each set of objects is respectively processed by means of the mapping functions 31, 32, 33 associated to the model 21, 22, 23 which has predicted it, thanks to the formula abovedescribed.

[0096] Following the example above, in step c), the first object may obtain a first standardized score of 0,9 and the second may obtain a second standardized score of 0,85. In this example, it means that the first model 21 is “over-pessimistic” since the first standardized score is higher than the first raw score and that the second model 22 is “over-optimistic” since the second standardized score is lower than the second raw score.

[0097] The spatial representations of the objects are not modified in step c).

[0098] The standardized scores are comparable to each other as they are adjusted to provide the same interpretation of the certainty of the predictions.

[0099] Such a comparison is performed in a final step d) of the method.

[0100] In the context of an offline processing, step d) may comprise comparing the standardized score of redundant objects to determine accuracy levels of the different models, so as to select the one(s) to install in the vehicle. Indeed, as the outputs of the models are calibrated, their performances can be directly compared using the same sets of raw data provided by the sensors of the vehicle.

[0101] In the context of an online processing, step d) may more specifically comprise merging the sets (the N sets in [Fig.3]) of redundant objects to obtain a single set of non-redundant objects, based on comparisons of the standardized scores.

[0102] The merging may comprise the following two sub-steps (performed by a merging unit 40 in [Fig.3]).

[0103] Firstly, a new set of predicted objects is obtained as the union of all the sets (the N sets in [Fig.3]).

[0104] Secondly, a duplicate suppression method is used to eliminate redundant objects. Here, the onboard computer 1 implements a non-maximum suppression (NMS) that computes the intersection over union overlap between all the bounding boxes of all theobjects predicted in step b) (in the frame of reference), and among those whose overlap (preferentially above a predetermined threshold), selects only one and suppresses the others according to a criterion based on the standardized score. More particularly, the object with the highest standardized score is maintained and the other are suppressed.

[0105] This means that merging comprises selecting, among redundant objects, the object with the highest standardized score to be further used in the advanced driver assistance system.

[0106] Following the example above, the first object is selected, and the second object is removed as the first standardized score is higher than the second standardized score.

[0107] As a variant, merging may comprise removing only the overlaps of the bounding boxes which have the lowest standardized scores.

[0108] As a variant, merging may comprise obtaining the detection as a weighted sum or an average of the redundant objects. For instance, when the first object, with a standardized score of 0,9, has a length of 2 meters and the second object, with a standardized score of 0,1, has a length of 3 meters, the length of the final object to be further used in the advanced driver assistance system may be computed as 0,9x2+0,lx3=2,l meters.

[0109] The outcome of step d) is a set of non-redundant objects that includes the most probable detections (i.e. objects) among all those provided by the models. In addition, they are provided with reliable confidence scores, namely the standardized scores, which can be interpreted as probabilities.

[0110] The method may comprise a further step (not represented) of using the set of non- redundant objects, typically the first object, in the advanced driver assistance system.

[0111] The invention is not limited to the embodiment described and illustrated and any variant in accordance with the appended claims could be provided.

[0112] In particular, the different models may be provided with input data coming from only one sensor. It is still advantageous to process their outputs according to the method above-described as said outputs are different (since the models are different). The spatial representations of the objects, which also allows computing overlaps, may differ from bounding boxes and may for instance be bounding spheres.

Claims

Claims

1. Computer-implemented method for comparing outputs of machine-learning detection models processing data acquired from at least one onboard sensor (11, 12, 13) of an automotive vehicle, the method comprising the steps of: a) receiving input data delivered by the at least one onboard sensor (11, 12, 13) of the automotive vehicle, said input data being representative of an environment exterior to the automotive vehicle, b) predicting at least a first object by inputting all or part of said input data into a first machine-learning detection model (21) and a second object by inputting all or part of said input data into a second machine-learning detection model (22), each object being characterized by a spatial representation and a raw score representative of a certainty in prediction, the first object and the second object having overlapping spatial representations, the method being characterized in that it further comprises the steps of : c) determining a standardized first score of the first object based on the raw score of the first object and on a first mapping function(31) associated to the first machine-learning detection model (21) and a standardized second score of the second object based on the raw score of the second object and on a second mapping function(32) associated to the second machine-learning detection model (22), d) comparing the first standardized score and the second standardized score.

2. Method according to claim 1, wherein step d) further comprises merging the spatial representation of the first object and the spatial representation of the second object based on the comparison between the first standardized score and the second standardized score.

3. Method according to claim 2, wherein in step d), merging comprises one of the following:- selecting the object, among the first object and the second object, with the higher standardized score,- removing the part of the spatial representation characterizing the object, among the first object and the second object, with the lowest standardized score which overlaps the spatial presentation of the other object,- performing a weighted sum of the spatial representations of the objects.

4. Method according to any one of claims 1 to 3, wherein said input data comprise first input data delivered by a first sensor (11) of the automotive vehicle and second input data delivered by a second sensor (12) of the automotive vehicle, the method comprising:- predicting the first object by inputting the first input data into the first machine-learning detection model,- predicting the second object by inputting the second input data into the second machine-learning detection model.

5. Method according to any one of claims 1 to 4, wherein said input data are of one or more of the following types: radar points, lidar points, images, ultrasound data.

6. Method according to any one of claims 1 to 5, wherein the mapping function (31, 32, 33) of each machine-learning detection model (21, 22, 23) is determined by means of the following steps: e) initializing said mapping function (31, 32, 33), f) providing a calibration dataset, distinct from a dataset on which said machine-learning detection model (21, 22, 23) has been trained, comprising calibration data and pre-labelled objects, g) predicting calibration objects by inputting the calibration data into said machine-learning detection model (21, 22, 23), h) determining standardized scores of the calibration objects based on the raw scores of the calibration objects and on said mapping function (31, 32, 33), i) determining efficiency data based on comparisons between spatial representations of the pre-labelled objects and spatial representations of the calibration objects, j) adjusting said mapping function (31, 32, 33) by minimizing a loss function between the standardized scores of the calibration objects and the efficiency data.

7. Method according to claim 6, wherein the calibration data are of the same type than those inputed in said machine-learning detection models (21, 22, 23) in step b).

8. Method according to claim 6 or 7, wherein said comparisons between the spatial representations of the pre-labelled objects and the spatial representations of the calibration objects are based on ratios of intersection over union between the spatial representations of the calibration objects and the spatial representations the pre-labelled objects.

9. Method according to claim 8, wherein, within the efficiency data, each calibration object is considered as a true positive if a ratio of intersection over union between the spatial representation of said calibration object and a spatial representation of one of the prelabelled objects is superior to a predetermined threshold and is considered as a false negative if not.

10. Computer (1), characterized in that it comprises: n a memory unit (3) storing:- a first machine-learning detection model (21) and a second machine-learning detection model (22),- a first mapping function (31) associated to the first machinelearning detection model (21) and a second mapping function (32) associated to the second machine-learning detection model (22), n a processing unit (2) programmed to implement the following steps: a) receiving input data delivered by at least one onboard sensor (11, 12, 13) of an automotive vehicle, said input data being representative of an environment exterior to the automotive vehicle, b) predicting at least a first object by inputting all or part of said input data into the first machine-learning detection model (21) and a second object by inputting all or part of said input data into the second object machine-learning detection model (22), each object being characterized by a spatial representation and a raw score representative of a certainty in prediction, the first object and the second object having overlapping spatial representations, c) determining a standardized first score of the first object based on the raw score of the first object and on the first mapping function (31) and a standardized second score of the second object based on the raw score of the second object and on the second mapping function (32),d) comparing the first standardized score and the second standardized score.

11. Computer-readable storage medium (3) comprising instructions which, when executed by a computer, cause the computer to carry out a method according to any one of claims 1 to 9.

Images