A computer-implemented method for comparing the outputs of machine learning detection models processing data acquired from at least one sensor installed in a motor vehicle.

Abstract

The invention relates to a computer-implemented method for comparing outputs of machine learning detection models processing data acquired from at least one on-board sensor (11, 12, 13) of a motor vehicle, comprising the following steps: a) receiving input data representative of an environment external to the vehicle provided by at least one on-board sensor, b) predicting at least a first object, respectively a second, by injecting this input data into a first, second, machine learning detection model (21, 22), c) determining a first normalized score, respectively the second, of the first and second object, on the basis of the raw score of the first, second, object and the first, second, matching function (31, 32) associated with the first, second, machine learning detection model, d) comparing the first normalized score and the second normalized score.Figure for the summary: Fig. 3.

Description

Title of the invention: Computer-implemented method for comparing the outputs of machine learning detection models processing data acquired from at least one sensor installed in a motor vehicle. Technical field of the invention

[0001] The invention relates to the field of object recognition in motor vehicles

[0002] More specifically, the invention relates to a device and a method for comparing outputs of machine learning detection models processing data acquired from at least one sensor on board a motor vehicle.

[0003] The invention also relates to a computer implementing said method, such as a vehicle on-board computer or a remote computer.

[0004] The method for comparing the outputs of detection models according to the invention makes it possible in particular to implement a method for analyzing redundant objects to be used in an advanced driver assistance system. State of the art

[0005] Advanced Driver Assistance Systems (ADAS) are becoming increasingly sophisticated and widespread as the automotive industry strives to achieve higher levels of automation and safety. Object detection is one of the most critical tasks because it enables awareness of relevant elements and agents in the immediate environment of the motor vehicle.

[0006] Recent advances in machine learning, particularly deep learning, have significantly improved the accuracy of detection models such as neural networks. However, despite these advances, detection models are still far from infallible. The quality of the results they produce generally depends heavily on the characteristics of the input data, and the training and inference processes are influenced by various stochastic variables that are difficult to predict and control.

[0007] To improve object detection, particularly for motor vehicles with the highest levels of automation, on-board computers can implement ensemble methods that combine different detection models to achieve better predictive performance. In particular, within the framework of on-board perception of the vehicle's environment, data from Different sensors (including cameras, radars and lidar scanners) can respectively be processed by different detection models.

[0008] When the different detection models provide for redundant objects which can be representative of the same physical entity outside the vehicle at the same time (e.g. the camera and the radar detect two pedestrians partially in the same place), the on-board computers implement a fusion step which can, for example, consist of selecting one of these redundant objects to use in the driver assistance system.

[0009] The fusion can be based on the scores associated with the objects, these scores characterizing the uncertainty of the predictions. Classically, the object with the highest score is preferred to the others.

[0010] Although scores are generally normalized by softmax functions to conform to the definition of a probability distribution, there is no guarantee that they can be directly interpreted as a probability. Indeed, some "overly optimistic" models will often be wrong (meaning they will produce many false positives), while some "overly pessimistic" models will almost always be correct (meaning they will produce false positives). The "optimism" of a model depends on the training data and the learning procedure, and is extremely difficult to control.

[0011] Consequently, there is a need for better merging of redundant objects from different detection models.

[0012] There is also a need for a method to efficiently compare different prediction models, in order to select the one that provides the best performance for implementation in the vehicle. Presentation of the invention

[0013] In this context, one of the objects of the invention is to provide a computer-implemented method for comparing outputs of machine learning detection models processing data acquired from at least one on-board sensor of a motor vehicle, the method comprising the following steps: a) receiving input data provided by at least one on-board sensor of the motor vehicle, this input data being representative of an environment external to the motor vehicle, b) predicting at least a first object by injecting all or part of this input data into a first machine learning detection model and a second object by injecting all or part of this input data into a second machine learning detection model, each object being characterized by a spatial representation and a raw score representative of a prediction confidence, the first object and the second object having overlapping spatial representations, c) determine a first normalized score for the first object based on the raw score of the first object and a first matching function associated with the first machine learning detection model, and a second normalized score for the second object based on the raw score of the second object and a second matching function associated with the second machine learning detection model, d) compare the first normalized score and the second normalized score.

[0014] Thanks to the matching functions, the scores of the objects predicted by the different models can be compared accurately. The matching is specific to each detection model and is based on pre-calculated parameters. In other words, the normalized scores are comparable to each other because they are adjusted to provide the same interpretation of the uncertainty of the predictions.

[0015] Normalized scores, for example, allow for the efficient merging of predictions from different models. Typically, the most probable detection among redundant objects can then be used in the driver assistance system.

[0016] The normalized scores also make it possible to compare the models and to select the one or those offering the best performance to be implemented in the vehicle.

[0017] Furthermore, the normalized score of the selected object can be used in the driver assistance system, for example, to ignore low-score detections that are unlikely to be correct. Indeed, normalized scores are more representative of the uncertainty of predictions than raw scores.

[0018] Advantageously, the normalized scores do not require any further training or modification of the internal parameters of the models involved.

[0019] Other advantageous and non-limiting features of the process according to the invention are: - step d) further includes merging the spatial representation of the first object and the spatial representation of the second object based on the comparison between the first normalized score and the second normalized score; - in step d), the merging includes one of the following: selecting the object, from the first object and the second object, with the highest normalized score; removing the part of the spatial representation characterizing the object, from the first object and the second object, with the lowest normalized score that overlaps the spatial presentation of the other object; performing a weighted sum of the spatial representations of the objects; - the input data includes first input data provided by a first sensor of the motor vehicle and second input data provided by a second sensor of the motor vehicle, the method including: the prediction of the first object by injecting the first input data into the first machine learning detection model; the prediction of the second object by injecting the second input data into the second machine learning detection model; - said input data belong to one or more of the following types: radar points, lidar points, images, ultrasound data; - The matching function of each machine learning detection model is determined using the following steps: e) initialize said matching function, (f) provide a calibration dataset separate from a dataset on which said machine learning detection model was trained, comprising calibration data and pre-labeled objects, (g) predict calibration objects by inputting calibration data into said machine learning detection model, (h) determine normalized scores for the calibration objects based on the raw scores of the calibration objects and said matching function, i) determine efficiency data from comparisons between the spatial representations of pre-labeled objects and the spatial representations of calibration objects, j) adjust said matching function by minimizing a loss function between the normalized scores of the calibration objects and the efficiency data; - the calibration data are of the same type as those injected into the said machine learning detection models in step b); - said comparisons between the spatial representations of the pre-labeled objects and the spatial representations of the calibration objects are based on intersection rates on union between the spatial representations of the calibration objects and the spatial representations of the pre-labeled objects; - in the efficiency data, each calibration object is considered a true positive if an intersection-on-union rate between the spatial representation of that calibration object and a spatial representation of one of the pre-labeled objects is greater than a predetermined threshold and is considered a false negative otherwise.

[0020] The invention also relates to an on-board computer for a motor vehicle comprising: H is a memory unit storing: - a first machine learning detection model and a second machine learning detection model, - a first matching function associated with the first machine learning detection model and a second matching function associated with the second machine learning detection model, H a processing unit programmed to implement the following steps: a) receive input data provided by at least one on-board sensor of a motor vehicle, this input data being representative of an environment external to the motor vehicle, b) predict at least a first object by injecting all or part of said input data into the first machine learning detection model and a second object by injecting all or part of the input data into the second machine learning detection model;each object being characterized by a spatial representation and a raw score representative of a prediction confidence, the first object and the second object having overlapping spatial representations, c) determine a first normalized score of the first object on the basis of the raw score of the first object and the first correspondence function, and a second normalized score of the second object on the basis of the raw score of the second object and the second correspondence function, d) compare the first normalized score and the second normalized score. ;

[0021] The invention also relates to a computer-readable storage medium comprising instructions which, when executed by a computer, enable the computer to execute the method described above. Detailed description of the invention

[0022] The following description, enhanced by accompanying drawings which should be considered as non-limiting examples, will help to understand the invention and how it can be realized.

[0023] In the attached drawings:

[0024] - [Fig. 1] schematically represents an on-board computer of a vehicle automobile according to an embodiment of the invention,

[0025] - [Fig.2] is a block diagram representing the steps allowing determine the matching functions stored in the on-board computer of the [Fig.1],

[0026] - [Fig. 3] is a schematic representation of the computer architecture of edge of [Fig.l] implementing a method for comparing the results of machine learning detection models.

[0027] An on-board computer 1 of a motor vehicle, such as a car, is shown in [Fig. 1]. The on-board computer 1 is a configured data processing device, i.e. programmed, to implement a method for comparing the outputs of machine learning detection models in order to analyze redundant objects for use in a driver assistance system of a motor vehicle.

[0028] Alternatively, the computer may be a computer located outside a motor vehicle, configured to implement a method of comparing, in a remote (or offline) manner, outputs of machine learning detection models acquired from one or more sensors of a motor vehicle in order to select a detection model.

[0029] As shown in [Fig. 1], the on-board computer 1 comprises a processing unit 2, for example a processor, and a memory 3, which is non-volatile. The memory 3, which forms a computer-readable storage medium, includes instructions which, when executed by the processing unit 2, enable the implementation of the method. The data stored in the memory 3 constitutes a computer program product, comprising instructions which, when the program is executed by a processing unit 2, cause the on-board computer 1 to carry the method.

[0030] The data stored in memory 3 includes, in particular, machine learning detection models. As shown in [Fig. 3], memory 3 stores N machine learning detection models 21, 22, 23, where N is an integer greater than or equal to two. The machine learning detection models 21, 22, 23 are here referred to as neural networks. However, machine learning detection models can have different structures and can, for example, be support vector machines. Hereafter, machine learning detection models are more simply referred to as "models".

[0031] Before being stored in memory 3, models 21, 22, and 23 were trained to predict objects from input data representative of the vehicle's environment. The objects are digital representations that are typically representative of physical entities, i.e., real objects external to the vehicle. However, an object may not correspond to a physical entity, particularly when said object is a false positive (i.e., a misdetection). By "prediction," it is understood that each model is designed to detect or recognize objects by processing the input data. In practice, the prediction is therefore a computational step performed by the on-board computer 1 using the model.

[0032] Each object is more specifically characterized by a spatial representation, here in a reference system (2D or 3D) linked to the vehicle, and at least one score representative of confidence in the prediction of the object. The spatial representation and scores of one or more objects are assigned by the model predicting said object and depend on the training of said model.

[0033] The spatial representation of each object describes its dimensions and location within the reference frame. As will become apparent later, the spatial representations of the different objects are comparable to one another. Here, the spatial representation of each object is a bounding box, for example, an arbitrarily oriented bounding box. When a model is designed to process 2D images, the bounding box of an object predicted by that model is typically defined as the smallest rectangle encompassing the object. Here, each object is thus described in the reference frame by a bounding box composed of values ​​expressing the position of its center, its dimensions, and its orientation.

[0034] Here, each object is also characterized by a category, also assigned by the model that predicts it. The category is the result of a classification performed by the model. The different categories into which the object can be classified are predetermined. The categories correspond to the objects that can be present on or near a road. For example, the categories are: car, bus, truck, motorcycle, bicycle, pedestrian, road sign.

[0035] Each object is characterized by a plurality of scores. The scores are provided by the models when they are asked to perform classification tasks. Each score represents a certainty, that is, a confidence (or conversely, an uncertainty) in the prediction. This confidence is expressed by a numerical value, for example, a floating-point number between 0 and 1. It is therefore comparable, but not strictly equal, to a probability.

[0036] The different scores of an object are relative to its category and its spatial representation, i.e., the values ​​composing its bounding box. Here, the different scores of an object are therefore representative of a confidence in the category classification, a confidence in the estimation of the dimensions, a confidence in the estimation of the location of the center, and a confidence in the estimation of the orientation.

[0037] Alternatively, each object is characterized by a single score, which is for example related to the category of the object (i.e. representative of a confidence in the category classification).

[0038] The models 21, 22, 23 stored in memory 3 are different from each other. This means, in particular, that the models 21, 22, 23 may differ in terms of the type of data they process (e.g., images or radar points), in terms of structure (e.g., a different number of neurons or layers), in terms of training datasets, or in terms of learning protocols.

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] (for example, supervised or unsupervised). Therefore, even when processing the same inputs, two models can produce different outputs. As illustrated in the embodiment of [Fig. 1], the on-board computer 1 also includes a communication means 3 which enables the on-board computer 1 to receive input data. The input data is generated, i.e., supplied, by at least one on-board sensor of the vehicle. In the example shown in [Fig. 3], the on-board computer 1 receives input data from three separate sensors 11, 12, 13, for example, a camera, a radar, and a lidar scanner. One of the sensors may also be an ultrasonic sensor. The on-board computer 1 is also programmed to implement an advanced driver assistance system (ADAS), based on objects detected by the models. The advanced driver assistance system is a system that helps the driver steer or control the vehicle to increase safety, for example: intelligent speed adaptation, lane departure warning, adaptive cruise control, or even autonomous driving. The data stored in memory 3 also includes the matching functions 31, 32, 33 associated with the models. More specifically, they include a matching function by model and by category that the model uses for classification, that is, by category into which the model is designed to classify objects. As described below, the role of each matching function is to transform the scores of an object directly affected by its model, hereafter referred to as raw scores, into normalized scores. Here, each matching function is a function similar to a temperature scaling function of the following form: ii । stand ~ “■ Where Cstand is the object's normalized score, Ci, for i=1,...,Mi are the M raw scores of the object provided by the associated model (representing different confidence levels, e.g., confidence in the category classification, confidence in the size estimation, etc.), and Ai, for i=1,...,Mi are predefined calibration parameters. As detailed below, these calibration parameters are predefined in the sense that they are determined prior to being stored in memory. As an alternative, when the model produces a single score (Craw) for the object (e.g., representing confidence in the category classification), the previously introduced formula takes the following form: i Castand =

[0048] Each matching function is thus defined by its calibration parameters, here by at least two calibration parameters.

[0049] The number of calibration parameters to be defined for all the matching functions is limited by N x M' x T with N the number of models, M' the maximum number of scores provided by a model and T the number of predefined categories that can be detected.

[0050] At this stage, a method for determining the matching functions is described in [Fig. 2]. This determination is an offline procedure, carried out before storing the matching functions on the memory 3 of the on-board computer 1. This method can be carried out by processing means separate from the on-board computer 1. Since the determination of the different matching functions is identical, the determination of one of the matching functions (called the matching function) of one of the models (called the model) for a category (called the category) is described below.

[0051] Determining the correspondence function means determining its calibration parameters.

[0052] The principle of this determination is to compare the model's output with a faithful representation of reality, commonly called ground truth. The matching function is adjusted, that is, optimized, to increase the model's accuracy, accuracy being its ability to predict objects that strictly correspond to real objects.

[0053] As shown in [Fig.2], the method for determining the correspondence function begins with a step e) of initializing the correspondence function.

[0054] In step e), the calibration parameters are defined on the initial values. The initial values ​​are, for example, all defined as one.

[0055] In the following steps of the method, the objective is to determine the calibration parameters by modifying the values ​​of the calibration parameters.

[0056] The method also includes a step f) of providing a calibration data set comprising calibration data and pre-labeled objects.

[0057] The calibration dataset comprises data suitable for object prediction. The calibration dataset is distinct from a dataset on which the model was trained.

[0058] The calibration data are of the same type as those injected into the model when used by the on-board computer 1 (in step b) below). For example, when said model is designed to process 2D images from a camera, the calibration data are also 2D images. Preferably, the calibration dataset is close to the application domain of the on-board computer 1, for example in the sense that the data set is obtained from sensors and perspectives similar to those of the vehicle, so that the normalized scores provided by the matching function are more reliable.

[0059] Pre-labeled objects correspond to physical entities represented in the calibration data that the model must be able to detect. These objects are "pre-labeled" in the sense that they have already been assigned a spatial representation and a category. In practice, pre-labeled objects are labeled manually by an operator who adjusts the selection boxes to the objects and selects their category.

[0060] In other words, the calibration dataset is a labeled dataset in which the pre-labeled objects correspond to the ground truth associated with the calibration data.

[0061] Preferably, the same calibration dataset is used to determine all the matching functions. In this case, when the models are injected with different types of data, the calibration dataset also includes different types of data.

[0062] The method then continues with a step g) of predicting calibration objects by entering the calibration data into the model.

[0063] Only the calibration objects of the category (associated with the matching function) are predicted. Alternatively, objects of all categories of the model can be predicted, but only those of the category are considered for the following steps.

[0064] The method continues with a step h) of determining the normalized scores of the calibration objects. A normalized score is more specifically determined for each calibration object.

[0065] The normalized score of each calibration object is determined on the basis of the raw scores of said calibration object (obtained from step g)) and the correspondence function (obtained from step e) or step j)). The normalized score is calculated according to the formula mentioned above based on the raw scores of the calibration object and on the calibration parameters of the correspondence function.

[0066] The method then includes a step i) of determining efficiency data based on comparisons between the spatial representations, here bounding boxes, of pre-labeled objects, and the spatial representations, here bounding boxes, of calibration objects.

[0067] As shown in [Fig.2], the pre-labeled objects are obtained in step f) because they represent the ground truth of the calibration objects.

[0068] The efficiency data are representative of the model's performance in predicting objects in the category associated with the matching function. For this To do this, the calibration objects of the category are compared to the pre-labeled objects, that is, to the ground truth, of the category.

[0069] By way of example, the efficiency data are determined more precisely as follows.

[0070] For each calibration object, intersection over union (IOU) rates, i.e., Jaccard indices, are calculated between its bounding box and the bounding boxes of the pre-labeled objects (of the category). Alternatively, a single IOU rate is calculated between the bounding box of this calibration object and the bounding box of the pre-labeled object that is spatially closest to this calibration object.

[0071] The calibration object is then considered a true positive if the intersection-on-union ratio between its bounding box and a bounding box of one of the pre-labeled objects is greater than a predetermined threshold. Otherwise (when this condition is not met), the calibration object is considered a false negative. The predetermined threshold is, for example, between 0.5 and 0.7.

[0072] The efficiency data classify each calibration object as a true positive, in which case the value one can be associated with this calibration object, or as a false negative, in which case the value zero can be associated with this calibration object.

[0073] The efficiency data therefore comprise a plurality of values, for example zeros and ones, which may take the form of a vector. More specifically, they comprise one value per calibration object.

[0074] The method finally includes a step j) of adjusting the matching function.

[0075] In step j), a loss function is first defined between the normalized scores of the calibration objects and the efficiency data. This loss function takes into account all the calibration objects (of the category).

[0076] The loss function here is a logistic loss function. Alternatively, the loss function is a square loss function or an exponential loss function.

[0077] The calibration parameters of the correspondence function are then determined by minimizing the loss function, for example using the Nelder-Mead method. This minimization is an iterative process in which the values ​​of the calibration parameters are changed repeatedly and the normalized scores are recalculated so that the loss function is minimized. In [Fig. 2], this iteration is illustrated by a feedback branch from step j) to step h).

[0078] The general idea is to adjust the matching function so that calibration objects that are true positives have a normalized score close to one (because they actually correspond to a real physical entity) and that the false negative calibration objects have a normalized score close to zero (because they do not correspond to a real physical entity).

[0079] The method for comparing the outputs of machine learning detection models processing data acquired from at least one sensor on board the motor vehicle is now described with reference to [Fig.3].

[0080] This method is described here in the context of an online procedure implemented by the processing unit 2 of the on-board computer 1 to analyze the redundant objects to be used by a driver assistance system.

[0081] However, the method can also be used as an offline procedure, generally to estimate the performance of various detection models. The models and matching functions are stored in memory 3 of the onboard computer 1.

[0082] As shown in [Fig. 3], this method begins with a step a) of receiving input data. In the example illustrated in [Fig. 3], the input data is provided by the three on-board sensors 11, 12, 13 of the vehicle.

[0083] The input data may, for example, include first input data provided by a first sensor 11 of the vehicle and second input data provided by a second sensor 12 of the vehicle.

[0084] Remarkably, the method is particularly effective when different sensors are involved since the input data contain rich and varied information.

[0085] The method continues with a step b) of object detection.

[0086] In step b), each model 21, 22, 23 predicts objects based on the input data. In other words, the input data is used as input to models 21, 22, 23 to predict the objects.

[0087] Each model 21, 22, 23 can use input data from one or more sensors 11, 12, 13.

[0088] By way of example, as illustrated in [Fig.3], a first model 21 predicts objects from the first input data provided by the first sensor 11 and a second model 22 predicts objects from the second input data provided by the second sensor 12. As shown in [Fig.3], a third model 23 also uses input data from several sensors 12, 13.

[0089] In step b), the raw scores (i.e. the scores directly assigned by the models) of each object are normalized between zero and one, here by the models themselves.

[0090] At least two objects, predicted by distinct models, are redundant. Here, this generally means that their spatial representations overlap (i.e. (that they are superimposed) in the reference frame. This means that the intersection of their spatial representations is non-zero. In other words, their bounding boxes have a common part. In practice, this means that the two objects represent the same physical entity in the real world.

[0091] Preferably, two objects are considered redundant when the intersection on union of their bounding boxes is greater than a predetermined threshold, for example includes between 50% and 70%.

[0092] By way of example, a first object predicted by the first model 21 and a second object predicted by the second object 22 have overlapping bounding boxes. In this example, the first object is classified as a car with only a first raw score of 0.7 and the second object is also classified as a car, but with only a second raw score of 0.8.

[0093] More generally, each model predicts a set of objects, the sets (N sets in [Fig.3], corresponding to the N models) being redundant because they include objects having superimposed spatial representations.

[0094] The method then includes a step c) of determining the normalized scores of the objects predicted in step b).

[0095] In step c), each of the N sets undergoes score normalization, which includes a matching of the raw scores of the predicted objects. More precisely, a normalized score is calculated for each object. As shown in [Fig. 3], each set of objects is respectively processed using the matching functions 31, 32, 33 associated with the model 21, 22, 23 that predicted it, thanks to the formula described above.

[0096] Following the example above, in step c), the first object can obtain a first normalized score of 0.9 and the second can obtain a second normalized score of 0.85. In this example, this means that the first model 21 is "too pessimistic" since the first normalized score is higher than the first raw score and that the second model 22 is "too optimistic" since the second normalized score is lower than the second raw score.

[0097] The spatial representations of the objects are not modified in step c).

[0098] The normalized scores are comparable to each other because they are adjusted to provide the same interpretation of the confidence in the predictions.

[0099] Such a comparison is carried out in a final step d) of the method.

[0100] In the context of offline processing, step d) may include comparing the normalized score of the redundant objects to determine the accuracy levels of the different models, in order to select the one(s) to be installed in the vehicle. Indeed, since the outputs of the models are calibrated, their performance can be directly compared using the same raw data sets provided by the vehicle's sensors.

[0101] In the context of online processing, step d) may more specifically include merging the sets (the N sets in [Fig.3]) of redundant objects to obtain a single set of non-redundant objects, based on comparisons of normalized scores.

[0102] The fusion can include the following two substeps (carried out by a fusion unit 40 in [Fig.3]).

[0103] First, a new set of predicted objects is obtained as the union of all the sets (the N sets in [Fig.3]).

[0104] Secondly, a duplicate removal method is used to eliminate redundant objects. Here, the onboard computer 1 implements a non-maximal removal (NMS) method that calculates the intersection-on-union overlap between all bounding boxes of all objects predicted in step b) (in the reference frame), and among those overlapping (preferably above a predetermined threshold), selects only one and removes the others according to a criterion based on the normalized score. More specifically, the object with the highest normalized score is kept and the other is removed.

[0105] This means that the fusion consists of selecting, from among the redundant objects, the object with the highest normalized score to be used subsequently in the advanced driver assistance system.

[0106] Following the example above, the first object is selected, and the second is deleted because the first normalized score is greater than the second normalized score.

[0107] As a variant, the merging may consist of removing only the overlaps of the bounding boxes that have the lowest normalized scores.

[0108] As an alternative, the fusion may include obtaining detection using a weighted sum or an average of the redundant objects. For example, when the first object, with a normalized score of 0.9, has a length of 2 meters and the second object, with a normalized score of 0.1, has a length of 3 meters, the length of the final object to be used in the advanced driver assistance system may be calculated as 0.9x2+0.1x3=2.1 meters.

[0109] The result of step d) is a set of non-redundant objects that includes the most probable detections (i.e., the objects) among all those provided by the models. Furthermore, they are assigned reliable confidence scores, namely normalized scores, which can be interpreted as probabilities.

[0110] The method may include an additional step (not shown) of using the set of non-redundant objects, typically the first object, in the advanced driver assistance system.

[0111] The invention is not limited to the embodiment described and illustrated and any variant conforming to the attached claims could be provided.

[0112] In particular, the different models can be supplied with input data from a single sensor. It is always advantageous to process their outputs according to the method described above because said outputs are different (since the models are different). The spatial representations of the objects, which also allow for the calculation of overlaps, can differ from bounding boxes and can, for example, be bounding spheres.

Claims

Demands

1. A computer-implemented method for comparing outputs of machine learning detection models processing data acquired from at least one on-board sensor (11, 12, 13) of a motor vehicle, the method comprising the following steps: a) receiving input data provided by at least one on-board sensor (11, 12, 13) of the motor vehicle, this input data being representative of an environment external to the motor vehicle, b) predicting at least a first object by injecting all or part of this input data into a first machine learning detection model (21) and a second object by injecting all or part of this input data into a second machine learning detection model (22), each object being characterized by a spatial representation and a raw score representative of a prediction confidence,the first object and the second object having overlapping spatial representations, the method being characterized in that it further comprises the following steps: c) determining a first normalized score of the first object on the basis of the raw score of the first object and a first matching function (31) associated with the first machine learning detection model (21) and a second normalized score of the second object on the basis of the raw score of the second object and a second matching function (32) associated with the second machine learning detection model (22), d) comparing the first normalized score and the second normalized score.

2. Method according to claim 1, wherein step d) further comprises merging the spatial representation of the first object and the spatial representation of the second object based on the comparison between the first normalized score and the second normalized score.

3. A method according to claim 2, wherein in step d), the melting comprises one of the following: - select the object, from the first object and the second object, with the highest normalized score, - remove the part of the spatial representation characterizing the object, from the first object and the second object, having the lowest normalized score which overlaps the spatial presentation of the other object, - perform a weighted sum of the spatial representations of the objects.

4. A method according to any one of claims 1 to 3, wherein said input data comprise first input data provided by a first sensor (11) of the motor vehicle and second input data provided by a second sensor (12) of the motor vehicle, the method comprising: - predicting the first object by injecting the first input data into the first machine learning detection model, - predicting the second object by injecting the second input data into the second machine learning detection model.

5. A method according to any one of claims 1 to 4, wherein said input data are of one or more of the following types: radar points, lidar points, images, ultrasound data.

6. A method according to any one of claims 1 to 5, wherein the matching function (31, 32, 33) of each machine learning detection model (21, 22, 23) is determined by means of the following steps: e) initializing said matching function (31, 32, 33), f) providing a calibration dataset distinct from a dataset on which said machine learning detection model (21, 22, 23) has been trained, comprising calibration data and pre-labeled objects, g) predicting calibration objects by inputting the calibration data into said machine learning detection model (21, 22, 23), h) determining the normalized scores of the calibration objects on the basis of the raw scores of the calibration objects and said matching function (31, 32, 33), i) determine efficiency data from comparisons between the spatial representations of the pre-labeled objects and the spatial representations of the calibration objects, j) adjust said correspondence function (31, 32, 33) by minimizing a loss function between the normalized scores of the calibration objects and the efficiency data.

7. Method according to claim 6, wherein the calibration data are of the same type as those injected into said machine learning detection models (21, 22, 23) in step b).

8. Method according to claim 6 or 7, wherein said comparisons between the spatial representations of the pre-labeled objects and the spatial representations of the calibration objects are based on intersection-on-union rates between the spatial representations of the calibration objects and the spatial representations of the pre-labeled objects.

9. Method according to claim 8, wherein, in the efficiency data, each calibration object is considered a true positive if an intersection-on-union rate between the spatial representation of that calibration object and a spatial representation of one of the pre-labeled objects is greater than a predetermined threshold and is considered a false negative otherwise.

10. A computer (1), characterized in that it comprises: H a memory unit (3) storing: - a first machine learning detection model (21) and a second machine learning detection model (22), - a first matching function (31) associated with the first machine learning detection model (21) and a second matching function (32) associated with the second machine learning detection model (22), H a processing unit (2) programmed to implement the following steps: a) receive input data provided by at least one on-board sensor (11, 12, 13) of a motor vehicle, this input data being representative of an environment external to the motor vehicle, b) predict at least a first object by injecting all or part of these input data into the first machine learning detection model (21) and a second object by injecting all or part of these input data into the second machine learning detection model (22), each object being characterized by a spatial representation and a raw score representative of a prediction confidence, the first object and the second object having overlapping spatial representations, c) determine a first normalized score of the first object on the basis of the raw score of the first object and the first matching function (31) and a second normalized score of the second object on the basis of the raw score of the second object and the second matching function (32), d) compare the first normalized score and the second normalized score.

11. Computer-readable storage medium (3) comprising instructions which, when executed by a computer, compel the computer to execute a method according to any one of claims 1 to 9.