Method for assessing the conformity of a tracking system using a limited number of measurements and associated devices

Abstract

Method for assessing the conformity of a tracking system using a limited number of measurements and associated devices The present invention relates to a method for assessing the conformity of a tracking system (10) to a set of requirements, the method comprising at least the following steps: - acquiring a set of values, - for each requirement, calculating a robustness value, the robustness being calculated as the p-value of a statistical test indicating whether the requirement in question is met or not, - for each requirement, comparing each robustness value to a predefined threshold according to a respective comparison criterion, and - determining the requirements with which the tracking system (10) complies, the requirements determined as compliant being the requirements for which, in the comparison step, it has been determined that the comparison criterion is met. Figure for abstract: 1

Description

Title of the invention: Method for assessing the conformity of a tracking system using a limited number of measurements and associated devices FIELD OF THE INVENTION

[0001] The present invention relates to a method for evaluating the conformity of a tracking system to a set of requirements. The present invention also relates to an evaluation module allowing the implementation of the method and a tracking system comprising such an evaluation module. TECHNOLOGICAL BACKGROUND OF THE INVENTION

[0002] The proposed invention lies in the field of air traffic control.

[0003] Airspace control is ensured by monitoring all aircraft. It This involves supervising air traffic to prevent collisions between aircraft and controlling traffic, both in cruising flight and around airports - takeoff and landing.

[0004] To implement such control, air traffic controllers use an air traffic monitoring system based on aircraft tracking, which is called a tracking system. The tracking system is capable in real time of estimating from data coming from sensors the best possible estimate of the position, heading and speed of each aircraft in a given flight information region.

[0005] The flight information region is often designated by the acronym FIR referring to the corresponding English name of “Flight Information Region”.

[0006] The set of estimated data forms a track.

[0007] The quality of the track estimation (tracking) varies depending on the configuration of the tracking system, the quality of the data acquired by the sensors or even environmental data such as the topology of the terrain or the weather.

[0008] In this respect, it is desirable for an air traffic controller to know the quality of the estimation of the tracks reconstructed by the tracking system.

[0009] For this, it is known to use the requirements of the ESASSP standard which is detailed in the document entitled “EUROCONTROL Specification for ATM Surveillance System Performance” (Volume 1) whose 1TSBN is 978-2-87497-022-1 and which was published in March 2012.

[0010] More specifically, a set of requirements defined by this ESASSP standard is evaluated by comparison between the outputs of the tracking system and an estimate of the ideal trajectory of the aircraft.

[0011] However, calculating the requirements of the standard requires obtaining a large quantity of measurement points, typically between 50,000 and 100,000.

[0012] This poses a problem in practice for obtaining an on-the-fly evaluation and in particular detecting possible failures during operation. Summary of the invention

[0013] There is therefore a need for a method for evaluating the quality of tracking carried out by a tracking system with a reduced number of measurements.

[0014] For this purpose, the description describes a method for evaluating the conformity of a tracking system to a set of requirements, at least one requirement being a mandatory requirement and at least one requirement being a recommended requirement, each requirement requiring that a physical quantity be greater than or equal to a threshold or that a physical quantity be less than or equal to a threshold, the evaluation method being implemented by computer, the evaluation method comprising at least the following steps:

[0015] - acquiring a set of values, the set of values comprising the values accessible to the tracking system over a predefined time interval,

[0016] - for each requirement, calculation of a robustness value, the robustness being calculated as the p-value of a statistical test indicating whether the requirement under consideration is met or not,

[0017] - for each requirement, comparison of each robustness value to a threshold predefined according to a respective comparison criterion, and

[0018] - determination of the requirements to which the tracking system complies, the requirements determined to be compliant being those requirements for which, at the comparison stage, it has been determined that the comparison criterion is met.

[0019] According to particular embodiments, the evaluation method has one or more of the following characteristics, taken in isolation or in all technically possible combinations:

[0020] - the method further comprises, for each robustness value not respecting the comparison criterion, a step of determining the cause of non-compliance with the comparison criterion, the cause being either that the requirement is not met or that the number of values is too low to guarantee a predefined reliability threshold for assessing conformity to the requirement in question.

[0021] - when the determined cause is too low a number of values, the method includes a step of deducing a number of additional values to be acquired to obtain an evaluation reliability greater than the predefined reliability threshold.

[0022] - the method comprises a reiteration of the steps of acquisition, calculation, com parison until the number of values is sufficient to obtain an evaluation reliability greater than the predefined reliability threshold or reaches a maximum number predefined with an assessment reliability lower than the predefined threshold.

[0023] - the method further comprises a step of issuing an alert for the requirements not met or for requirements where the number of values in the acquisition, calculation and comparison steps has reached the predefined maximum number with an evaluation reliability lower than the predefined threshold.

[0024] - the maximum number is between 40000 and 60000.

[0025] - during the calculation step, an integral of an incomplete beta function is calculated regularized.

[0026] - during the calculation step, one p-value is equal to the result of the integral and another p-value is a linear function of the result of the integral.

[0027] - during the calculation step, a distribution function of the distribution is calculated reduced centered normal.

[0028] - the method further comprises:

[0029] - a step of applying an evaluation function specific to each requirement to the p-values to obtain evaluation values, and

[0030] - a step of using the evaluation values to obtain an overall score assessing the compliance of the tracking system with all requirements.

[0031] - the number of requirements is greater than or equal to 20.

[0032] - the tracking system provides measurement estimates from data an previous data from multiple sensors, the set of values including measurement estimates.

[0033] The description also describes a module for evaluating the conformity of a tracking system to a set of requirements, at least one requirement being a mandatory requirement and at least one requirement being a recommended requirement, each requirement requiring that a physical quantity be greater than or equal to a threshold or that a physical quantity be less than or equal to a threshold, the evaluation module being suitable for:

[0034] - acquire a set of values, the set of values comprising the values ac transferable to the tracking system over a predefined time interval,

[0035] - for each requirement, calculate a robustness value, the robustness being calculated as the p-value of a statistical test indicating whether the requirement under consideration is met or not,

[0036] - for each requirement, compare each robustness value to a predefined threshold according to a respective comparison criterion, and

[0037] - determining requirements to which the tracking system complies, the requirements determined to be compliant being those requirements for which, at the comparison stage, it has been determined that the comparison criterion is met.

[0038] The description also provides a tracking system comprising a module devaluation.

[0039] In the present description, the expression “suitable for” means indifferently “adapted for”, “adapted to” or “configured for”. Brief description of the drawings

[0040] Characteristics and advantages of the invention will appear on reading the description which follows, given solely by way of non-limiting example, and made with reference to the appended drawings, in which:

[0041] - [Fig.l] [Fig.l] is a schematic representation of an example of a system of tracking in interaction with a set of sensors,

[0042] - [Fig.2] [Fig.2] is a flowchart of an example implementation of a method for assessing the conformity of the tracking system of [Fig.l] to a set of requirements, and

[0043] - [Fig.3] [Fig.3] graphically illustrates the variation of the p-value as a function of the observed value for a given example requirement.

[0044] DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS GENERAL CASE

[0045] [Fig.l] schematically illustrates a tracking system 10 interacting with a set 12 of sensors.

[0046] The tracking system 10 is capable of observing an airspace to determine the trajectories of aircraft passing through an observed space.

[0047] This makes it possible to carry out air traffic control in the space in question.

[0048] For this, the tracking system 10 is capable of collecting data from all of the sensors and analyzing the collected data.

[0049] The tracking system 10 outputs calculated data.

[0050] The calculated data are, for example, trajectories of aircraft in the observed space, i.e. position, speed or heading data for aircraft passing in the observed space.

[0051] The tracking system 10 forms, with the set 12 of sensors, an air traffic monitoring system.

[0052] Such a system is often referred to by the acronym SSTA.

[0053] The set 12 of sensors is capable of obtaining data in the observed space.

[0054] According to the example described, the set 12 of sensors comprises an ADS unit 14, a WAM unit 16 and a radar 18.

[0055] The ADS 14 unit is a cooperative surveillance system for air traffic control and other related applications. An aircraft equipped with an ADS 14 unit determines its position by a global positioning system (GPS) and periodically sends this position and other information to ground stations.

[0056] The abbreviation ADS refers to the corresponding English term for “Automatic Dependent Surveillance” literally meaning “automatic dependency monitoring”.

[0057] Such an ADS unit 14 is sometimes also called a B-unit, the abbreviation ADS-B referring to the corresponding English name of "Automatic Dependent Surveillance-Broadcast" literally meaning "automatic dependency surveillance-multicast".

[0058] A WAM 16 unit uses data from multiple sensors to obtain the location of an aircraft.

[0059] The abbreviation WAM refers to the corresponding English term "Wide Area Multilateration" and designates an aircraft surveillance technology based on the principle of the difference in arrival time which is used at an airport.

[0060] For example, the WAM unit 16 collects data from several ground antennas to apply mathematical calculations to obtain the position of the aircraft.

[0061] A radar 18 makes it possible to detect the presence of aircraft in the sky and to determine their position. The radar 18 emits electromagnetic pulses into the sky, and the detection and location of an aircraft are obtained by analyzing the wave reflected by the aircraft and retransmitted in the direction of the radar 18.

[0062] According to the example of [Fig.l], the tracking system 10 comprises an analysis module 20, an evaluation module 22.

[0063] The analysis module 20 is capable of analyzing the data from the sensors to estimate new data.

[0064] The analysis module 20 thus regularly receives data relating to the aircraft in the observed environment, in particular their altitude, their horizontal position or their speed.

[0065] In particular, the analysis module 20 is capable of estimating the trajectory of an aircraft.

[0066] Such an analysis module 20 is often referred to by the English term “tracker”, literally meaning “follower”.

[0067] The analysis module 20 is also capable of estimating measurements from previous data from the set 12 of sensors.

[0068] The evaluation module 22 is capable of implementing the steps of a method for evaluating the conformity of the tracking system 10 to a set of requirements.

[0069] According to the example described, the evaluation module 22 comprises a calculation sub-unit 24 and an alert sub-unit 26 whose roles will appear in the remainder of the description.

[0070] An example of implementation of the evaluation method is now described in reference to [Fig.2].

[0071] The evaluation method aims to evaluate the conformity of the tracking system 10 to a set of requirements.

[0072] Among this set of requirements, at least one requirement is a mandatory requirement and at least one requirement is a recommended requirement.

[0073] By definition, a mandatory requirement is a requirement whose non-compliance results in the disqualification of the tracking system 10. The term necessary requirement is sometimes used to designate this type of requirement.

[0074] By definition, a recommended requirement is a requirement whose non-compliance does not result in the disqualification of the tracking system 10 but whose compliance is desirable. The term desirable requirement is sometimes used to designate this type of requirement.

[0075] According to the proposed example, the number of requirements is greater than or equal to 20.

[0076] For the remainder of this discussion, it is assumed that each requirement sets a threshold to be met. This means that each requirement can be formulated in the form of a physical quantity greater than or equal to the threshold or a physical quantity less than or equal to the threshold.

[0077] According to the example described, the evaluation method comprises an acquisition step E100, a calculation step E102, a comparison step E104, a first determination step E106, a second determination step E108, a deduction step El 10, an iteration step El 12, a transmission step El 14, an application step El 16 and a use step El 18.

[0078] Among these steps, only the transmission step El 14 is implemented by the alert sub-module 26, the other steps being implemented by the calculation sub-module 24.

[0079] During the acquisition step El00, the evaluation module 22 acquires a set of values.

[0080] The set of values includes the values accessible to the tracking system 10 over a predefined time interval.

[0081] According to the example described, the accessible values are the values coming from the set 12 of sensors.

[0082] In addition, the values also include the data estimated by the tracking system 10 and in particular the measurement estimates obtained from previous data by the analysis module.

[0083] The predefined time interval is chosen according to the requirements related to the conformity assessment.

[0084] For example, if one wishes to make a regular evaluation, the predefined time interval will correspond to the time interval chosen between two evaluations.

[0085] In particular, a predefined time interval between 10 minutes and 1 time. The predefined time interval chosen will depend in practice on the requirement considered.

[0086] As detailed later, the number of values will usually be chosen to allow for near real-time evaluation and is generally lower, or even much lower, than the number of measurement points usually required by standards for evaluating compliance with requirements.

[0087] During the calculation step E102, the evaluation module 22 calculates a robustness value for each requirement.

[0088] For this, robustness is calculated as the p-value of a statistical test indicating whether the requirement considered is met or not.

[0089] A statistical test is a mathematical tool used to check whether data allows a hypothesis to be rejected or not.

[0090] This hypothesis that we seek to reject is called the “null hypothesis”.

[0091] The implementation of the statistical test gives in particular a value named “p-value”, located between 0 and 1, which will be used to reject or not the hypothesis.

[0092] The “p-value” is also called “p-value” in reference to the corresponding English term “p-value”.

[0093] In the following, the term “p-value” is used.

[0094] The p-value is defined as the probability, assuming the null hypothesis is true, of observing the data, or even more "extreme" data.

[0095] As will be explained later, each p-value is associated with a requirement-dependent metric.

[0096] This metric generally includes moments of the assumed probability distributions on the acquired data.

[0097] In such a case, it may be beneficial to calculate each moment first and then the p-values to avoid possible redundant calculations.

[0098] During the comparison step E104, the evaluation module 22 compares each robustness value to a predefined threshold according to a respective comparison criterion.

[0099] In other words, the evaluation module 22 compares the p-value to a predefined threshold.

[0100] Typically, the term "p-value" is compared to a threshold chosen at 5%.

[0101] This value of 5% is given as an example, the threshold depending on the needs of the user of the tracking system 10.

[0102] During the determination step, the evaluation module 22 determines the requirements to which the tracking system 10 complies.

[0103] For this, the evaluation module 22 considers that the requirements determined as compliant are those for which at the comparison step E104, the Assessment module 22 determined that the comparison criterion is met.

[0104] If the p-value is less than the threshold, it is considered very unlikely to have observed such data, given that the null hypothesis is true. This leads to rejecting the hypothesis with good certainty.

[0105] To better explain this idea, the simple case of a coin toss is developed.

[0106] Consider a coin, with one side heads, and one side tails. The following assumption is made: "the coin is balanced, and can therefore land on heads or tails with a probability U2".

[0107] In a first example, the coin is tossed 10 times and 7 heads and 3 tails are obtained. This is the acquired data.

[0108] The question then is whether the null hypothesis can be rejected or not.

[0109] The probability of this outcome is then calculated, as well as more extreme outcomes (namely, the probability that there was a number of stacks between 7 and 10 inclusive). We obtain a probability of 0.17, i.e. a p-value of 0.17.

[0110] This means that calculation step E102 leads to a 17% chance of getting 7 or more heads if the coin were balanced.

[0111] This p-value is greater than 5%, so it is not possible to reject the hypothesis with sufficient certainty.

[0112] In a second example, the coin is tossed 10 times and 9 heads and 1 tail are obtained. This is the acquired data.

[0113] The probability of getting 9 or 10 heads is then calculated, knowing that the coin is balanced. This leads to a p-value of 0.01.

[0114] With the threshold of 0.05, this leads to determining that the hypothesis is to be rejected. Otherwise formulated, it can be considered that, knowing the coin balanced, it is very unlikely to observe so many tails.

[0115] A third example can also illustrate the fact that the p-value also evolves as a function of the number of throws.

[0116] Assume that 1000 throws are made and that 700 tails are obtained.

[0117] In such a case, the p-value is less than 105, which corresponds to the fact that the probability of obtaining 7 or more tails on 10 throws (first example) is much higher than that of obtaining 700 or more tails on 1000 throws, given a balanced coin

[0118] A calculation shows that obtaining 527 tails out of 1000 throws is enough to conclude with a threshold of 5% that the null hypothesis is to be rejected, that is to say that the coin is biased.

[0119] More elaborate examples but based on what has just been described are given in the section "Application to the particular case of the ESASSP standard". In this section, particular cases relating to evaluation criteria of the system are developed. tracking 10.

[0120] In this section it will be explained how to derive from a given requirement the expression for calculating the p-value.

[0121] More precisely, a given requirement generally corresponds to the calculation of a metric and the respect by this metric of a threshold value.

[0122] By analyzing the evolution of the p-value as a function of the value of the requirement, it is possible to derive a suitable expression for this p-value.

[0123] Such construction is done before the implementation of the method by a so-called a priori analysis.

[0124] In this analysis, as explained later, it is also possible to derive the number of values to be obtained to determine each requirement to ensure a high level of confidence on the requirements, while applying to reduced geographical areas and time intervals.

[0125] Such a number of values corresponds to a predefined reliability threshold which will be used during the second determination step El08.

[0126] During the second determination step El08, the evaluation module 22 determines the cause of non-compliance with the comparison criterion for each robustness value not complying with the comparison criterion.

[0127] According to the example described, there are two possible causes.

[0128] The first possible cause is that the requirement is not met.

[0129] Indeed, it may take only a few points for us to already know that the requirement will not be met even if the number of measurement points were higher.

[0130] The second cause corresponds to the fact that the number of acquired values is too low to guarantee a predefined reliability threshold for evaluating conformity to the requirement considered.

[0131] This may be due to lower aircraft traffic than expected in the geographical area in question (off-peak hours or strikes, for example).

[0132] It is assumed for the following that, for at least one requirement, the determined cause is the second cause.

[0133] During the deduction step E1 10, the evaluation module 22 calculates the number of additional values to be acquired to obtain an evaluation reliability greater than the predefined reliability threshold.

[0134] For this, the evaluation module 22 uses the number of points ensuring compliance with the reliability threshold, this number of points having been calculated before the implementation of the method.

[0135] The evaluation module 22 therefore reads the value in a memory of the tracking system 10 and deduces therefrom, for example, by subtraction the number of additional values to be acquired.

[0136] Such a calculation can be carried out in a more elaborate manner by calculating the number of points required to pass the requirements, while dynamically taking into account the values observed so far. This corresponds to a real-time recalculation.

[0137] This is now illustrated for examples developed in the section “application to the particular case of the ESASSP standard”.

[0138] Thus, for the case of update probability type requirements, it is assumed that for the currently observed / / values among n* acquired values, it is possible to calculate a p-value p* on the test which aims to verify whether the minimum required value Pr is exceeded in a statistically significant manner.

[0139] It is then possible to calculate the minimum number of additional values to be acquired so that the requirement is passed with a given threshold. This corresponds to assuming all the next values as good, and giving the minimum value m such that m + k*successes among m + n* reports allow the requirement to be passed with a sufficiently low p-value.

[0140] This number corresponds to the minimum number of new points (new values) necessary to reject the null hypothesis.

[0141] Alternatively or in addition, the evaluation module 22 modifies the spatio-temporal discretization of the zones on which the calculation of the p-value is carried out.

[0142] This spatio-temporal discretization is generally provided with the calculation of the p-value associated with the requirement considered, but can be modified a little without altering the quality of the evaluation of the requirement.

[0143] During the iteration step El 12, at least the acquisition, calculation and comparison steps are repeated until a condition is verified.

[0144] This is indicated by arrow 120 in [Fig.2].

[0145] Other steps such as the deduction step El 10 may also be iterated.

[0146] In the proposed example, the condition is to respect either a first condition or a second condition.

[0147] The first condition is to obtain an evaluation reliability greater than the predefined reliability threshold.

[0148] This is obtained by exceeding the minimum number of additional values to be acquired if this has been determined only once or when the minimum number of additional values to be acquired is determined as zero at an iteration of the deduction step E1 10.

[0149] The second condition is to reach a predefined maximum number with an evaluation reliability lower than the predefined threshold.

[0150] This corresponds to the case where the p-value is too high for the requirement to be met with sufficient certainty, but where the values observed so far seem to corroborate the fact that the requirement is met (formally in the example of , but rc* is too weak for us to have enough update described previously: F > n* * r of certainty).

[0151] It is then possible to calculate the number of points needed to pass the requirement with fairly good certainty, assuming that the success ratio will remain the same. Formally, noting p* = L, we seek the number m of ratios n* (acquired values) such that observing p*m successes among m reports allows obtaining a p-value lower than the required threshold.

[0152] In practice, the predefined maximum number is often between 40,000 and 60,000.

[0153] During the emission step El 14, the evaluation module 22 emits an alert for the requirements not met or for the requirements for which the number of values of the acquisition, calculation, comparison steps has reached the predefined maximum number with an evaluation reliability lower than the predefined threshold.

[0154] These cases correspond to the fact that neither the first condition nor the second condition of the iteration step El 12 are respected.

[0155] The alert is, for example, an audible alert or a visual alert sent to an air traffic controller.

[0156] During the application step El 16, the system applies an evaluation function specific to each requirement to the values p to obtain evaluation values.

[0157] Each evaluation function specific to each requirement is a utility function.

[0158] As explained above, the p-value noted P is interpreted as meaning that the requirement is satisfied if the hypothesis Ho is rejected and therefore if the p-value P is less than a threshold £.

[0159] It is possible to convert the p-value into a utility function giving a value between 0 and 1 depending on the value of the p-value.

[0160] For example, the utility function is a piecewise linear function equal to 1 as soon as P - s (case corresponding to compliance with the requirement) then decreasing linearly to 0 for values of the p-value P greater than the thresholds.

[0161] The linear function is, for example, the following:

[0162] 1- p

[0163] Alternatively, it may be considered to normalize the decreasing utility function by including a supervised learning phase to learn this function.

[0164] Advantageously, a decreasing threshold normalization function is used.

[0165] For example, the evaluation module 22 may impose that if the p-value is greater than a value to be defined by an expert, then the utility is zero.

[0166] The value defined by the expert is determined before the implementation of the method, so that it is accessible to the evaluation module 22 by reading its value in a tracking system memory 10.

[0167] Then, the evaluation module 22 uses the evaluation values to obtain an overall score during the use step El 18.

[0168] The overall score evaluates the compliance of the tracking system 10 with all the requirements.

[0169] Such a score is often called QoS in reference to the English term “Quality of Service” literally meaning “quality of service”.

[0170] For this, the evaluation module 22 can implement different techniques.

[0171] For example, with the previous utility function, the utility is 1 if the p-value is less than the threshold and with very high confidence that the requirement is met, and has a value strictly less than 1 otherwise.

[0172] In doing so, the evaluation module 22 can use an aggregation function.

[0173] According to one example, the aggregation function is a strictly increasing compensatory function.

[0174] By applying this technique only to the case of mandatory requirements, this makes it possible to obtain a QoS score equal to 1 if all the mandatory requirements are passed with high confidence and strictly less than 1 if at least one mandatory requirement is not completely respected.

[0175] According to another example, the use step E1 18 comprises supervised learning of a requirement compliance evaluation function in the form of a piecewise linear function, the linear function comprising at least three pieces, preferably less than six pieces, to obtain learned evaluation functions, applying the learned evaluation functions to the tracking system to obtain evaluation values, and using the evaluation values to obtain an overall score evaluating the compliance of the tracking system 10 with all the requirements.

[0176] The evaluation module 22 then calculates the overall score for each type of requirement by aggregating the evaluation functions of all the requirements of the type considered, the overall score evaluating the conformity of the tracking system to all the requirements being obtained by aggregating the overall scores by type of requirement.

[0177] Alternatively, instead of this calculation, the use step E1 18 comprises supervised learning of aggregation functions of the evaluation values, the aggregation function evaluating the conformity of the tracking system to a part of the requirements of the set of requirements corresponding to a particular quantity, and a use step comprising the use of the learned evaluation values and aggregation functions to obtain a value for each part of the requirements of the set of requirements corresponding to a particular quantity and the use of the values obtained during

[0178]

[0179]

[0180]

[0181]

[0182]

[0183]

[0184]

[0185]

[0186]

[0187]

[0188]

[0189] use as well as aggregation functions to obtain the overall score. These techniques are also applicable to the case of recommended requirements. According to another approach, the evaluation module 22 aggregates the values differently QoS° (value for mandatory requirements) and QoSR (value for recommended requirements) to obtain the overall QoS. The normalized scores are interpreted here as the confidence level that the requirements are met. Given the mandatory nature of the requirements underlying QoS°>, the slightest failure of a mandatory requirement must be reflected in the overall score and cannot be compensated by the recommended requirements. For this purpose, in this approach, the evaluation module 22 uses the following formulas: if QoS° < 1 if QoS°=l In the previous techniques, the aggregation functions are compensatory, which implies that poor compliance with one requirement can be compensated by good compliance with another requirement. In some cases, it may be desirable that this compensation is not possible, particularly with regard to mandatory requirements. As soon as at least one mandatory requirement is not met, it would then no longer be possible to compensate for this non-compliance even if all the other requirements are met. For this, another type of aggregation function called t-norms can be used. These are binary operators T ( x, y) G [ 0,1 ] where 3' are also valued in [0,1]. A binary operator T: [0,1] X [0,1] [0,1] is a t-norm if it satisfies the following properties: • (neutral element) T ( 1, X ) = x for all x G [0,1], • (commutativity) T ( x, y ) = T(y, x) for all x, ye [0,1], • (monotonicity) T ( x, y ) < T(w, v) for 0 < x < u < 1 and 0 < y < v < 1, and • (associativity) T( x, T ( y, z ) ) = T( T( x, y ), z, ) For all x, y, ze [0,1], The function T satisfies certain properties. This function provides a pessimistic evaluation compared to a compensatory function because T (x, y) < min(x, y), that is, it is never possible to have a grade better than the worst grade. Several bad grades worsen the overall grade. It can be shown that the binary operator T : [0,1] X [0,1] [0,1] is a t-

[0190]

[0191]

[0192]

[0193]

[0194]

[0195]

[0196]

[0197]

[0198]

[0199]

[0200]

[0201]

[0202] strict and Archimedean norm if and only if there exists a function f : [0,1] [0, «) continuous, strictly decreasing with / ( 1 ) = 0 such that ^(a; y) =r1(min( / (%) + / (vX / (0) ) )• By definition, a t-norm T is Archimedean if the following two conditions are satisfied: • T is continuous, and • T(x, x) <x pour tout xe (0,1). This binary operator T is easily transformed into an n-ary operator: T(x^ min( / '(0) ) ) Or : . xA, ..., xm are the normalized scores of the p-values of the mandatory requirements, • • • •, ) is the QoS° score, and • the function f is to be constructed by supervised learning. Alternatively, the sum is replaced by an ordered weighted average, which is written mathematically as: T(X], ..., xm)=f^ min(E" f(), / (0))) where is a permutation on [1, ... ,m} ordering the scores as follows: — -^-(2) — • • • — %n(m) The method which has just been described therefore makes it possible to obtain a robust evaluation of the conformity of the tracking system 10 to a set of requirements. In fact, rather than having a simple binary value (criterion met or not), the process provides access to an estimate of a margin with which the criterion is met, that is to say an estimate of the confidence that can be had in the estimate. Furthermore, if there are too few values to have a reliable estimate, the criterion will always be considered unfulfilled, which is desirable in a critical context like air traffic, where it is better to be wrongly alarmist than overconfident. Furthermore, the method ensures that the result cannot change instantly from "not met" to "met", with the p-value evolving smoothly with new data. However, there may be a passage above and below the threshold, in which case the alarm is triggered, but there is always a certain margin to travel before the criterion is actually violated. In addition, the process uses a lower number of points than those recommended in the standard.

[0203] This reduces the amount of memory occupied as well as the stress on the hardware components of the computer system 10.

[0204] The method also has the advantage that the evaluation is adaptive.

[0205] In particular, the method greatly reduces the data acquisition time for a reliable calculation of score, and therefore of the conformity or non-conformity of the criteria. Thus, the data used are less old, and the estimation of the quality of service is therefore more reactive.

[0206] This is beneficial when a problem occurs since in the case of rapid degradation, the metric degrades more quickly, and therefore the operator is informed more quickly.

[0207] The converse is also true, when a problem is solved, the return to normal happens more quickly.

[0208] This means that the method allows extreme drifts to be taken into account.

[0209] When the drift is minor, with the number of data fixed a priori, the method is not able to validate the requirement with high confidence. The number of points can then be temporarily increased to absorb this minor temporary deviation. This avoids raising an alert that is not justified. To increase the number of measurement points, the data collection time is either increased (if this is permitted) or geographical areas are grouped together. Both of these measures are temporary.

[0210] When the drift is major, from the start of data collection over a time interval, the method makes it possible to realize that there is no chance of satisfying the requirement given the excessive number of bad data. Instead of waiting for the end of the information collection interval, the method sends an alert as soon as possible to the operator.

[0211] The method also defines a mechanism for controlling the number of measuring points, this control mechanism being able to be static (determined a priori) and / or dynamic (adapted from the measured values).

[0212] Thus, it is possible to implement the method in real time over a time interval which can be of a fixed length configured in advance or dynamic depending on the rate of acquisition of the values. This makes it possible to detect the presence of an anomaly in the tracking system in real time.

[0213] In each of the examples described, each module or sub-module is each produced in the form of software, or a software brick.

[0214] All the modules or sub-modules are then produced, i.e. in the form of a computer program, also called a computer program product, and are also capable of being recorded on a medium, not shown, that can be read by a computer. The computer-readable medium is, for example, a medium capable of storing information. electronic instructions and to be coupled to a bus of a computer system. For example, the readable medium is an optical disk, a magneto-optical disk, a ROM memory, a RAM memory, any type of non-volatile memory (for example FLASH or NVRAM) or a magnetic card. On the readable medium is then stored a computer program comprising software instructions, stored in a memory executable by a processor.

[0215] In a variant not shown, each module or sub-module is produced in the form of a programmable logic component, such as an FPGA (Field Programmable Gate Array) or an integrated circuit, such as an ASIC (Application Specific Integrated Circuit).

[0216] The invention relates to any technically possible combination of the embodiments described above. APPLICATION OF THE ESASSP STANDARD TO THE SPECIFIC CASE Presentation of the criteria

[0217] The method which has just been described can be applied to any type of standard but will be illustrated in the following using the example of the ESASSP standard which is detailed in the document entitled “EUROCONTROL Specification for ATM Surveillance System Performance” (Volume 1) whose ISBN is 978-2-87497-022-1 and which was published in March 2012.

[0218] The objective of this standard is to ensure the quality required to avoid collisions. This quality aims at better separation between aircraft in (civil) air traffic. In order to increase air capacity, it is essential to have tracking tools that can ensure increasingly fine separation between aircraft. Two standards have been defined to ensure two distance separations: 5 NM and 3 NM.

[0219] The ESASSP standard defines 22 constraints named from RI to R22, i.e. 22 specific metrics. The 22 constraints are briefly explained in the following.

[0220] The first RI constraint concerns the measurement interval for the probability of update (called in English “Measurement Interval for Probability of Update”). The first RI constraint is broken down into two requirements: a mandatory requirement and a recommended requirement.

[0221] The second constraint R2 concerns the probability of updating the horizontal position (called in English “Probability of update of horizontal position”). The second constraint R2 is broken down into two requirements: a mandatory requirement and a recommended requirement.

[0222] The third constraint R3 concerns the ratio of time the aircraft has not been tracked (called in English “ratio of missed 3D position involved in long gaps”). The third constraint R3 corresponds to a mandatory requirement.

[0223] The fourth constraint R4 concerns the square error in horizontal position (called in English "Horizontal position RMS error"). The square error is often considered as a principal error representative of the precision of the measurements. The fourth constraint R4 is broken down into two requirements: a mandatory requirement and a recommended requirement.

[0224] The fifth constraint R5 concerns a ratio of consecutive correlated errors (called in English “Consecutive correlated error ratio”). The fifth constraint R5 corresponds to a recommended requirement.

[0225] The sixth constraint R6 concerns the maximum time interval with close proximity (called in English "Max delta time in close proximity"). Such a constraint corresponds to the fact that, on a radar, very close tracks which are not refreshed at the same time are present. In the presentation to the air traffic controller, there must not be two side-by-side tracks which have distant update dates. This could lead to risks of collision, i.e. serious safety problems. The sixth constraint R6 corresponds to a recommended requirement.

[0226] The seventh constraint R7 concerns the probability of updating the pressure at altitude (called in English “Probability of update of pressure altitude”). The seventh constraint R7 corresponds to a mandatory requirement.

[0227] The eighth constraint R8 relates to the average data age of forwarded pressure altitude. The eighth constraint R8 corresponds to a mandatory requirement.

[0228] The ninth constraint R9 relates to the maximum age of forwarded pressure altitude data (called in English “Max data age of forwarded pressure altitude”). The ninth constraint R9 corresponds to a mandatory requirement.

[0229] The tenth RIO constraint relates to the ratio of incorrect forwarded pressure altitude data. It should be noted that the tenth RIO constraint is an estimate of the error corresponding to a spurious error. The tenth RIO constraint corresponds to a mandatory requirement.

[0230] The eleventh constraint RI 1 concerns the unsigned errors of the pressure at altitude (called in English "Pressure altitude unsigned error"). It should be noted that the eleventh constraint RI 1 is an estimate of the error of type principal error. The eleventh constraint Rll corresponds to a mandatory requirement.

[0231] The twelfth constraint R12 concerns the delay of appearance of the emergency indicators (called in English “Delay of apparition of the emergency indicator / SPI report”). The twelfth constraint R12 corresponds to a mandatory requirement.

[0232] The thirteenth constraint R13 relates to the delay of change in aircraft identification (called in English “Delay of change in Aircraft Id”). The thirteenth constraint RI3 corresponds to a mandatory requirement.

[0233] The fourteenth constraint R14 concerns the probability of updating with a correct identification (called in English “Probability of update of aircraft identity with correct value”). The fourteenth constraint R14 is broken down into two requirements: a mandatory requirement and a recommended requirement.

[0234] The fifteenth constraint RI5 relates to the incorrect identification report of the aircraft (called in English "Ratio of Incorrect Aircraft Identity"). It should be noted that the fifteenth constraint R15 is an estimate of the error of the parasitic error type. The fifteenth constraint RI5 corresponds to a mandatory requirement.

[0235] The sixteenth constraint R16 concerns the quadratic error of the rate of climb or descent (called in English "Rate of climb / descent RMS error"). In this context, the estimated error is of the principal error type. The sixteenth constraint R16 corresponds to a recommended requirement.

[0236] The seventeenth constraint R17 concerns the track velocity RMS error. In this context, the estimated error is of the principal error type. The seventeenth constraint R17 corresponds to a recommended requirement.

[0237] The eighteenth constraint R18 concerns the quadratic error of the tracking velocity angle (called in English "Track velocity angle RMS error"). In this context, the estimated error is of the principal error type. The eighteenth constraint RI8 corresponds to a recommended requirement.

[0238] The nineteenth constraint R19 concerns the density of uncorrelated false target reports. In this context, the estimated error is of the spurious error type. The nineteenth constraint R19 corresponds to a recommended requirement.

[0239] The twentieth constraint R20 concerns the hourly rate of false tracks close to true tracks. In this context, the estimated error is of the correlated error type. The twentieth constraint R20 corresponds to a recommended requirement.

[0240] The twenty-first constraint R21 concerns continuity. The twenty-first constraint R21 corresponds to a recommended requirement.

[0241] The twenty-second constraint R22 concerns manual investigations (manual analyses of the results) which must be carried out when constraints R2, R4, R12 or R13 are not met. The twenty-second constraint R22 is divided into two requirements: a mandatory requirement and a recommended requirement.

[0242] In summary, the following table 1 can be established, the presence of a cross indicating the presence of an associated requirement:

[0243] [Tables 1] Contraintes Nom anglais utilisé dans la norme Unité Exigence obligatoire Exigence recommandée RI Measurement Interval for Pro-bability of Update s X X R2 Probability of update of horizontal position % X X R3 ratio of missed 3D position involved in long gaps % X R4 Horizontal position RMS error m X X R5 Consecutive correlated error ratio % X R6 Max delta time in close proximity s X R7 Probability of update of pressure altitude with correct value % X R8 Average data âge of forwarded pressure altitude s X R9 Max data âge of forwarded pressure altitude s X RIO Ratio of incorrect forwarded pressure altitude % X Rll Pressure altitude unsigned error % X R12 Delay of apparition of the emergency indicator / SPI report s X R13 Delay of change in Aircraft Id s X R14 Probability of update of aircraft identity with correct value % X X R15 Ratio of Incorrect Aircraft Identity % X R16 Rate of climb / descent RMS error m / s X R17 Track velocity RMS error m / s X R18 Track velocity angle RMS error degrés XR19 Density of uncorrelated false target reports number R20 Number per hour of false tracks close to true tracks number

[0244] Table 1: Association for each constraint of its name in English, of T associated unit as well as the presence of a mandatory requirement and a recommended requirement

[0245] In summary, the ESASSP standard is a set of 14 mandatory requirements and 12 recommended requirements, representing 26 requirements in total, these 26 requirements being estimated according to 22 distinct metrics. Calculation of p-values associated with the criteria

[0246] For the calculation of p-values, the criteria defined in the ESASSP can be grouped into several families. These families are five in number and are set out below.

[0247] The first family groups together the criteria linked to update probabilities, these criteria being criteria R2, R7 and R14.

[0248] These criteria R2, R7 and R14 have in common that they relate to a proportion between the number of reports received containing a certain element (horizontal position, pressure altitude, and identity of the aircraft respectively) among all the reports received (which may or may not contain the element in question). These criteria require that the value of the proportion be greater than a threshold value.

[0249] The second family groups together the criteria linked to the root of the mean square error, these criteria being criteria R4, R6, R16, R17 and R18.

[0250] Indeed, the criteria of the second family relate to a calculation of the average Euclidean distance between a value and a reference measurement, often calculated a posteriori by the SSTA. These criteria require that the value of the proportion be lower than a threshold value.

[0251] The third family groups together criteria linked to ratios, these criteria being criteria R3, R5, RIO, RI 1 and R15 and require that a ratio between two quantities has a value lower than a predefined value.

[0252] The fourth family groups together the criteria linked to averages, these criteria being criteria R8, R12 and R13.

[0253] The fifth family groups together criteria that do not fall within the four families already presented. These are criteria R19, R20 and R9.

[0254] It can be noted that these families do not group together all the criteria. In fact, the criterion RI is a parameter of the tracking system 10 which is therefore fixed a priori, the criterion R9 is not a criterion, but a filter on the validity of the data, the criterion involves not having an incident over periods of several years (which cannot be assessed over a short period) and criterion R22 is not a quantitative criterion.

[0255] It is now described how to calculate the probabilities of obtaining a new value not respecting the requirement knowing the acquired values, that is to say the p-values for each of the aforementioned families. First family

[0256] For the first family, it is first explained how to determine the p-values for the criterion R2 involving a simple update probability.

[0257] Criterion R2 involves a metric calculated as the ratio between a number of portions of observed trajectories (noted n) and the number of these trajectories which contain a horizontal position (noted k). The probability is therefore:

[0258] P = k / n

[0259] Where: • n is the number of measurement points (for example the number of sections of identical size in all trajectories, for criterion R2) and • k is the number of measurement points having a certain property (for example the number of trajectory sections in which the tracking system 10 has received at least one horizontal position update from the aircraft, for criterion R2).

[0260] It is assumed that the value of k is a realization of a random variable NR that follows a binomial distribution of probability P, with n trials. This assumes the independence of successive observations. The rule requires that this probability be above a certain threshold (noted P^.

[0261] According to a simple embodiment, the null hypothesis can be chosen as:

[0262] Ho: p .

[0263] The calculation of the p-value noted P can then be implemented using the formula:

[0264] p^P(NR>^G)

[0265] Where Nk is a random variable following a binomial distribution of probability P, with 11 trials.

[0266] This corresponds to the following formulation: "knowing that the actual probability does not meet the criterion, what is the probability that the tracking system 10 observes an equal or greater number of reports containing a horizontal position?". If this value is sufficiently low, it is considered that the observation of k is probably not the result of chance, and that the threshold is therefore passed with a high probability.

[0267] According to a more elaborate embodiment and in particular in the context of a critical system, it is preferable to impose larger margins.

[0268] For this, the null hypothesis becomes Hq: p = p and the calculation of the p-value as for the simple embodiment.

[0269] This corresponds to the following formulation: "Given that the actual probability just meets the criterion, what is the probability that we observe a greater than or equal number of reports containing a horizontal position?".

[0270] Thus, it is more difficult for the system to meet the criterion, and therefore greater robustness of the system is ensured.

[0271] For this criterion, and under these assumptions, the p-value is calculated:

[0272] p - 1 -I ^,[11 - k + 1, k]

[0273] where I is the regularized incomplete beta function.

[0274] The curve of the p-value thus obtained is represented in Figure 3 for requirement R2. The x-axis represents the ratio p = ~, the y-axis represents the p-value. As a reminder, requirement R2 requires that the ratio be greater than or equal to p — 0.97, shown by the dotted vertical line. The dotted horizontal line corresponds to the 0.05 threshold commonly used on p-values. In other words, if the p-value is below this bar, then we have very high confidence that requirement R2 has been met.

[0275] Each curve corresponds to a number of points. Each curve takes the form of a decreasing sigmoid whose inflection point is reached at p — 0.97. We note that the curve is all the steeper around this inflection point as the number of points increases.

[0276] This means that with a lot of points, there is a very high confidence that the R2 requirement is met as long as we are barely above the threshold p — 0.97. On the other hand, for a low number of points, we must be significantly above the threshold to be confident that the requirement is met.

[0277] Such a curve can be produced for each of the requirements and leads to the results which will be detailed below. For the sake of simplicity, these curves are not introduced in what follows to simplify the discussion but the results obtained use these curves which were produced by the Applicant.

[0278] In some cases, it is required that the value calculated for a criterion be reached for all tracked trajectories. This is the case in particular for the “recommended” criterion of criterion R2.

[0279] In this case, the values are used, which are calculated trajectory by trajectory. This means that, for a given trajectory, the values are calculated only on the points belonging to this trajectory.

[0280] Let Pj be the p-value associated with a given trajectory 1 and T the set of trajectories.

[0281] In the case of criterion R2, it is a question of verifying that, for all t GT, the probability of underlying update Pt verifies p > 0.97.

[0282] It is thus calculated for a trajectory ? the p-value noted Pt of the test defined in the previous paragraph, for which only the information of the reports which concern the trajectory t- is used

[0283] It comes like this:

[0284] p = P^tcT, NR>k\H^

[0285] This can be rewritten as:

[0286] n>r =

[0287] Assuming that the probabilities are independent following the different trajectories, it comes: 102881 p=in^ (ip(

[0289] Thus, the p-value of the test aiming to verify that all trajectories verify p > 0.97 is calculated as:

[0290]

[0291] It should be noted that this approach is applicable to all subsequent p-values for which we want to verify that all the trajectories meet a given criterion.

[0292] It is then explained how to determine the p-values for the criteria R7 and R14 involving a double update probability.

[0293] Metrics involving a double update probability are calculated in the same way as for metrics involving a single update probability with one difference. More precisely, the probability is written:

[0294] P = z / k

[0295] Where: • k is the number of reference realizations (e.g., for criterion R7, the number of time intervals where an altitude update is received), and • z is the number of realizations having a certain property (e.g., for criterion R7, the number of time intervals where a non-erroneous update of the altitude)

[0296] In this case, the numbers k and are considered to come from binomial laws. This assumes the independence of successive observations, and the fact that these observations are identically distributed.

[0297] According to a particular embodiment, the number k always corresponds to the number of reports containing a pressure altitude. As for the number z, it is, among these k reports, those whose pressure altitude value is correct (in the case of criterion R7) or an aircraft identifier (criterion R14).

[0298]

[0299]

[0300]

[0301]

[0302]

[0303]

[0304]

[0305]

[0306]

[0307]

[0308]

[0309]

[0310]

[0311]

[0312]

[0313] Thus, the number k follows the binomial distribution Nr of the previous paragraph, with an unknown probability Pr, and the number follows a binomial distribution with parameters k and p- In this context, the p-value is unknown and it is a matter of verifying that its value is greater than a given threshold Pr. Still taking the null hypothesis Ho: ^ — p., the p-value is then calculated as follows: p- 1- J lMP(n-z+ 1, —dl / =0 ' ftî'jéïj-A'^ 0 ' In practice, the integral can be calculated by numerical methods such as a Monte Carlo draw or an approximation method using rectangles. Second family For the second family, it is a matter of calculating the p-value for the root mean square errors. A criterion of the second family is therefore a metric which is a mean square error between an observed value and a reference value. The criterion is met when this metric is less than a given value. Let EQMR be the root mean square error. This square error is written: Or: • And is the squared error on a given quantity for the same measurement. For example, for criterion R4, the quantity is the error in the horizontal position of the aircraft. Assuming that all squared errors Et are identically distributed and independent, and that the corresponding distribution has mean p2 and variance <72 (unknown), the criterion requires that the mean p2 be less than a certain value p2, which is the threshold of the requirement. Let the observed mean squared errors Et. By making the null hypothesis: Hq: p2 = p2, the central limit theorem allows us to obtain the following relation: p = P(EQMR < pr \Hq)= O Or: • is the distribution function of the reduced centered normal distribution. For some criteria, metrics are calculated as the minimum over different trajectories of the mean squared error calculated over each trajectory. This is for example criterion R4.

[0314] The approach is then the same as previously. Third family

[0315] For the third family, as for the first family, there are two types of criteria in this family, a first type called simple ratio gathering the criteria based on a metric involving a variable divided by a constant and a second type called double ratio gathering the criteria based on a metric involving a variable divided by another variable. For both types, the metric must be lower than a predefined threshold.

[0316] First, it is explained how to calculate the p-value for criteria based on a simple ratio, as is the case for criteria R3, R5 and RI 1.

[0317] This calculation is quite similar to the case of the simple update probability since the ratios are of the form:

[0318] R = K

[0319] With: • n a given number (for example, the number of track updates received for criterion R5), and • k a random variable that follows a binomial distribution with parameters n and P (for example, with respect to criterion R, the number of track updates for which at least 3 consecutive horizontal position updates have an error on the same side of the trajectory).

[0320] This involves testing whether P is significantly less than Pr, the required value, in view of the observations.

[0321] Thus, taking as null hypothesis, Hq: p — pr and, for this metric, and under these hypotheses, the p-value is calculated according to the following formula:

[0322] p — I\_p^n - k + 1,

[0323] Here the symmetry with the simple update probability can be noted. This is explained by the fact that, in one case, it is a question of maximizing the simple update probability while here, it is a question of minimizing the simple ratios.

[0324] As regards double ratios, the following approach can be followed. This approach is particularly suitable for the RIO and RI5 criteria.

[0325] In the case of double ratios, the metric is written in the following form:

[0326] R = i

[0327] Where: • NI is the number of reference realizations (e.g., the number of track updates containing a barometric altitude, for the RIO criterion), and

[0328]

[0329]

[0330]

[0331]

[0332]

[0333]

[0334]

[0335]

[0336]

[0337]

[0338]

[0339]

[0340]

[0341] • Nv is the number of realizations having a certain property (for example, the number of track updates containing incorrect barometric altitude, for the RIO criterion). For these metrics, the aim is to verify that the ratio of two binomial variables is less than a threshold. By analogy with simple ratios, the p-value is calculated as the symmetric of those calculated for double update probabilities: P = J l ï-pi ( Nt -Nj, 1 + Nj ) ~~—------dl 0 ■ ' Or: • l plays the role of the observed probability of the intermediate value (i.e. the underlying probability of the distribution that determines the value Nv\ as Pr was in the case of double update probabilities. The previous expression for the p-value corresponds to integrating over all possible values of l, weighted by the likelihood of l given the value of Nv. Fourth family For the fourth family, the criteria involve metrics of the form: M = Or: • X is a variable of interest (for example, the age of the barometric pressure information when it is sent, for criterion R8), given by the ratio i, and • 11 is the number of reports considered. The criterion is met when these averages are below a threshold, noted Assuming the variables X are identically distributed and independent on the ratios, and a sufficient number of ratios, the central limit theorem can be applied. Let the (unknown) mean and standard deviation of the variable X be / / the observed mean and standard deviation. As previously stated, this is that P is smaller than ^r. By making the null hypothesis: p = by the central limit theorem, it comes: Or: • 0 is the cumulative function of the reduced centered normal distribution. Fifth family For the fifth family, each of the remaining criteria are examined. As regards criterion R19, a metric corresponding to the number of false reports obtained for each one-hour interval and for each 900 NM2 area.

[0342] This gives an empirical distribution (representable in the form of a histogram), for each time interval.

[0343] Let D be this distribution, M its mean and 5 its standard deviation.

[0344] This metric is calculated over one hour, this gives at least 450 time intervals since the time interval length is at most 8 seconds.

[0345] Furthermore, the metric is cumulative.

[0346] This implies that:

[0347] K=

[0348] Where: • d, denotes a realization of D on an interval i, • I denotes the set of intervals i which make up the observation time, and • K denotes the number of false trails observed during the hour of observation, on a predefined area of 900 NM2 data.

[0349] Considering a single area of 900 NM2, for A" intervals in one hour, the criterion corresponds to a maximum of nr false tracks.

[0350] A renormalized empirical distribution D* can then be defined. By definition, p* is equivalent to D, but the values are multiplied by .

[0351] It comes like this:

[0352] K

[0353] Where: • denotes a realization of fÿ on an interval i, and • K* is a random variable of the variable of interest (number of false leads) knowing that the distribution of the number of false leads per interval is distributed according to the D* law.

[0354] The expectation of A" is namely the required threshold.

[0355] The null hypothesis is that the number of false trails observed in the area during an interval follows the law

[0356] The p-value is written as follows:

[0357] p =

[0358] Where: • P denotes the p-value, • P denotes the probability, • n> the number of false trails observed in the area during interval i, and • means “follows the law”, and therefore here that follows the law D*.

[0359]

[0360]

[0361]

[0362]

[0363]

[0364]

[0365]

[0366]

[0367]

[0368]

[0369]

[0370]

[0371]

[0372]

[0373]

[0374]

[0375] Since the number of intervals is several hundred intervals i, it is possible to apply the central limit theorem, giving: Or: • $ is the cumulative function of the reduced centered normal distribution. Thus, for each of the z zones of 900NM2, it is possible to calculate the p-value on this zone Pz. As before, it can be verified that it is indeed the maximum of these values on each of the zones which is lower than the threshold. Thus, the global p-value (over all areas ~ e Z where Z is the set of areas) is calculated as follows: Pz~ 1 '^X^l -Pz) Regarding the R20 criterion, a metric is considered corresponding to the number of false tracks close to real tracks for each one-hour interval. The same reasoning as for criterion R19 leads to the following expression for the p-value: p = P(K^< | d- ~D ) Where D' is the renormalized empirical distribution of the number of false leads close to real leads on each of the intervals. As for criterion R9, a similar strategy can be applied. Either : • D the observed distribution of ages of ratios containing a pressure-altitude, • £)* the D distribution rescaled to have a mean of 16 seconds (the required threshold), and • C* the cumulative function of £)*. The p-value of the hypothesis test can then be calculated, the null hypothesis of which is that the age variable of reports containing pressure altitude follows the distribution [) *. Thus comes the following expression for the p-value: Or: • R denotes a set of ratios, each ratio including a pressure altitude, • a denotes a ratio, and • designates the age of the report and is assumed to be independent of other ages.

[0376] Calculation of the optimal value of the number of points for determining a criterion

[0377] The parameters influencing the optimal number of points for each criterion according to the previous families are given in the following table.

[0378] [Tables2] Requirement Family Parameters influencing the optimal number of points R2 1 - number of errors made - number of trajectories - number of errors made per trajectory R3 3 - number of errors made R4 2 - mean - standard deviation - number of trajectories - mean per trajectory - standard deviation per trajectory R5 3 - number of errors made R6 2 - mean - standard deviation R7 1 - number of errors made - number of cases where the reference situation does not occur R8 4 - mean - standard deviation R9 5 - moments on the empirical distributions RIO 3 - number of errors made - number of cases where the reference situation does not occur Rll 3 - number of errors made R12 4 - mean - standard deviation R13 4 - mean - standard deviation R14 1 - number of errors made - number of cases where the reference situation does not occur - number of trajectories - number of errors made per trajectory - number of cases where the reference situation does not occur per trajectory R15 3 - number of errors made - number of cases where the reference situation does not occur R16 2 - mean - standard deviation R17 2 - mean - standard deviation R18 2 - mean - standard deviation R19 5 - empirical mean of past data - empirical standard deviation of past data - number of errors made R20 5 - empirical mean of past data - empirical standard deviation of past data - number of errors made

[0379] Thus, to determine the number of points for each requirement as well as the spatio-temporal discretization, it is necessary to enter the values of the influence parameters.

[0380] For this, it is assumed that we have a sample D of normal acquired values (which must meet all the requirements) representative of the situation being studied (data from the same airport or the same En-Route zone).

[0381] The average number of points needed for the p-value to reach the desired threshold (if the data supports it) is then calculated a priori for each of the requirements. Indeed, if the number of points is too low, the p-value will be too high to reject the null hypothesis with sufficient certainty, even if the data are "perfect" (i.e. they tend to validate the requirement).

[0382] Therefore, assuming perfect data, it is possible to identify the minimum number of points needed for the p-value to fall below the required threshold.

[0383] This then makes it possible to decide whether to perform the calculation in real time, based on a larger amount of data. To do this, with the help of an expert, a compromise is determined between a small amount of data, which allows for a much more responsive system since fewer time intervals are observed, and a large amount of data, which reduces the dynamism of the system, but offers more precision (and therefore a better view of the situation).

[0384] Thus, the identification of the minimum number of points can be implemented by implementing a series of operations.

[0385] During a first operation, the desired maximum monitoring duration is obtained in order to ensure sufficient responsiveness. This maximum duration is provided by the user of the tracking system 10.

[0386] From this duration, a maximum number of Mmax points that can be obtained over this duration is deduced.

[0387] In a second operation, a sample D of normal values is obtained, i.e. a sample meeting all the requirements. These normal values are called normal data in the following.

[0388] In a third operation, for each normal data d D, the values of the parameters influencing the minimum number of points for each requirement are calculated. The minimum number md of points that would be required to meet requirement i on this data d is deduced.

[0389] In practice, three cases will arise.

[0390] In a first case, if mf < Mnuix for a requirement i and any data d, then this is the ideal case where all the data available passes the requirements with very high confidence, using Mmax data.

[0391] One can also use a number of data for requirement i depending on m^md (to which a margin is added).

[0392] According to a second case, nid < Mmax for the large majority of requirements i and the majority of data d, but > Mmax in a few isolated cases.

[0393] In such a case, the number or percentage of data in D that do not pass the requirements with high confidence is quantified. The user sets a tolerance threshold, below which correct data are allowed to raise alerts. In this case, Mmax data is used, or a number of data for KPI i depending on m^xmd.

[0394] In the third case, a significant number of D's data do not pass the requirements.

[0395] An example of a significant number in this context is, for example, half of the total data. Indeed, if one measurement out of two is bad, it can be considered that there is a problem with the operation of the tracking system 10 or at least that its operation should be checked.

[0396] In such a case, this means that these requirements are not compatible with real-time evaluation. This is for example the case of the detection of rare events.

[0397] It may be noted here that another possible gain in terms of the number of data points without extending the acquisition time is to widen the geographical observation area. To do this, two adjacent areas through which little traffic passes may be grouped together, in order to have, over this wider area, sufficient data to accurately estimate whether or not the requirements are met.

[0398] The initial zones (before grouping) are predetermined, notably chosen by the user of the tracking system 10.

[0399] In the following, the number of points necessary to obtain a p-value less than or equal to 0.05 is studied for certain criteria as a function of data degrading compliance with the criterion.

[0400] [Tables3] Criterion Threshold to be reached Number of errors 0 1 2 3 R2 more than 100 points >0.97 99 157 208 257 >0.99 299 473 628 773 R2 less than 100 points >0.97 99 >100 >100 >100 R3 < 0.005 598 947 1258 1549 R5 < 0.0003 9985 15812 20984 25844 Rll (case of a flight at constant altitude) < 0.001 2995 4742 6294 7752 Rll (case of a landing or a takeoff) <0.015 199 315 418 515

[0401] Table 3: Number of points needed to reach a p-value of 0.05 or less for the R2, R5 and RI 1 criteria

[0402] In Table 3, an error is a problematic ratio tending to degrade the criterion.

[0403] [Tables4] ^R «A 0 1 2 3 4 5 10 20 0 57 100 138 173 208 241 401 702 1 81 119 154 188 222 254 411 710 2 100 135 169 202 235 267 422 718 3 116 150 183 215 247 278 431 726 4 130 163 195 227 258 289 440 733 5 143 175 206 238 268 299 449 741 10 195 225 255 285 315 344 490 776 20 273 302 331 360 388 417 558 838

[0404] Table 4: Number of points needed to reach a p-value of 0.05 or less, for criterion R7

[0405] In Table 4, eR is the number of reports not containing a horizontal position and eA is the number of reports containing a horizontal position but not containing a pressure altitude.

[0406] [Tables5] 0 1 2 3 4 5 10 20 0 115 200 276 348 418 485 806 1410 1 163 238 300 375 444 510 825 1422 2 200 271 339 405 470 534 844 1438 3 232 299 365 430 493 556 863 1453 4 260 325 389 453 515 577 881 1468 5 285 349 412 474 536 597 898 1462 10 388 448 508 567 626 685 977 1549 20 539 596 654 711 768 825 1107 1666

[0407] Table 5: Number of points needed to reach a p-value of 0.05 or less, for criterion R14

[0408] In Table 5, eR is the number of ratios not containing a horizontal position and eA is the number of ratios.

[0409] [Tableauxô] pR Ea 0 1 2 3 4 5 10 20 100 1.0 3000 4800 6300 7800 9200 10600 17000 29100 ND 0.98 3100 4900 6500 8000 9400 10800 17400 29700 ND 0.96 3200 5000 6600 8100 9600 11000 17700 30300 ND 0.9 3400 5300 7000 8700 10200 11700 18900 32300 ND 0.75 4000 6400 8400 10400 12300 14100 22700 38800 ND 0.5 6000 9500 12600 15600 18400 21100 34000 ND ND 0.25 12000 19000 25200 31100 36700 42100 NA NA NA

[0410] Table 6: Number of points needed to reach a p-value of 0.05 or less, for the RIO criterion

[0411] In Table 6, the number of points required is rounded to the nearest hundred. In addition, Pg denotes the proportion of reports containing a pressure altitude (resp aircraft identifier), is the number of reports containing invalid pressure altitude and ND means that this number is greater than 50,000.

[0412] Analysis of Tables 3 to 6 shows that, in most cases, the optimal number of points is less than 50,000.

[0413] The only cases where the optimal number exceeds 50,000 are cases where the number of errors is very large.

[0414] Since a large number of errors is a relatively rare event, these tables show that the method allows, in normal operations, to obtain evaluations of the conformity of the criteria with a reduced number of measurements.

Claims

Claims

1. A method for evaluating the conformity of a tracking system (10) to a set of requirements, at least one requirement being a mandatory requirement and at least one requirement being a recommended requirement, each requirement requiring that a physical quantity be greater than or equal to a threshold or that a physical quantity be less than or equal to a threshold, the evaluation method being implemented by computer, the evaluation method comprising at least the following steps: - acquiring a set of values, the set of values comprising the values accessible to the tracking system (10) over a predefined time interval, - for each requirement, calculating a robustness value, the robustness being calculated as the p-value of a statistical test indicating whether the requirement in question is met or not, - for each requirement, comparing each robustness value to a predefined threshold according to a respective comparison criterion,and - determining the requirements to which the tracking system (10) complies, the requirements determined to be compliant being the requirements for which, at the comparison step, it has been determined that the comparison criterion is met.,

2. Method according to claim 1, in which the method further comprises, for each robustness value not meeting the comparison criterion, a step of determining the cause of the non-compliance with the comparison criterion, the cause being either that the requirement is not met or that the number of values is too low to guarantee a predefined reliability threshold for evaluating compliance with the requirement in question.

3. Method according to claim 2, in which, when the determined cause is a number of values that is too low, the method comprises a step of deducing a number of additional values to be acquired to obtain an evaluation reliability greater than the predefined reliability threshold.

4. Method according to claim 3, in which the method comprises a reiteration of the steps of acquisition, calculation, comparison until the number of values allows obtaining an evaluation reliability greater than the predefined reliability threshold or reaches a predefined maximum number with an evaluation reliability lower than the predefined threshold.

5. Method according to claim 4, in which the method further comprises a step of issuing an alert for requirements not met or for requirements for which the number of values of the acquisition, calculation, comparison steps has reached the predefined maximum number with an evaluation reliability lower than the predefined threshold.

6. The method of claim 4 or 5, wherein the maximum number is between 40000 and 60000.

7. A method according to any one of claims 1 to 6, wherein, in the calculating step, an integral of a regularized incomplete beta function is calculated.

8. The method of claim 7, wherein in the calculating step, one p-value is equal to the result of the integral and another p-value is a linear function of the result of the integral.

9. A method according to any one of claims 1 to 8, wherein in the calculating step, a distribution function of the reduced centered normal distribution is calculated.

10. Method according to any one of claims 1 to 9, in which the method further comprises: - a step of applying an evaluation function specific to each requirement to the values p to obtain evaluation values, and - a step of using the evaluation values to obtain an overall score evaluating the conformity of the tracking system (10) to all the requirements.

11. The method of any one of claims 1 to 10, wherein the number of requirements is greater than or equal to 20.

12. A method according to any one of claims 1 to 11, wherein the tracking system (10) provides measurement estimates from past data from a plurality of sensors (12), the set of values comprising the measurement estimates.

13. Evaluation module (22) of the conformity of a tracking system (10) to a set of requirements, at least one requirement being a mandatory requirement and at least one requirement being a recommended requirement, each requirement requiring that a physical quantity be greater than or equal to a threshold or that a physical quantity be less than or equal to a threshold, the evaluation module (22) being capable of: - acquiring a set of values, the set of values comprising the values accessible to the tracking system (10) over a time interval predefined, - for each requirement, calculate a robustness value, robustness being calculated as the p-value of a statistical test indicating whether the requirement considered is met or not, - for each requirement, compare each robustness value to a predefined threshold according to a respective comparison criterion, and - determining requirements to which the tracking system (10) conforms, the requirements determined to conform being the requirements for which, at the comparison step, it has been determined that the comparison criterion is met.

14. A tracking system (10) comprising an evaluation module (22) according to claim 13.

Images