Method for evaluating the compliance of a tracking system using a restricted number of measurements and associated devices

Abstract

The present invention relates to a method for evaluating the compliance of a tracking system (10) with a set of requirements, the method comprising at least the following steps: - acquiring a set of values, - for each requirement, calculating a robustness value, the robustness being calculated as the value p of a statistical test indicating whether or not the requirement considered is met, - for each requirement, comparing each robustness value with a predefined threshold according to a respective comparison criterion, and - determining the requirements with which the tracking system (10) complies, with the requirements determined as being compliant being the requirements for which it was determined that the comparison criterion is met in the comparison step.

Description

[0001] Method for assessing the conformity of a tracking system using a limited number of measurements and associated devices FIELD OF THE INVENTION The present invention relates to a method for assessing the conformity of a tracking system to a set of requirements. The present invention also relates to an assessment module enabling the implementation of the method and a tracking system comprising such an assessment module. TECHNOLOGICAL BACKGROUND OF THE INVENTION The proposed invention lies in the field of air traffic control. Airspace control is ensured by monitoring all aircraft. This involves supervising air traffic to prevent collisions between aircraft and controlling traffic, both in cruising flight and around airports – takeoff and landing. To implement such control,Air traffic controllers use an air traffic monitoring system based on aircraft tracking called a tracking system. The tracking system is capable of estimating in real time from data coming from sensors the best possible estimate of the position, heading and speed of each aircraft in a given flight information region. The flight information region is often referred to by the acronym FIR referring to the corresponding English term for "Flight Information Region". All the estimated data form a track. The quality of the track estimation (tracking) varies depending on the configuration of the tracking system, the quality of the data acquired by the sensors or environmental data such as the topology of the terrain or the weather. As such, it is desirable for an air traffic controller to know the quality of the estimation of the tracks reconstructed by the tracking system. To do this,it is known to use the requirements of the ESASSP standard which is detailed in the document entitled "EUROCONTROL Specification for ATM Surveillance System Performance" (Volume 1) whose ISBN is 978-2-87497-022-1 and which was published in March 2012. More precisely, a set of requirements defined by this ESASSP standard is evaluated by comparison between the outputs of the tracking system and an estimate of the ideal trajectory of the aircraft. However, the calculation of the requirements of the standard requires obtaining a large quantity of measurement points, typically between 50,000 and 100,000. This poses a problem in practice for obtaining an on-the-fly evaluation and in particular detecting possible failures during operation. SUMMARY OF THE INVENTION There is therefore a need for a method for evaluating the quality of the tracking carried out by a tracking system with a reduced number of measurements. For this purpose,the description describes a method for assessing the conformity of a tracking system to a set of requirements, at least one requirement being a mandatory requirement and at least one requirement being a recommended requirement, each requirement requiring that a physical quantity be greater than or equal to a threshold or that a physical quantity be less than or equal to a threshold, the assessment method being implemented by computer, the assessment method comprising at least the following steps: - acquisition of a set of values, the set of values ​​comprising the values ​​accessible to the tracking system over a predefined time interval, - for each requirement, calculation of a robustness value, the robustness being calculated as the p-value of a statistical test indicating whether the requirement considered is met or not, - for each requirement, comparison of each robustness value with a predefined threshold according to a respective comparison criterion,and - determining the requirements with which the tracking system complies, the requirements determined as compliant being the requirements for which, at the comparison step, it was determined that the comparison criterion is met. According to particular embodiments, the evaluation method has one or more of the following characteristics, taken in isolation or in all technically possible combinations: - the method further comprises, for each robustness value not meeting the comparison criterion, a step of determining the cause of the non-compliance with the comparison criterion, the cause being either that the requirement is not met or that the number of values ​​is too low to guarantee a predefined reliability threshold for assessing compliance with the requirement in question. - when the determined cause is too low a number of values,the method comprises a step of deducing a number of additional values ​​to be acquired to obtain an evaluation reliability greater than the predefined reliability threshold. - the method comprises a reiteration of the acquisition, calculation, and comparison steps until the number of values ​​either allows obtaining an evaluation reliability greater than the predefined reliability threshold or reaches a predefined maximum number with an evaluation reliability lower than the predefined threshold. - the method further comprises a step of issuing an alert for requirements not met or for requirements for which the number of values ​​of the acquisition, calculation, and comparison steps has reached the predefined maximum number with an evaluation reliability lower than the predefined threshold. - the maximum number is between 40,000 and 60,000. - during the calculation step, an integral of a regularized incomplete beta function is calculated. - during the calculation step,a p-value is equal to the result of the integral and another p-value is a linear function of the result of the integral. - during the calculation step, a distribution function of the reduced centered normal distribution is calculated. - the method further comprises: - a step of applying an evaluation function specific to each requirement to the p-values ​​to obtain evaluation values, and - a step of using the evaluation values ​​to obtain an overall score evaluating the conformity of the tracking system to the set of requirements. - the number of requirements is greater than or equal to 20. - the tracking system provides measurement estimates from previous data from several sensors, the set of values ​​comprising the measurement estimates. The description also describes a module for evaluating the conformity of a tracking system to a set of requirements,at least one requirement being a mandatory requirement and at least one requirement being a recommended requirement, each requirement requiring that a physical quantity be greater than or equal to a threshold or that a physical quantity be less than or equal to a threshold, the evaluation module being capable of: - acquiring a set of values, the set of values ​​comprising the values ​​accessible to the tracking system over a predefined time interval, - for each requirement, calculating a robustness value, the robustness being calculated as the p-value of a statistical test indicating whether the requirement in question is met or not, - for each requirement, comparing each robustness value to a predefined threshold according to a respective comparison criterion, and - determining requirements with which the tracking system complies, the requirements determined as compliant being the requirements for which, at the comparison step,it has been determined that the comparison criterion is met. The description also provides a tracking system comprising an evaluation module. In the present description, the expression "specific to" means indifferently "adapted for", "adapted to" or "configured for". BRIEF DESCRIPTION OF THE DRAWINGS Characteristics and advantages of the invention will appear on reading the description which follows, given solely by way of non-limiting example, and made with reference to the appended drawings, in which: - Figure 1 is a schematic representation of an example of a tracking system in interaction with a set of sensors, - Figure 2 is a flowchart of an example of implementation of a method for evaluating the conformity of the tracking system of Figure 1 to a set of requirements,and - Figure 3 graphically illustrates the variation of the p-value as a function of the observed value for a given exemplary requirement. DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS GENERAL CASE Figure 1 schematically illustrates a tracking system 10 interacting with a set 12 of sensors. The tracking system 10 is capable of observing an airspace to determine the trajectories of aircraft passing through an observed space. This makes it possible to carry out air traffic control in the space considered. For this, the tracking system 10 is capable of collecting data from all the sensors and analyzing the collected data. The tracking system 10 outputs calculated data. The calculated data are, for example, aircraft trajectories in the observed space, i.e. position data,of speed or heading for aircraft passing in the observed space. The tracking system 10 forms with the set 12 of sensors an air traffic surveillance system. Such a system is often referred to by the acronym SSTA. The set 12 of sensors is suitable for obtaining data in the observed space. According to the example described, the set 12 of sensors comprises an ADS unit 14, a WAM unit 16 and a radar 18. The ADS unit 14 is a cooperative surveillance system for air traffic control and other related applications. An aircraft equipped with an ADS unit 14 determines its position by a satellite positioning system (GPS) and periodically sends this position and other information to ground stations. The abbreviation ADS refers to the corresponding English name of "Automatic Dependent Surveillance" literally meaning "automatic dependence surveillance". Such an ADS unit 14 is sometimes also called a B-unit,the abbreviation ADS-B referring to the corresponding English name of "Automatic Dependent Surveillance-Broadcast" literally meaning "automatic dependency surveillance-multicast". A WAM unit 16 uses data from several sensors to obtain the location of an aircraft. The abbreviation WAM refers to the corresponding English name of "Wide Area Multilateration" literally meaning "wide area multilateralization" and designates an aircraft surveillance technology based on the principle of time difference of arrival which is used at an airport. For example, the WAM unit 16 collects data from several antennas on the ground to apply mathematical calculations to obtain the position of the aircraft. A radar 18 makes it possible to detect the presence of aircraft in the sky and to determine their position. The radar 18 emits electromagnetic pulses into the sky,and the detection and location of an aircraft are obtained by analyzing the wave reflected by the aircraft and retransmitted in the direction of the radar 18. According to the example of FIG. 1, the tracking system 10 comprises an analysis module 20, an evaluation module 22. The analysis module 20 is capable of analyzing the data from the sensors to estimate new data. The analysis module 20 thus regularly receives data relating to the aircraft in the observed environment, in particular their altitude, their horizontal position or their speed. In particular,the analysis module 20 is capable of estimating the trajectory of an aircraft. Such an analysis module 20 is often referred to as a “tracker” literally meaning “follower”. The analysis module 20 is also capable of estimating measurements from previous data from the set 12 of sensors. The evaluation module 22 is capable of implementing the steps of a method for evaluating the conformity of the tracking system 10 to a set of requirements. According to the example described, the evaluation module 22 comprises a calculation sub-unit 24 and an alert sub-unit 26 whose roles will appear in the remainder of the description. An example of implementation of the evaluation method is now described with reference to FIG. 2. The evaluation method aims to evaluate the conformity of the tracking system 10 to a set of requirements. Among this set of requirements,at least one requirement is a mandatory requirement and at least one requirement is a recommended requirement. By definition, a mandatory requirement is a requirement the non-compliance of which results in the disqualification of the tracking system 10. The term necessary requirement is sometimes used to designate this type of requirement. By definition, a recommended requirement is a requirement the non-compliance of which does not result in the disqualification of the tracking system 10 but the compliance of which is desirable. The term desirable requirement is sometimes used to designate this type of requirement. According to the example proposed, the number of requirements is greater than or equal to 20. For the following, it is assumed that each requirement sets a threshold to be respected. This means that each requirement can be formulated in the form of a physical quantity greater than or equal to the threshold or a physical quantity less than or equal to the threshold. According to the example described, the evaluation method comprises an acquisition step E100,a calculation step E102, a comparison step E104, a first determination step E106, a second determination step E108, a deduction step E110, an iteration step E112, a transmission step E114, an application step E116 and a use step E118. Among these steps, only the transmission step E114 is implemented by the alert sub-module 26, the other steps being implemented by the calculation sub-module 24. During the acquisition step E100, the evaluation module 22 acquires a set of values. The set of values ​​comprises the values ​​accessible to the tracking system 10 over a predefined time interval. According to the example described, the accessible values ​​are the values ​​originating from the set 12 of sensors. In addition,the values ​​also include the data estimated by the tracking system 10 and in particular the measurement estimates obtained from previous data by the analysis module. The predefined time interval is chosen according to the requirements related to the conformity assessment. For example, if we wish to carry out a regular assessment, the predefined time interval will correspond to the time interval chosen between two assessments. In particular, we can choose a predefined time interval between 10 minutes and 1 hour. The predefined time interval chosen will depend in practice on the requirement considered. As detailed later, the number of values ​​will usually be chosen to allow an assessment in near real time and is generally lower, or even much lower, than the number of measurement points usually required by the standards for assessing compliance with requirements. During the calculation step E102,The evaluation module 22 calculates a robustness value for each requirement. For this, robustness is calculated as the p-value of a statistical test indicating whether the requirement considered is met or not. A statistical test is a mathematical tool used to check whether data allows a hypothesis to be rejected or not. This hypothesis that we seek to reject is called the "null hypothesis". The implementation of the statistical test gives in particular a value called the "p-value", located between 0 and 1, which will be used to reject or not the hypothesis. The "p-value" is also called the "p-value" in reference to the corresponding English term "p-value". In the following, the term "p-value" is used. The p-value is defined as the probability, assuming that the null hypothesis is true, of observing the data, or even more "extreme" data. As will be explained later,each p-value is associated with a metric depending on the requirement. This metric generally comprises moments of the assumed probability distributions on the acquired data. In such a case, it may be beneficial to calculate each moment first and then the p-values ​​to avoid possible redundant calculations. During the comparison step E104, the evaluation module 22 compares each robustness value to a predefined threshold according to a respective comparison criterion. In other words, the evaluation module 22 compares the p-value to a predefined threshold. Typically, the term “p-value” is compared to a threshold chosen at 5%. This value of 5% is given as an example, the threshold depending on the needs of the user of the tracking system 10. During the determination step, the evaluation module 22 determines the requirements to which the tracking system 10 complies. For this,the evaluation module 22 considers that the requirements determined as compliant are those for which at the comparison step E104, the evaluation module 22 determined that the comparison criterion is met. If the p-value is lower than the threshold, it is considered that it is very unlikely to have observed such data, knowing that the null hypothesis is true. This leads to rejecting the hypothesis with good certainty. To better explain this idea, the simple case of a coin toss is developed. Consider a coin, with one side heads, and one side tails. The following hypothesis is made: "the coin is balanced, and can therefore fall on heads or on tails with a probability ½". According to a first example, the coin is tossed 10 times and 7 heads and 3 tails are obtained. This is the acquired data. The question is then to know whether the null hypothesis can be rejected or not. The probability of this result is then calculated,as well as more extreme results (namely, the probability that we had a number of heads between 7 and 10 inclusive). We obtain a probability of 0.17, that is, a p-value of 0.17. This means that the calculation step E102 leads to a 17% chance of obtaining 7 heads or more if the coin were fair. This p-value is greater than 5%, so it is not possible to reject the hypothesis with sufficient certainty. In a second example, the coin is tossed 10 times and 9 heads and 1 tail are obtained. This is the acquired data. The probability of obtaining 9 or 10 heads is then calculated, given that the coin is fair. This leads to a p-value of 0.01. With the threshold of 0.05, this leads to the determination that the hypothesis is to be rejected. Otherwise formulated, it can be considered that, given that the coin is fair,it is very unlikely to observe that many tails. A third example can also illustrate the fact that the p-value also changes as a function of the number of tosses. Suppose that 1000 tosses are made and that 700 tails are obtained. In such a case, the p-value is less than 10, -5, which corresponds to the fact that the probability of obtaining 7 or more tails out of 10 tosses (first example) is much higher than that of obtaining 700 or more tails out of 1000 tosses, given a balanced coin. A calculation shows that obtaining 527 tails out of 1000 tosses is enough to conclude with a threshold of 5% that the null hypothesis is to be rejected, i.e. that the coin is biased. More elaborate examples but based on what has just been described are given in the section "Application to the particular case of the ESASSP standard". In this section, particular cases relating to evaluation criteria of the tracking system 10 are developed. In this section, it will be explained how to derive from a given requirement the expression for calculating the p-value. More precisely, a given requirement generally corresponds to the calculation of a metric and the respect by this metric of a threshold value.By analyzing the evolution of the p-value as a function of the value of the requirement, it is possible to derive a suitable expression for this p-value. Such a construction is done before the implementation of the method by a so-called a priori analysis. In this analysis, as explained later, it is also possible to derive the number of values ​​to be obtained to determine each requirement to ensure a high level of confidence on the requirements, while applying to reduced geographical areas and time intervals. Such a number of values ​​corresponds to a predefined reliability threshold which will be used during the second determination step E108. During the second determination step E108, the evaluation module 22 determines the cause of non-compliance with the comparison criterion for each robustness value not complying with the comparison criterion. According to the example described, there are two possible causes.The first possible cause corresponds to the fact that the requirement is not met. Indeed, it may be enough to have a few points for it to be known that the requirement will not be met even if the number of measurement points were higher. The second cause corresponds to the fact that the number of acquired values ​​is too low to guarantee a predefined reliability threshold for assessing compliance with the requirement in question. This may be due to aircraft traffic lower than expected in the geographical area in question (off-peak hours or strikes for example). It is assumed for the following that, for at least one requirement, the determined cause is the second cause. During the deduction step E110, the evaluation module 22 calculates the number of additional values ​​to be acquired to obtain an assessment reliability higher than the predefined reliability threshold.To do this, the evaluation module 22 uses the number of points ensuring compliance with the reliability threshold, this number of points having been calculated before the implementation of the method. The evaluation module 22 therefore reads the value in a memory of the tracking system 10 and deduces therefrom, for example, by subtraction the number of additional values ​​to be acquired. Such a calculation can be carried out in a more elaborate manner by calculating the number of points necessary to meet the requirements, while dynamically taking into account the values ​​observed until then. This corresponds to a recalculation in real time. This is now illustrated for examples developed in the section "application to the particular case of the ESASSP standard". Thus, for the case of update probability type requirements, it is assumed that for the moment observed ^^. ∗ values ​​among ^^ ∗ acquired values, it is possible to calculate a p-value ^^ ∗on the test which aims to verify if the minimum value ^^ ^^required is exceeded in a statistically significant manner. It is then possible to calculate the minimum number of additional values ​​to be acquired for the requirement to be passed with a given threshold. This corresponds to assuming all the next values ​​as good, and giving the minimum value ^^ such that ^^ + ^^∗successes among ^^ + ^^∗ ratios allow the requirement to be passed with a sufficiently low p-value. This number corresponds to the minimum number of new points (new values) necessary to reject the null hypothesis. Alternatively or in addition, the evaluation module 22 modifies the spatio-temporal discretization of the areas on which the p-value calculation is performed. This spatio-temporal discretization is generally provided with the calculation of the p-value associated with the requirement considered, but can be modified a little without altering the quality of the requirement evaluation.During the iteration step E112, at least the acquisition, calculation and comparison steps are reiterated until a condition is verified. This is indicated by the arrow 120 in Figure 2. Other steps such as the deduction step E110 may also be iterated. In the proposed example, the condition is to meet either a first condition or a second condition. The first condition is to obtain an evaluation reliability greater than the predefined reliability threshold. This is obtained by exceeding the minimum number of additional values ​​to be acquired if this has been determined only once or when the minimum number of additional values ​​to be acquired is determined as zero at an iteration of the deduction step E110. The second condition is to reach a predefined maximum number with an evaluation reliability lower than the predefined threshold.This corresponds to the case where the p-value is too high for the requirement to be met with sufficient certainty, but where the observed values ​​so far seem to corroborate that the requirement is met (formally in the update example ∗ described previously:. ^^ > ^^ , my ∗ ^ ^∗ ^^ is ^^ is too low for us to have enough certainty). It is then possible to calculate the number of points needed to pass the requirement with a fairly good certainty, assuming that the success ratio will remain the same. Formally, by noting ^^ = ^ ∗ me ∗ ^ ^ ^∗ , we seek the number ^^ of reports (acquired values) such that observe ^^ ∗^^ success among ^^ reports allows to obtain a p-value lower than the required threshold. In practice, the predefined maximum number is often between 40000 and 60000. During the emission step E114, the evaluation module 22 emits an alert for the requirements not met or for the requirements for which the number of values ​​of the acquisition, calculation, comparison steps has reached the predefined maximum number with an evaluation reliability lower than the predefined threshold. These cases correspond to the fact that neither the first condition nor the second condition of the iteration step E112 are met. The alert is, for example, an audible alert or a visual alert sent to an air traffic controller. During the application step E116, the system applies an evaluation function specific to each requirement to the p-values ​​to obtain evaluation values. Each evaluation function specific to each requirement is a utility function.As explained above, the p-value noted ^^ is interpreted as meaning that the requirement is satisfied if the hypothesis H0 is rejected and therefore if the p-value ^^ is less than a threshold ^^. It is possible to convert the p-value into a utility function giving a value between 0 and 1 depending on the value of the p-value. For example, the utility function is a piecewise linear function equal to 1 as soon as ^^ ≤ ^^ (case corresponding to compliance with the requirement) then decreasing linearly to 0 for values ​​of the p-value ^^ greater than the threshold ^^. The linear function is, for example, the following: 1. − ^^Alternatively, it may be envisaged to normalize the decreasing utility function by including a supervised learning phase to learn this function. Advantageously, a decreasing normalization function with thresholds is used. For example, the evaluation module 22 may impose that if the p-value is greater than a value to be defined by an expert, then the utility is zero. The value defined by the expert is determined before the implementation of the method, so that it is accessible to the evaluation module 22 by reading its value in a memory of the tracking system 10. Then, the evaluation module 22 uses the evaluation values ​​to obtain an overall score during the use step E118. The overall score evaluates the compliance of the tracking system 10 with all the requirements. Such a score is often called QoS with reference to the English term “Quality of Service” literally meaning “quality of service”.To this end, the evaluation module 22 may implement different techniques. For example, with the previous utility function, the utility is equal to 1 if the p-value is less than the threshold and with very high confidence that the requirement is met, and has a value strictly less than 1 otherwise. In doing so, the evaluation module 22 may use an aggregation function. According to one example, the aggregation function is a strictly increasing compensatory function. By applying only this technique to the case of mandatory requirements, this makes it possible to obtain a QoS score equal to 1 if all the mandatory requirements are met with high confidence and strictly less than 1 if at least one mandatory requirement is not completely met.According to another example, the use step E118 comprises the supervised learning of a requirement compliance assessment function in the form of a piecewise linear function, the linear function comprising at least three pieces, preferably less than six pieces, to obtain learned assessment functions, the application of the learned assessment functions to the tracking system to obtain assessment values, and the use of the assessment values ​​to obtain an overall score assessing the compliance of the tracking system 10 with all the requirements. The assessment module 22 then calculates the overall score for each type of requirement by aggregating the assessment functions of all the requirements of the type considered, the overall score assessing the compliance of the tracking system with all the requirements being obtained by aggregating the overall scores by type of requirement.Alternatively, instead of this calculation, the use step E118 comprises the supervised learning of aggregation functions of the evaluation values, the aggregation function evaluating the compliance of the tracking system with a part of the requirements of the set of requirements corresponding to a particular quantity, and - a use step comprising the use of the learned evaluation values ​​and aggregation functions to obtain a value for each part of the requirements of the set of requirements corresponding to a particular quantity and the use of the values ​​obtained during the use as well as the aggregation functions to obtain the overall score. These techniques are also applicable to the case of recommended requirements. According to another approach, the evaluation module 22 aggregates the values ​​^^ ^^ ^^ differently. ^^ (value for mandatory requirements) and ^^ ^^ ^^ ^^(value for the recommended requirements) to achieve the overall QoS. The normalized scores are interpreted here as the confidence level that the requirements are passed. Given the mandatory nature of the requirements underlying ^^ ^^ ^^ ^^ , it is necessary that the slightest failure of a mandatory requirement is reflected in the overall score and cannot be compensated by the recommended requirements. For this, in this approach, the evaluation module 22 uses the following formulas:^^ ^^ ^^ ^^ ^^ ^^ 2 ^^ ^^ ^^ ^^ < 1^^ ^^ ^^ =1 + ^^ ^^ ^^^^ ^^ ^^ ^^ ^^ { 2 ^ = 1In the previous techniques, the aggregation functions are compensatory, which implies that poor compliance with one requirement can be compensated by good compliance with another requirement. In some cases, it may be desired that this compensation is not possible, particularly with regard to mandatory requirements. As soon as at least one mandatory requirement is not met, it would no longer be possible to compensate for this non-compliance even if all the other requirements are met. For this, another type of aggregation function called t-norms can be used. These are binary operators ^^( ^^, ^^) ∈ [0,1] where ^^, ^^ also have values ​​in [0,1]. A binary operator ^^: [ 0.1 ] × [ 0.1 ] → [0,1] is a t-norm if it satisfies the following properties: ^ (neutral element) ^^ ( 1, ^^ ) = ^^ for all ^^ ∈ [0,1], ^ (commutativity) ^^ ( ^^, ^^ )= ^^( ^^, ^^) for all ^^, ^^ ∈ [0,1], ^ (monotonicity) ^^ ( ^^, ^^ ) ≤ ^^( ^^, ^^) for 0 ≤ ^^ ≤ ^^ ≤ 1 and 0 ≤ ^^ ≤ ^^ ≤ 1, and ^ (associativity) ^^( ^^, ^^ ( ^^, ^^ ) ) = ^^ ( ^^ ( ^^, ^^ ) , ^^ ) for all ^^, ^^, ^^ ∈ [0,1]. The function T satisfies certain properties. This function provides a pessimistic evaluation with respect to a compensatory function because ^^( ^^, ^^) ≤ min( ^^, ^^), that is, it is never possible to have a grade better than the worst grade. Several bad grades worsen the overall grade. It can be shown that the binary operator ^^: [ 0.1 ] × [ 0.1 ] → [0,1] is a strict and Archimedean t-norm if and only if there exists a function ^^: [ 0.1 ] → [0,∞) continuous, strictly decreasing with ^^(1) = 0 such that ^^( ^^, ^^) = ^^ −1(min( ^^( ^^) + ^^( ^^), ^^(0))). By definition, a t-norm ^^ is Archimedean if the following two conditions are satisfied: ^ ^^ is continuous, and ^ ^^ ( ^^, ^^ ) < ^^ for all ^^ ∈ ( 0.1 ) . This binary operator ^^ is easily transformed into an n-ary operator: where: ^ are the normalized scores of the p-values ​​of the mandatory requirements, ^ ^^( ^^1, … , ^^ ^^ ) is the score ^^ ^^ ^^ ^^ , and ^ the function f is to be constructed by supervised learning. Alternatively, the sum ∑ ^ ^ ^= ^ 1 ^^( ^^ ^^ ) is replaced by a weighted average, which is mathematically written as: where ^^ is a permutation on {1,…, ^^} ordering the scores as follows: The method just described therefore makes it possible to obtain a robust assessment of the compliance of the tracking system 10 with a set of requirements. In fact, rather than having a simple binary value (criterion met or not), the method provides access to an estimate of a margin with which the criterion is met, i.e. an estimate of the confidence that can be had in the estimate. Furthermore, in the presence of too few values ​​to have a reliable estimate, the criterion will always be considered as not met, which is desirable in a critical context such as air traffic, where it is better to be wrongly alarmist than overconfident. In addition, the method provides the certainty that the result cannot vary instantly from "not met" to "met", the p-value evolving smoothly with new data. However, there may be a passage above and below the threshold, in which case the alarm is launched, but there always remains a certain margin toto go through before actually reaching the criterion violation. In addition, the method uses a lower number of points than those recommended in the standard. This reduces the amount of memory occupied as well as the stress on the hardware components of the computer system 10. The method also has the advantage that the evaluation is adaptive. In particular, the method greatly reduces the data acquisition time for a reliable score calculation, and therefore of compliance or non-compliance with the criteria. Thus, the data used are less old, and the estimation of the quality of service is therefore more reactive. This is beneficial when a problem appears since in the event of rapid degradation, the metric degrades more quickly, and therefore the operator is informed more quickly. The converse is also true: when a problem is resolved, the return to normal occurs more quickly. This means that the method makes it possible to take into account theextreme drifts. When the drift is minor, with the number of data fixed a priori, the process is not capable of validating the requirement with high confidence. The number of points can then be temporarily increased to absorb this temporary minor drift. This avoids raising an alert that is not justified. To increase the number of measurement points, the data collection time is either increased (if this is authorized), or geographical areas are grouped together. These two measures are temporary. When the drift is major, from the start of data collection over a time interval, the process makes it possible to realize that there is no chance of satisfying the requirement given the excessive number of bad data. Instead of waiting for the end of the information collection interval, the process raises an alert as soon as possible to the operator. The process also defines a mechanism for controlling the number of measurement points, this mechanism ofcontrol that can be static (determined a priori) and / or dynamic (adapted from the measured values). Thus, it is possible to implement the method in real time over a time interval that can be of a fixed length configured in advance or dynamic depending on the acquisition rate of the values. This makes it possible to detect the presence of an anomaly in the tracking system in real time. In each of the examples described, each module or sub-module is each produced in the form of software, or a software brick. All of the modules or sub-modules are then produced, that is to say, in the form of a computer program, also called a computer program product, it is also capable of being recorded on a medium, not shown, that can be read by a computer. The computer-readable medium is, for example, a medium capable of storing electronic instructions and of being coupled to a bus of a computer system. For example, the readable medium isan optical disk, a magneto-optical disk, a ROM memory, a RAM memory, any type of non-volatile memory (for example FLASH or NVRAM) or a magnetic card. On the readable medium is then stored a computer program comprising software instructions, stored in a memory executable by a processor. In a variant not shown, each module or sub-module is produced in the form of a programmable logic component, such as an FPGA (Field Programmable Gate Array), or an integrated circuit, such as an ASIC (Application Specific Integrated Circuit). The invention relates to any technically possible combination of the embodiments previously described. APPLICATION TO THE SPECIFIC CASE OF THE ESASSP STANDARD Presentation of the criteria The method which has just been described can be applied to any type of standard but will be illustrated in the following through the example of the ESASSP standard which is detailed in the document entitled "EUROCONTROL Specification for ATM Surveillance System Performance» (Volume 1) with ISBN 978-2-87497-022-1 and published in March 2012. The objective of this standard is to ensure the quality required to avoid collisions. This quality aims at a better separation between aircraft in (civil) air traffic. In order to increase air capacity, it is essential to have tracking tools that can ensure an increasingly fine separation between aircraft. Two standards have been defined to ensure two distance separations: 5 NM and 3 NM. The ESASSP standard defines 22 constraints named from R1 to R22, i.e. 22 specific metrics. The 22 constraints are briefly explained below. The first constraint R1 concerns the measurement interval for the probability of update (called in English "Measurement Interval for Probability of Update"). The first constraint R1 is broken down into two requirements: a mandatory requirementand a recommended requirement. The second constraint R2 concerns the probability of updating the horizontal position. The second constraint R2 is broken down into two requirements: a mandatory requirement and a recommended requirement. The third constraint R3 concerns the ratio of time the aircraft was not tracked (called in English "ratio of missed 3D position involved in long gaps"). The third constraint R3 corresponds to a mandatory requirement. The fourth constraint R4 concerns the squared error in horizontal position (called in English "Horizontal position RMS error"). The squared error is often considered as a principal error representative of the measurement accuracy. The fourth constraint R4 is broken down into two requirements: a mandatory requirement and a recommended requirement. The fifth constraint R5 concerns a ratio of consecutive correlated errors (called in English"Consecutive correlated error ratio"). The fifth constraint R5 corresponds to a recommended requirement. The sixth constraint R6 concerns the maximum time interval with close proximity (called in English "Max delta time in close proximity"). Such a constraint corresponds to the fact that, on a radar, very close tracks that are not updated at the same time are present. In the presentation to the air traffic controller, there must not be two side-by-side tracks with distant update dates. This could lead to risks of collision, i.e. serious safety problems. The sixth constraint R6 corresponds to a recommended requirement. The seventh constraint R7 concerns the probability of update of pressure altitude (called in English "Probability of update of pressure altitude"). The seventh constraint R7 corresponds to a mandatory requirement. The eighth constraint R8 relates to the average age of the dataforwarded pressure altitude data (called "Average data age of forwarded pressure altitude"). The eighth constraint R8 corresponds to a mandatory requirement. The ninth constraint R9 relates to the maximum age of forwarded pressure altitude data (called "Max data age of forwarded pressure altitude"). The ninth constraint R9 corresponds to a mandatory requirement. The tenth constraint R10 relates to the ratio of incorrect forwarded pressure altitude data. Note that the tenth constraint R10 is an estimate of the error corresponding to a spurious error. The tenth constraint R10 corresponds to a mandatory requirement. The eleventh constraint R11 relates to unsigned pressure altitude errors (called "Pressure altitude unsigned error"). Note that the eleventh constraint R11 is an estimateof the main error type error. The eleventh constraint R11 corresponds to a mandatory requirement. The twelfth constraint R12 concerns the delay of appearance of the emergency indicators (called in English "Delay of apparition of the emergency indicator / SPI report"). The twelfth constraint R12 corresponds to a mandatory requirement. The thirteenth constraint R13 relates to the delay of change in Aircraft ID. The thirteenth constraint R13 corresponds to a mandatory requirement. The fourteenth constraint R14 concerns the probability of updating with a correct identification (called in English "Probability of update of aircraft identity with correct value"). The fourteenth constraint R14 is broken down into two requirements: a mandatory requirement and a recommended requirement. The fifteenth constraint R15 relates to the ratio of incorrect aircraft identification (called in English "Ratio ofIncorrect Aircraft identity"). It should be noted that the fifteenth constraint R15 is an estimate of the error of the spurious error type. The fifteenth constraint R15 corresponds to a mandatory requirement. The sixteenth constraint R16 concerns the quadratic error of the rate of climb or descent (called in English "Rate of climb / descent RMS error"). In this context, the estimated error is of the principal error type. The sixteenth constraint R16 corresponds to a recommended requirement. The seventeenth constraint R17 concerns the quadratic error of the tracking velocity (called in English "Track velocity RMS error"). In this context, the estimated error is of the principal error type. The seventeenth constraint R17 corresponds to a recommended requirement. The eighteenth constraint R18 concerns the quadratic error of the tracking velocity angle (called in English "Track velocity angle RMS error"). In this context, the estimated error is of the principal error type. The eighteenthconstraint R18 corresponds to a recommended requirement. The nineteenth constraint R19 concerns the density of uncorrelated false target reports. In this context, the estimated error is of the spurious error type. The nineteenth constraint R19 corresponds to a recommended requirement. The twentieth constraint R20 concerns the hourly rate of false tracks close to true tracks. In this context, the estimated error is of the correlated error type. The twentieth constraint R20 corresponds to a recommended requirement. The twenty-first constraint R21 concerns continuity. The twenty-first constraint R21 corresponds to a recommended requirement. The twenty-second constraint R22 concerns manual investigations (manual analyses of the results) which must be carried out when constraints R2, R4, R12 or R13 are not met.achieved. The twenty-second constraint R22 is broken down into two requirements: a mandatory requirement and a recommended requirement. In summary, the following Table 1 can be established, the presence of a cross indicating that an associated requirement is present: Requirement Requirement Constraints English name used in the standard Recommended mandatory unit Measurement Interval for R1 s XX Probability of Update Probability of update of horizontal R2 % XX position ratio of missed 3D position R3 % X involved in long gaps R4 Horizontal position RMS error m XX R5 Consecutive correlated error ratio % X R6 Max delta time in close proximity s X Probability of update of pressure R7 % X altitude with correct value Average data age of forwarded R8 s X pressure altitude Max data age of forwarded R9 s X pressure altitude Ratio of incorrect forwarded R10 % X pressure altitude R11 Pressure altitude unsigned error % X Delay of apparition of the R12 s X emergency indicator / SPI report R13Delay of change in Aircraft Id s X Probability of update of aircraft R14 % XX identity with correct value R15 Ratio of Incorrect Aircraft Identity % X R16 Rate of climb / descent RMS error m / s X R17 Track velocity RMS error m / s X R18 Track velocity angle RMS error degrees X Density of uncorrelated false R19 number X target reports Number per hour of false tracks R20 number X close to true tracks R21 Continuity % X R22 Investigations X Table 1: Association for each constraint of its name in English, the associated unit as well as the presence of a mandatory requirement and a recommended requirement In summary, the ESASSP standard is a set of 14 mandatory requirements and 12 recommended requirements, which represents 26 requirements in total, these 26 requirements being estimated according to 22 distinct metrics. Calculation of p-values ​​associated with criteria For the calculation of p-values, the criteria defined in the ESASSP can be grouped into several families. These families are atnumber of five and are set out in the following. The first family groups together criteria related to update probabilities, these criteria being criteria R2, R7 and R14. These criteria R2, R7 and R14 have in common that they relate to a proportion between the number of reports received containing a certain element (horizontal position, pressure altitude, and aircraft identity respectively) among all reports received (which may or may not contain the element in question). These criteria require that the value of the proportion be greater than a threshold value. The second family groups together criteria related to the root mean square error, these criteria being criteria R4, R6, R16, R17 and R18. Indeed, the criteria of the second family relate to a calculation of the average Euclidean distance between a value and a reference measurement, often calculated a posteriori by the SSTA. These criteria require that the value of the proportion be less than a threshold value.The third family includes criteria related to ratios, these criteria being criteria R3, R5, R10, R11 and R15 and require that a ratio between two quantities has a value lower than a predefined value. The fourth family includes criteria related to averages, these criteria being criteria R8, R12 and R13. The fifth family includes criteria not falling under the four families already presented. These are criteria R19, R20 and R9. It can be noted that these families do not include all the criteria. In fact, criterion R1 is a parameter of the tracking system 10 which is therefore fixed a priori, criterion R9 is not a criterion, but a filter on the validity of the data, the criterion implies not having an incident over periods of several years (which cannot be evaluated over a short period) and criterion R22 is not a quantitative criterion. It is now described how to calculate the probabilities of obtaining a new value notnot respecting the requirement knowing the acquired values, that is to say the p-values ​​for each of the aforementioned families. First family For the first family, it is first explained how to determine the p-values ​​for the criterion R2 involving a simple update probability. The criterion R2 involves a metric calculated as the ratio between a number of observed trajectory portions (noted ^^) and the number of these trajectories which contain a horizontal position (noted ^^). The probability is therefore: ^^ = ^^ / ^^ Where: ^ ^^ is the number of measurement points (for example the number of segments of identical size in all trajectories, for the criterion R2) and ^ ^^ is the number of measurement points having a certain property (for example the number of trajectory segments in which the tracking system 10 has received at least one horizontal position update from the aircraft, for the criterion R2). It is assumed that the value of ^^ is a realization of a variablerandom ^^ ^^ which follows a binomial distribution of probability ^^, with ^^ trials. This assumes the independence of successive observations. The rule requires that this probability be above a certain threshold (noted ^^ ^^ ). According to a simple embodiment, the null hypothesis can be chosen as:^^0 ∶ ^^ ≤ ^^ ^^The calculation of the p-value noted ^^ can then be implemented using the formula:^^ = ^^( ^^ ^^ ≥ ^^| ^^0)Where ^^ ^^is a random variable following a binomial distribution of probability ^^, with ^^ trials. This corresponds to the following formulation: "knowing that the actual probability does not meet the criterion, what is the probability that the tracking system 10 observes an equal or greater number of reports containing a horizontal position?". If this value is sufficiently low, it is considered that the observation of ^^ is probably not the result of chance, and that the threshold is therefore passed with a high probability. According to a more elaborate embodiment and particularly in the context of a critical system, it is preferable to impose larger margins. For this, the null hypothesis becomes ^^0∶ ^^ = ^^ ^^and the calculation of the p-value as for the simple implementation. This corresponds to the following formulation: "knowing that the actual probability just meets the criterion, what is the probability that we observe a greater or equal number of reports containing a horizontal position?". Thus, it is more difficult for the system to meet the criterion, and we therefore ensure greater robustness of the system. For this criterion, and under these assumptions, the p-value is calculated:^^ = 1 − ^^1− ^^ ^^( ^^ − ^^ + 1, ^^)where ^^ is the regularized incomplete beta function. The curve of the p-value thus obtained is represented in Figure 3 for the requirement R2. The x-axis represents the ratio ^^̂ = ^^ ^ ^ , the y-axis represents the p-value. As a reminder, the R2 requirement requires that ratio be greater than or equal to ^^ ^^= 0.97, shown by the vertical dotted line. The horizontal dotted line corresponds to the 0.05 threshold commonly used for p-values. In other words, if the p-value is below this bar, then we have very high confidence that the R2 requirement has been met. Each curve corresponds to a number of points. Each curve takes the form of a decreasing sigmoid whose inflection point is reached at ^^ ^^ = 0.97. We notice that the curve is steeper around this inflection point as the number of points increases. This means that with many points, there is a very high confidence that the R2 requirement is met as soon as we are barely above the threshold ^^ ^^= 0.97. On the other hand, for a small number of points, it is necessary to be significantly above the threshold to be confident that the requirement is met. Such a curve can be produced for each of the requirements and leads to the results that will be detailed below. For the sake of simplicity, these curves are not introduced in what follows to simplify the discussion but the results obtained use these curves which were produced by the Applicant. In certain cases, it is required that the value calculated for a criterion be reached for all the tracked trajectories. This is the case in particular for the "recommended" criterion of criterion R2. In this case, the values ​​are used, which are calculated trajectory by trajectory. This means that, for a given trajectory, the values ​​are calculated only on the points belonging to this trajectory. That is ^^ ^^the p-value associated with a given trajectory ^^ and ^^ the set of trajectories. In the case of criterion R2, it is a matter of verifying that, for all ^^ ∈ ^^, the underlying update probability ^^ ^^ check ^^ ^^ > 0.97. It is thus calculated for a trajectory ^^ the p-value noted ^^ ^^ of the test defined in the previous paragraph, for which only the information from the reports concerning the trajectory ^^ is used. I l vient ainsi : This can be rewritten as:^^ = 1 − ^^(∀ ^^ Assuming that the probabilities are independent along the different trajectories, we get: Thus, the p-value of the test aiming to verify that all trajectories verify ^^ ^^ > 0.97 is calculated as: It should be noted that this approach is applicable to all subsequent p-values ​​for which we want to verify that all trajectories meet a given criterion. It is then explained how to determine the p-values ​​for criteria R7 and R14 involving a double update probability. Metrics involving a double update probability are calculated in the same way as for metrics involving a single update probability up to one difference. More precisely, the probability is written:^^ = ^^ / ^^Where: ^ ^^ is the number of reference realizations (e.g., for criterion R7, the number of time intervals where an altitude update is received), and ^ ^^ is the number of realizations having a certain property (e.g., for criterion R7, the number of time intervals where a non-erroneous altitude update) In this case, the numbers ^^ and ^^ are considered to come from binomial distributions.This assumes the independence of successive observations, and the fact that these observations are identically distributed. According to a particular embodiment, the number ^^ always corresponds to the number of reports containing a pressure altitude. As for the number ^^, it is, among these^^ reports, those whose pressure altitude value is correct (in the case of criterion R7) or an aircraft identifier (criterion R14). Thus, the number ^^ follows the binomial distribution ^^. ^^ from the previous paragraph, with an unknown probability ^^ ^^ , and the number ^^ follows a binomial distribution with parameters ^^ and ^^̂. In this context, the value ^^̂ is unknown and it is a question of verifying that its value is greater than a given threshold ^^ ^^ . Still taking the null hypothesis ^^0∶ ^^̂ = ^^ ^^ , the p-value is then calculated as follows: In practice, the integral can be calculated by numerical methods such as a Monte Carlo draw or a rectangle approximation method. Second family For the second family, the p-value for root mean square errors is calculated. A criterion of the second family is therefore a metric that is a mean square error between an observed value and a reference value. The criterion is met when this metric is less than a given value. Let EQMR be the root mean square error. This square error is written: ^ ^^ ^^ is the squared error on a given quantity for the ^^-th measurement. For example, for criterion R4, the quantity is the error in the horizontal position of the aircraft. Assuming that all squared errors ^^ ^^ are identically distributed and independent, and that the corresponding law has a mean ^^ 2 and a variance ^^ 2(unknown), the criterion requires that the average ^^ 2 be less than a certain value ^^ ^ 2 ^ , which is the threshold of the requirement. Let ^^̂ 2 the observed mean squared errors ^^ ^^ . By making the null hypothesis: ^^0∶ ^^ 2 = ^^ ^ 2 ^ , the central limit theorem allows us to obtain the following relation: Where: ^ Φ is the distribution function of the reduced centered normal distribution. For some criteria, the metrics are calculated as the minimum over different trajectories of the mean square error calculated on each trajectory. This is for example the R4 criterion. The approach is then the same as before. Third family For the third family, as for the first family, there are two types of criteria in this family, a first type called simple ratio gathering criteria based on a metric involving a variable divided by a constant and a second type called double ratio gathering criteria based on a metric involving a variable divided by another variable. For both types, the metric must be lower than a predefined threshold. First, we explain how to calculate the p-value for criteria based on a simple ratio, as is the case for criteria R3, R5 and R11.This calculation is quite similar to the case of the simple update probability since the ratios are of the form:^. ^ = ^^ ^^ With: ^ ^^ a given number (e.g., the number of track updates received for criterion R5), and ^ ^^ a random variable that follows a binomial distribution with parameters ^^ and ^^ (e.g., for criterion R, the number of track updates for which at least 3 consecutive horizontal position updates have an error on the same side of the trajectory). This is to test whether ^^ is significantly less than ^^ ^^ , the required value, given the observations. Thus, taking as the null hypothesis, ^^0∶ ^^ = ^^ ^^and, for this metric, and under these assumptions, the p-value is calculated according to the following formula:^^ = ^^1− ^^ ^^( ^^ − ^^ + 1, ^^)The symmetry with the simple update probability can be noted here. This is explained by the fact that, in one case, it is a question of maximizing the simple update probability while here, it is a question of minimizing the simple ratios. For double ratios, the following approach can be followed. This approach is particularly suitable for criteria R10 and R15. In the case of double ratios, the metric is written in the following form: Where: ^ ^^ ^^ is the number of reference realizations (e.g., the number of track updates containing a barometric altitude, for criterion R10), and ^ ^^ ^^is the number of realizations with a certain property (e.g., the number of track updates containing an incorrect barometric altitude, for criterion R10). For these metrics, the aim is to verify that the ratio of two binomial variables is less than a threshold. By analogy with simple ratios, the p-value is calculated as the symmetric of those calculated for double update probabilities: Where: ^ ^^ plays the role of the observed probability of the intermediate value (i.e. the underlying probability of the distribution that determines the value ^^ ^^ ), like ^^ ^^ was in the case of double update probabilities. The previous expression for the p-value corresponds to integrating over all possible values ​​of ^^, weighted by the likelihood of ^^ given the value of ^^ ^^ . Fourth family For the fourth family, the criteria involve metrics of the form: Where: ^ ^^ is a variable of interest (e.g., the age of the barometric pressure information when it is sent, for criterion R8), given by the ratio ^^, and ^ ^^ is the number of ratios considered. The criterion is met when these averages are below a threshold, noted ^^ ^^ . Assuming the variables ^^ are identically distributed and independent on the ratios, and a sufficient number of ratios, the central limit theorem can be applied. Let ^^ be the (unknown) mean and ^^ be the standard deviation of the variable ^^, ^^̂ be the observed mean and ^^̂ be the standard deviation. As previously stated, this is that ^^ is smaller than ^^ ^^ .By making the null hypothesis ^^0 ∶ ^^ = ^^ ^^, by the central limit theorem, it comes: Where: ^ Φ is the cumulative function of the reduced centered normal distribution. Fifth Family For the fifth family, each of the remaining criteria are examined. Regarding criterion R19, a metric is considered, corresponding to the number of false reports obtained for each one-hour interval and for each 900 NM² area. This gives an empirical distribution (representable in the form of a histogram), for each time interval. Let ^^ be this distribution, ^^ its mean and ^^ its standard deviation. This metric is calculated over one hour, this gives at least 450 time intervals because the time interval length is at most 8 seconds. Moreover, the metric is cumulative. C eci implique que : ^^ = ∑ ^^ ^^ ^^∈ ^^ ^ ^^ ^^denotes a realization of ^^ over an interval ^^, ^ ^^ denotes the set of intervals i that make up the observation hour, and ^ K denotes the number of false tracks observed during the observation hour, over a predefined area of ​​900 NM² given. Considering a single area of ​​900 NM², for ^^ intervals in an hour, the criterion corresponds to a maximum of ^^ ^^ false leads. A renormalized empirical distribution can then be defined ^^ ∗ . By definition, ^^ ∗ is equivalent to ^^, but the values ​​are multiplied by ^ ^× ^^ . I l vient ainsi : Where: ^ ^^ ∗ ^ ^ designates an achievement of ^^ ∗ on an interval ^^, and ^ K* is a random variable of the variable of interest (number of false leads) knowing that the distribution of the number of false leads per interval is distributed according to the D* law. The expectation of ^^ ∗ is ^^ ^^, namely the required threshold. The null hypothesis is that the number of false trails observed in the area during an interval follows the law ^^ ∗ . The p-value is written as follows: Where: ^ ^^ denotes the p-value, ^ P denotes the probability, ^ ^^ ^^ the number of false trails observed in the area during the interval ^^, and ^~ means “follows the law”, and so here that ^^ ∗ ^ ^ follows the D* law. Since the number of intervals is several hundred intervals ^^, it is possible to apply the central limit theorem, giving: Where: ^ Φ is the cumulative function of the reduced centered normal distribution. Thus, for each of the zones ^^ of 900NM², it is possible to calculate the p-value on this zone ^^ ^^. As before, it can be verified that it is indeed the maximum of these values ​​on each of the zones which is lower than the threshold. Thus, the global p-value (on all the zones ^^ ∈ ^^ where ^^ is the set of zones) is calculated as follows: For the R20 criterion, a metric is considered corresponding to the number of false leads close to real leads for each one-hour interval. The same reasoning as for the R19 criterion leads to the following expression for the p-value: Where ^^ ∗ is the renormalized empirical distribution of the number of false tracks close to real tracks on each of the intervals. Regarding the R9 criterion, a similar strategy can be applied. Let: ^ ^^ be the observed distribution of the ages of reports containing a pressure-altitude, ^ ^^ ∗ the distribution ^^ rescaled to have an average of 16 seconds (the required threshold), and ^ ^^ ∗the cumulative function of ^^ ∗ . Then the p-value of the hypothesis test can be calculated, the null hypothesis of which is that the age variable of the reports containing a pressure altitude follows the distribution ^^ ∗ . Thus comes the following expression for the p-value: Where: ^ ^^ denotes a set of ratios, each ratio having a pressure altitude, ^ ^^ denotes a ratio, and ^ ^^ ^^denotes the age of the report ^^ and is assumed to be independent of other ages. Calculation of the optimal value of the number of points for determining a criterion The parameters influencing the optimal number of points for each criterion according to the previous families are given in the following table. Parameters influencing the optimal number Requirement Point family - number of errors made - number of trajectories R2 1 - number of errors made per trajectory R3 3 - number of errors made - mean - standard deviation R4 2 - number of trajectories - mean per trajectory - standard deviation per trajectory R5 3 - number of errors made - mean R6 2 - standard deviation - number of errors made R7 1 - number of cases where the reference situation does not occur - mean R8 4 - standard deviation R9 5 - moments on the empirical distributions - number of errors made R10 3 - number of cases where the reference situation does not occur R11 3 - number of errors made R12 4 - mean - standard deviation- mean R13 4 - standard deviation - number of errors made - number of cases where the reference situation does not occur - number of trajectories R14 1 - number of errors made per trajectory - number of cases where the reference situation does not occur per trajectory - number of errors made R15 3 - number of cases where the reference situation does not occur - mean R16 2 - standard deviation - mean R17 2 - standard deviation - mean R18 2 - standard deviation - empirical mean of past data - empirical standard deviation of past data R19 5 - number of errors made - empirical mean of past data - empirical standard deviation of past data R20 5 - number of errors made Thus, to determine the number of points for each requirement as well as the spatio-temporal discretization, the values ​​of the influence parameters must be entered. For this, it is assumed that we have a sample ^^ of normal acquired values ​​(which must pass all the requirements)representative of the situation being studied (data from the same airport or the same En-Route zone). It is then calculated a priori, for each of the requirements, the average number of points necessary for the p-value to reach the desired threshold (if the data corroborate it). Indeed, if the number of points is too low, the p-value will be too high to reject the null hypothesis with sufficient certainty, even if the data are "perfect" (i.e. they tend to validate the requirement). Consequently, assuming the data are perfect, it is possible to identify the minimum number of points necessary for the p-value to fall below the required threshold. This then makes it possible to decide to do the calculation in real time, based on a larger number of data. For this, with the help of an expert, a compromise is determined between a small number of data, which allows for a much more reactive system since fewer intervals are usedof observed time, and a large amount of data, which reduces the dynamism of the system, but offers more precision (and therefore a better vision of the situation). Thus, the identification of the minimum number of points can be implemented by implementing a series of operations. In a first operation, the desired maximum monitoring duration is obtained in order to ensure sufficient responsiveness. This maximum duration is provided by the user of the tracking system 10. From this duration, a number of points is deduced ^^ ^^ ^^ ^^maximum that can be obtained over this duration. In a second operation, a sample ^^ of normal values ​​is obtained, that is to say a sample respecting all the requirements. These normal values ​​are called normal data in the following. In a third operation, for each normal data ^^ ∈ ^^, the values ​​of the parameters influencing the minimum number of points for each requirement are calculated. The minimum number is deduced of points that would be needed to pass requirement i on this data d. In practice, three cases will arise. In the first case, if ^^ ^ ^ ^ ^ ≤ ^^ ^^ ^^ ^^ for a requirement i and any data d, then this is the ideal case where all available data passes the requirements with very high confidence, using ^^ ^^ ^^ ^^data. We can also use a number of data for the requirement i depending on dem ^^∈a ^x^ ^^ ^^^^ (to which a margin is added). According to a second case, ^^ ^ ^ ^ ^ ≤ ^^ ^^ ^^ ^^ for the vast majority of requirements i and the majority of data d, but > ^^ ^^ ^^ ^^ in some isolated cases. In such a case, the number or percentage of ^^ data that do not meet the requirements with high confidence is quantified. The user sets a tolerance threshold, below which correct data is allowed to raise alerts. In this case, ^^ is used ^^ ^^ ^^ data, or a number of data for KPI i depending on m ^^ ^ ^∈ has ^ x ^ ^^ ^^. In the third case, a significant number of ^^ data do not meet the requirements. An example of a significant number in this context is, for example, half of the total data. Indeed, if one measurement out of two is bad, it can be considered that there is a problem with the operation of the tracking system 10 or at least that its operation should be checked. In such a case, this means that these requirements are not compatible with a real-time evaluation. This is for example the case of the detection of rare events. It can be noted here that another possible gain in terms of the number of data points without extending the acquisition time is to widen the geographical observation area. For this, two adjacent areas through which little traffic passes can be grouped together, in order to have, over this larger area, enough data to accurately estimate whether or not the requirements are met.The initial zones (before grouping) are predetermined, notably chosen by the user of the tracking system 10. In what follows, the number of points necessary to obtain a p-value less than or equal to 0.05 is studied for certain criteria as a function of data degrading compliance with the criterion.

[0002] Threshold to Number of errors Criterion reach 0 1 2 3 > 0.97 99 157 208 257 R2 more than 100 points > 0.99 299 473 628 773 R2 less than 100 points > 0.97 99 >100 >100 >100 R3 < 0.005 598 947 1258 1549 R5 < 0.0003 9985 15812 20984 25844 R11 (case of a flight at constant altitude) < 0.001 2995 4742 6294 7752 R11 (case of a landing or takeoff) < 0.015 199 315 418 515 Table 3: Number of points needed to achieve a p-value of 0.05 or less for criteria R2, R5 and R11 In Table 3, an error is a problematic ratio tending to degrade the criterion. ^ ^ ^^ ^^ ^^0 1 2 3 4 5 10 200 57 100 138 173 208 241 401 702 1 81 119 154 188 222 254 411 710 2 100 135 169 202 235 267 422 718 3 116 150 183 215 247 278 431 726 4 130 163 195 227 258 289 440 733 5 143 175 206 238 268 299 449 741 10 195 225 255 285 315 344 490 776 20 273 302 331 360 388 417 558 838 Table 4: Number of points needed to reach a p-value of 0.05 or less, for criterion R7 In Table 4, e R is the number of reports not containing a horizontal position and e A is the number of reports containing a horizontal position but not containing a pressure altitude. ^ ^ ^^ ^^ ^^0 1 2 3 4 5 10 200 115 200 276 348 418 485 806 1410 1 163 238 300 375 444 510 825 1422 2 200 271 339 405 470 534 844 1438 3 232 299 365 430 493 556 863 1453 4 260 325 389 453 515 577 881 1468 5 285 349 412 474 536 597 898 1462 10 388 448 508 567 626 685 977 1549 20 539 596 654 711 768 825 1107 1666 Table 5: Number of points needed to reach a p-value of 0.05 or less, for criterion R14 In Table 5, e R is the number of reports not containing a horizontal position and e A is the number of ratios. ρ A R ε0 1 2 3 4 5 10 20 100 1.0 3000 4800 6300 7800 9200 10600 17000 29100 ND 0.98 3100 4900 6500 8000 9400 10800 17400 29700 ND 0.96 3200 5000 6600 8100 9600 11000 17700 30300 ND 0.9 3400 5300 7000 8700 10200 11700 18900 32300 ND 0.75 4000 6400 8400 10400 12300 14100 22700 38800 ND 0.5 6000 9500 12600 15600 18400 21100 34000 ND ND 0.25 12000 19000 25200 31100 36700 42100 ND ND ND Table 6: Number of points needed to reach a p-value of 0.05 or less, for the R10 criterion In Table 6, the number of points needed is rounded to the nearest hundred. In addition, ^^ ^^ denotes the proportion of reports containing a pressure altitude (resp aircraft identifier), ^^ ^^is the number of reports containing invalid pressure-altitude and ND means that this number is greater than 50000. Analysis of Tables 3 to 6 shows that, in most cases, the optimal number of points is less than 50000. The only cases where the optimal number exceeds 50000 are cases where the number of errors is very large. Since a large number of errors is a relatively rare event, these tables show that the method allows, in normal operations, to obtain evaluations of the conformity of the criteria with a reduced number of measurements.

Claims

CLAIMS 1. A method for evaluating the conformity of a tracking system (10) to a set of requirements, at least one requirement being a mandatory requirement and at least one requirement being a recommended requirement, each requirement requiring that a physical quantity be greater than or equal to a threshold or that a physical quantity be less than or equal to a threshold, the evaluation method being implemented by computer, the evaluation method comprising at least the following steps: - acquiring a set of values, the set of values ​​comprising the values ​​accessible to the tracking system (10) over a predefined time interval, - for each requirement, calculating a robustness value, the robustness being calculated as the p-value of a statistical test indicating whether the requirement considered is met or not, - for each requirement, comparing each robustness value to a predefined threshold according to a respective comparison criterion,and - determining the requirements with which the tracking system (10) complies, the requirements determined as compliant being the requirements for which, in the comparison step, it was determined that the comparison criterion is met.

2. Method according to claim 1, wherein the method further comprises, for each robustness value not meeting the comparison criterion, a step of determining the cause of the non-compliance with the comparison criterion, the cause being either that the requirement is not met or that the number of values ​​is too low to guarantee a predefined reliability threshold for assessing compliance with the requirement in question.

3. Method according to claim 2, wherein, when the determined cause is too low a number of values,the method comprises a step of deducing a number of additional values ​​to be acquired to obtain an evaluation reliability greater than the predefined reliability threshold.

4. Method according to claim 3, in which the method comprises a reiteration of the acquisition, calculation and comparison steps until the number of values ​​either allows obtaining an evaluation reliability greater than the predefined reliability threshold or reaches a predefined maximum number with an evaluation reliability lower than the predefined threshold., 5. Method according to claim 4, wherein the method further comprises a step of issuing an alert for requirements not met or for requirements for which the number of values ​​of the acquisition, calculation, comparison steps has reached the predefined maximum number with an evaluation reliability lower than the predefined threshold.

6. Method according to claim 4 or 5, wherein the maximum number is between 40000 and 60000.

7. Method according to any one of claims 1 to 6, wherein, during the calculation step, an integral of a regularized incomplete beta function is calculated.

8. Method according to claim 7, wherein, during the calculation step, a p-value is equal to the result of the integral and another p-value is a linear function of the result of the integral. 9.Method according to any one of claims 1 to 8, wherein during the calculation step, a distribution function of the reduced centered normal distribution is calculated.

10. Method according to any one of claims 1 to 9, wherein the method further comprises: - a step of applying an evaluation function specific to each requirement to the values ​​p to obtain evaluation values, and - a step of using the evaluation values ​​to obtain an overall score evaluating the conformity of the tracking system (10) to all the requirements.

11. Method according to any one of claims 1 to 10, wherein the number of requirements is greater than or equal to 20.

12. Method according to any one of claims 1 to 11, wherein the tracking system (10) provides measurement estimates from previous data from several sensors (12), the set of values ​​comprising the measurement estimates. 13.Module (22) for evaluating the conformity of a tracking system (10) to a set of requirements, at least one requirement being a mandatory requirement and at least. a requirement being a recommended requirement, each requirement requiring that a physical quantity be greater than or equal to a threshold or that a physical quantity be less than or equal to a threshold, the evaluation module (22) being capable of: - acquiring a set of values, the set of values ​​comprising the values ​​accessible to the tracking system (10) over a predefined time interval, - for each requirement, calculating a robustness value, the robustness being calculated as the p-value of a statistical test indicating whether the requirement in question is met or not, - for each requirement, comparing each robustness value to a predefined threshold according to a respective comparison criterion, and - determining requirements with which the tracking system (10) complies, the requirements determined as compliant being the requirements for which, in the comparison step, it has been determined that the comparison criterion is met.Tracking system (10) comprising an evaluation module (22) according to claim 13.

Images