Safeguarding system against false non-triggers

Abstract

To provide a computer-implemented method for safeguarding a system against false non-triggers.SOLUTION: The method includes: receiving a time series of criticality, a system including functionality that is triggered when the criticality meets a first predetermined criterion; computing a time series of reference, the reference being comparison criticality for triggering the functionality of the system; computing a time series of an error measure based at least on the time series of the criticality and the time series of the reference, where a non-trigger of the functionality of the system is evaluated as a false non-trigger when a portion of the time series of the error measure meets a second predetermined criterion; and identifying at least one near-false non-trigger, where a non-trigger of the functionality of the system is evaluated as a near-false non-trigger when a portion of the time series of the error measure meets a third predetermined criterion and does not meet the second predetermined criterion.SELECTED DRAWING: Figure 1a

Description

[Technical Field]

[0001] Conventional technology A technical system may have a function that is triggered under specific conditions (for example, according to a predetermined first criterion, as shown below). Often, such specific conditions relate to the system's surrounding conditions and / or environmental data, which are detected and subsequently processed, for example, by sensors in the system or other systems. For example, based on a time series of sensor data and / or a time series of data derived from sensor data, it is possible to check whether a specific condition is met at a given point in time, and therefore whether the system's function should be triggered. [Background technology]

[0002] Systems may be exposed to environments that can change during their operation. For example, this system could be a driving system, particularly a vehicle driving system, where the environment dynamically changes during operation. Typically, attempts are made to obtain some degree of automated recognition from the environment, via ambient conditions and / or environmental data, so that it can be determined whether a trigger for a system function at a particular point in time is an appropriate system response. What constitutes an appropriate system response may relate, for example, to the system's specifications and / or its interaction with the environment. Even when a wide range of sensors are used for environment detection (e.g., for computer vision), automated recognition of the environment is an open-context issue, because the environment can generally change almost arbitrarily. Furthermore, even when generalized algorithms (e.g., machine learning algorithms, or Maschinenlern-Algorithmen in German) are used, it is impossible to completely eliminate erroneous automated recognition.

[0003] Therefore, in relation to the quality of automated perception of the environment (and / or specific conditions), a trigger for a system function at a given point in time can be either correct or incorrect: if a system function is correctly triggered at a given point in time (for example, from the perspective of a knowledgeable observer), then a correct trigger exists (true positive). Conversely, if a system function is incorrectly triggered at a given point in time (similarly from a knowledgeable perspective), then a false trigger exists (false positive). Furthermore, in relation to the quality of automated perception of the environment (and / or specific conditions), a non-trigger for a system function at a given point in time can be either correct or incorrect: if a system function is not correctly triggered at a given point in time (for example, from the perspective of a knowledgeable observer), then a correct non-trigger exists (true negative). Conversely, if a system function is not incorrectly triggered at a given point in time (similarly from a knowledgeable perspective), then a false non-trigger exists (false negative).

[0004] Ideally, a system should have as few false triggers and as few false non-triggers as possible during operation. Therefore, the development of a system or its functions is often already structured with the aim of reducing or nearly eliminating false triggers and / or false non-triggers. While it is clearly desirable to avoid false triggers and / or false non-triggers as much as possible during system operation, demonstrating a low rate of false triggers and / or false non-triggers in release testing becomes increasingly difficult as the number of false triggers and / or false non-triggers measured (or measurable) in such release testing decreases. This often results in release testing becoming long, cumbersome, and lengthy run tests. A release to a system or its functions may be permitted, for example, if no false triggers and / or false non-triggers are observed (or their corresponding frequency falls below a certain threshold) during at least one long run, i.e., over a certain period. [Overview of the project] [Problems that the invention aims to solve]

[0005] System protection against false triggers and / or false non-triggers is particularly important in safety-critical systems, especially when an inappropriate system response resulting from a false trigger or false non-trigger could cause irreparable damage (e.g., bodily injury or resulting fatality). For example, such a system could be a safety-critical driver assistance system (ein sicherheitskritisches Fahrerassistenzsystem FAS) or an automated driving system, particularly a highly automated driving system (ein hochautomatisiertes Fahrsystem HAF). The function of such a system might be directed towards collision avoidance, such as automatic emergency braking (AEB). In other examples, particularly in HAFs, the system's function for collision avoidance may also include avoidance or a combination of braking and avoidance. Such systems must be protected (i.e., demonstrated) against risks based on incomplete system specifications before being introduced to the market. In relation to security architecture, such protection may also be required in various safety layers. For protection, large-scale test runs (long-term runs) are typically performed. Selectively or additionally, virtual test runs may be simulated. However, due to the complexity and diversity of environmental impacts, as well as the wide range of often unknown driving conditions, protection of environmental detection FAS or HAF in such virtual test runs (simulations) may be insufficient. This can occur particularly if the simulation itself could not be verified, or if the simulation itself could only be partially verified and therefore could not adequately represent reality.

[0006] Often, systems designed to avoid accidents, such as emergency brake assist or collision avoidance systems in HAF (Human-Assisted Frontale), can only intervene in vehicle operation (e.g., emergency braking / evasive maneuvers) in rare (and unusual) circumstances.

[0007] Risks to vehicles equipped with this system and to other traffic participants can arise from undue intervention (false triggers), such as when the system in a preceding vehicle improperly activates emergency braking, thereby causing a collision. The maximum acceptable rate at which undue intervention may occur (false trigger rate) is generally related to the difficulty and controllability of the intervention and can be determined, for example, through objective hazard and risk analysis. Generally, it can be assumed that false triggers should never occur more frequently than correct triggers. The long-term driving performance to be performed can be determined in relation to the acceptable rate of false triggers. In this case, false triggers should not be observed during long-term driving. Often, only the (non-)occurrence of undue triggers is evaluated, and a more detailed evaluation of the system, for example, its internal interfaces, is not performed. Since accidents (near-accidents) occur rarely in actual driving phenomena, release tests verify extremely low occurrence rates or long time intervals between two false triggers.

[0008] On the other hand, risks to vehicles in which this system is implemented, and risks to other traffic participants, can also arise from missed interventions that would have been legitimate and necessary, i.e., false non-triggers. This could occur, for example, if a system within a vehicle fails to apply sufficient brakes that would have been legitimate and necessary to avoid a collision with a significantly slower vehicle ahead. Delayed interventions can also be evaluated as missed interventions and, consequently, false non-triggers, in relation to the time at which the intervention should have occurred. The maximum acceptable rate of missed legitimate interventions (false non-trigger rate) is generally related to the type of system and the environment in which it is used. Generally, it can be assumed that false non-triggers should never occur more frequently than the typical number of accidents in the scope of use. Precise target values ​​are discussed among experts and, in any case, need to be verified to be a very low false non-trigger rate. Often, the protection of a system against false non-triggers is tested only for some systems (e.g., automatic emergency braking) based on dedicated tests on test sections (e.g., Euro NCAP), without direct indicators of performance in actual traffic events. In HAF, a large amount of data is input, but this data is generally insufficient for the statistical validation or indicators that would be needed to protect the target values ​​actually required. The long-term driving performance to be achieved can be identified in relation to the acceptable rate of false non-triggers. [Means for solving the problem]

[0009] Disclosure of the invention A first general aspect of this disclosure relates to a computer-implemented method for protecting a system against false non-triggers. This method may include receiving a time series of criticality, the system may include a function that is triggered when the criticality meets a first predetermined criterion. This method may further include calculating a criterion time series, the criterion of which may be a comparative criticality for triggering a function of the system. This method may further include calculating an error scale time series based on at least the criticality time series and the criterion time series, wherein a non-trigger of a function of the system may be evaluated as a false non-trigger if a portion of the error scale time series meets a second predetermined criterion. This method may further include identifying at least one quasi-false non-trigger, wherein a non-trigger of a function of the system may be evaluated as a quasi-false non-trigger if a portion of the error scale time series meets a third predetermined criterion but does not meet a second predetermined criterion.

[0010] A second general aspect of the present disclosure relates to a computer-implemented method for protecting a system against false non-triggers, wherein the computer-implemented method may be implemented independently of or following the computer-implemented method according to the first general aspect (or its embodiments), wherein the system includes a function that can be triggered if a certain criticality satisfies a first predetermined criterion, and the system may include a system of subsystems having at least one subsystem. The method may include receiving a time series of subcriticalities for each subsystem of the system, wherein the criticality may relate to a subcriticality of a subsystem of the system, and optionally the criticality relates to whether or to what extent at least one subcriticality satisfies a first predetermined subcriterion to which it belongs. This method may further include, for each subsystem of the system, the calculation of a time series of subcriteria, where each subcriteria is a comparative subcritarity to the subsystem's subcritarity. This method may further include, for each subsystem of the system, the calculation of a time series of a suberror scale based on at least the time series of the subsystem's subcritarity and the time series of the subsystem's subcriteria, where a time series of vectors of the suberror scale may be generated. Evaluation errors in a subsystem may exist if a portion of the time series of the suberror scale satisfies a second predetermined subcriteria to which it belongs (e.g., for each subsystem). Quasi-evaluation errors in a subsystem may exist if a portion of the time series of the suberror scale satisfies a third predetermined subcriteria to which it belongs, but does not satisfy a second predetermined subcriteria to which it belongs. This method may further include, for each subsystem of the system, the identification of at least one quasi-evaluation error and / or at least one evaluation error.

[0011] Selectively, a method for protecting a system against false non-triggers according to a first general aspect (or embodiment) of the present disclosure may be performed following a computer-implemented method for protecting a system against false non-triggers according to a second general aspect (or embodiment) of the present disclosure.

[0012] A third general aspect of this disclosure relates to a system protected according to a computer-implemented method for protecting the system against false non-triggers, as described in the first general aspect (or its embodiments) and / or the second general aspect (or its embodiments).

[0013] As described in the prior art, protection of a system against false non-triggers is important for the reliable operation of the system and / or its environment. Such protection is all the more important the more dramatic the effect of a false non-trigger on the system and / or its environment may be. The methods for protecting a system against false non-triggers proposed in this disclosure, according to the first general aspect (or embodiment thereof) and / or the second general aspect (or embodiment thereof), can be used for function or system release. Selectively or additionally, the method can be used for diagnostic purposes, i.e., (continuous) monitoring, during system operation, for example, during the operation of a vehicle's running system. In this case, the method for protecting a system against false non-triggers may be implemented, for example, in a control device (or control unit). Selectively, each time series may be sent to a cloud server and evaluated by the cloud server. In this case, the method described in this disclosure may be implemented in a computer system, which includes, for example, at least one control unit in the system, a cloud server, and a corresponding network for data communication between at least one control unit and the cloud server.

[0014] Similarly, as already described in the prior art, non-triggers and / or false non-triggers in a system can be rare. Particularly in safety-critical systems, especially in driving systems such as emergency braking assistance and / or avoidance assistance for collision avoidance, it is often the case that a system is only usable when false non-triggers are rare, for example, rarer than the typical number of accidents in the range of use. Despite being well-desired, the drawback of a low false non-trigger rate (or false non-trigger probability) is that false non-triggers may not be measurable, or virtually unmeasurable, in long-term driving tests. Consequently, the actual false non-trigger rate may not be projectable at all to a specific operating time of the system, especially during the system's development period, or it may only be projectable to a specific operating time of the system with a large error / large uncertainty. Similarly, reliable projections are almost impossible for many such systems (e.g., a fleet of vehicles in the field). This makes it difficult to achieve the necessary release of functionality and / or the system.

[0015] The advantage of the method for protecting a system from false non-triggers proposed in this disclosure is that, in addition to false non-triggers, quasi-false non-triggers and / or evaluation errors in subsystems, quasi-evaluation errors in subsystems may also be identifiable and evaluateable. A quasi-false non-trigger may be a situation in which the system already exhibits signs of erroneous behavior, but this erroneous behavior has not yet clearly manifested as a false non-trigger. Similarly, a quasi-evaluation error in a subsystem may be a situation in which the subsystem already exhibits signs of erroneous behavior, but this erroneous behavior has not yet clearly manifested as an evaluation error. In practice, quasi-false non-triggers may occur more frequently than false non-triggers. This allows for better and more accurate identification of the rate of false non-triggers (due to the relatively large amount of data). Furthermore, this improves the in-operation diagnosis of the system. For example, if quasi-false non-triggers increase, the system user (e.g., vehicle occupants or drivers) may be provided with information and alerts before a false non-trigger may occur, and before a serious accident may be caused, for example. Alarms may include, for example, error displays and / or requests for service personnel visits. In the case of a cloud server, for example, identified false non-triggers and / or identified quasi-false non-triggers can be communicated back into the system, particularly to the driving system. Thus, the identification of quasi-false non-triggers is an analysis of the system's operating state, which can help in identifying errors within the system.

[0016] Evaluation errors in subsystems can be rare, but not necessarily rare (at least not as rare as false non-triggers). The more subsystems a system contains, and / or the more distinct the subsystems in the system, the higher the probability that each subsystem will have a different evaluation, and consequently, the higher the probability of evaluation errors. For example, a system may contain at least two subsystems (e.g., a radar subsystem and a video subsystem), where the criticality may arise from an OR combination of the subcriticalities of these subsystems. In such an OR combination (ODER-Fusion in German), which may be used in systems that are particularly critical in terms of safety, the system's function is triggered only if, for example, at least one of the subsystems decides to trigger based on each subcriticality itself. Thus, the probability of false non-triggers is low here. On the other hand, a configuration is entirely possible in which, for example, one subsystem decides against triggering, while at least one of the other subsystems decides in favor of triggering. In this case, an evaluation error exists in at least one subsystem.

[0017] In any case, similarly, quasi-evaluation errors may occur more frequently than evaluation errors in subsystems. Quasi-evaluation errors and / or evaluation errors can similarly be taken into account in calculating the rate of false non-triggers. This can further improve the accuracy or reliability of the rate of false non-triggers. Evaluation errors and / or quasi-evaluation errors in subsystems can also be taken into account while the system is running. For example, if evaluation errors and / or quasi-evaluation errors increase in a subsystem (for example, if the camera sensor is dirty and the camera subsystem can no longer analyze meaningful images), the subsystem can be deactivated. Evaluation errors may occur more frequently than false non-triggers, and / or quasi-evaluation errors may occur more frequently than quasi-false non-triggers. (Therefore,) by taking evaluation errors and / or quasi-evaluation errors into account, a larger amount of data can be generated, and based on this larger amount of data, the rate of false non-triggers can be calculated and / or diagnosed. Thus, by the method according to the second general aspect (or its embodiments), protection against false non-triggers and, additionally, system awareness at the subsystem level can also be improved.

[0018] Means for identifying quasi-false non-triggers and / or quasi-evaluation errors can be based on the fact that the time series relating to criticality, criteria and / or error scales, or subcriticality, sub-criteria and / or sub-error scales are (almost) continuous, i.e., non-binary. This allows for the analysis and evaluation of intermediate states, such as between triggers and non-triggers, and between correct non-triggers and false non-triggers.

[0019] The methods proposed in this disclosure according to the first aspect (or its embodiments) and / or the second aspect (or its embodiments) can be used to increase the amount of data, as described above, but can also be used to shorten the long-running tests required for release (for example, by generating the same amount of data / representation as the prior art in the shortened long-running tests). This can reduce development effort and / or costs. A further advantage may be the ability to compare quasi-false non-triggers and / or quasi-evaluation errors in systems with various but occasionally similar projects (for example, in subsequent applications). Such comparisons may be particularly advantageous when the system is in the early stages of development and sufficient long-running data is not yet available.

[0020] The proposed second aspect (or its embodiment) of the method may be applied advantageously (for example, without using the proposed first aspect). This is when the acceptable rate of false non-triggers is low, for example according to specifications, and long-term running tests must be of a length that is no longer technically and / or economically significant, or sometimes a length that is no longer feasible, for example, in a tightly timed product development cycle. The proposed second aspect (or its embodiment) of the method may be applied even more advantageously, for example, when it is shown that the number of false non-triggers and / or quasi-false false non-triggers is too small to reliably calculate the rate of false non-triggers in the proposed first aspect (or its embodiment). The probability of such circumstances occurring may increase as the system development progresses. The proposed second aspect (or its embodiment) of the method may be applied even more advantageously, for example, when it is shown that layering or heterogeneity occurs in the error scale for quasi-false false non-triggers in the proposed first aspect (or its embodiment). Layering or heterogeneity may be based, for example, on the fact that the distribution of error measures for a situation arises from at least two distributions (e.g., sub-error measures of at least two subsystems of the system). For example, layering / heterogeneity may exist if the observed quasi-false non-triggered error measures are distributed in subsets that are not related in terms of magnitude / interval of the error measures (e.g., in two or more discrete clusters of the error measures). Selectively or additionally, layering / heterogeneity may also exist, for example, if the fit of the distribution function to the error measures for a situation is insufficient (despite there being reasons for the assumption that the modeling is correct). Layering / heterogeneity may occur, for example, in coupled systems (e.g., OR coupled systems). In this case, in the method according to the second aspect (or its embodiments), situations with similar error measures (e.g., discrete clusters of error measures) can often be assigned evaluation errors and / or quasi-evaluation errors in one of the subsystems of multiple subsystems.This enables the subsystem to be optimized as intended, thereby further reducing the rate of false non - triggers, and ultimately making the system more robust. Furthermore, this can lead to design decisions for subsequent applications.

Brief Description of the Drawings

[0021] [Figure 1a] A diagram schematically showing a computer - implemented method for protecting a system against false non - triggers according to a first aspect (or an embodiment thereof). [Figure 1b] A diagram schematically showing a computer - implemented method for protecting a system against false non - triggers according to a first aspect (or an embodiment thereof). [Figure 1c] A diagram schematically showing a computer - implemented method for protecting a system against false non - triggers according to a first aspect (or an embodiment thereof). [Figure 1d] A diagram schematically showing a computer - implemented method for protecting a system against false non - triggers according to a second aspect (or an embodiment thereof). [Figure 2a] A diagram showing an exemplary relevance of the criticality or criteria of a sensor system and / or another sensor system. [Figure 2b] A diagram showing an exemplary relevance of the criticality or criteria of a sensor system and / or another sensor system. [Figure 3] A diagram showing a system including two subsystems. [Figure 4a] A diagram visualizing an exemplary functional relationship of an error metric in relation to criticality and criteria. [Figure 4b] A diagram visualizing an exemplary functional relationship of an error metric in relation to criticality and criteria. [Figure 5] A diagram visualizing the calculation (extrapolation) of the rate of false non - triggers based on the distribution function of an error metric.

Modes for Carrying Out the Invention

[0022] explanation Methods 100 and 101 proposed in this disclosure aim to protect System 200 from false negatives of system functions. As discussed in the prior art, a false negative is, for example, a failure to trigger a function that should have been performed from the perspective of a knowledgeable observer. Therefore, a missed trigger of a function can be a false negative. A delayed intervention can also be evaluated as a missed intervention, and thus as a false negative, in relation to the time when the intervention should have been performed. Hereafter, a false negative may be referred to as a critical event or critical situation.

[0023] First, a computer-implemented method 100 for protecting system 200 against false non-triggers is disclosed, which may include receiving a time series 110 of criticality (KRIT). Here, system 200 may include a function that can be triggered if the criticality (at a point in the time series of criticality, or at a point in the time series of criticality, e.g., the last three points in the time series) satisfies a first predetermined criterion. The criticality (KRIT) at each point in the time series can be encoded by at least two numerical values ​​(e.g., binary: 0 and 1; or ternary: 0, 1 and 2; or by a number of numerical values, e.g., in MATLAB vector notation: [0.00:0.01:1.00]). In other words, criticality can be considered a numerical measure, which is important for triggering or not triggering based on a first predetermined criterion. However, high criticality values ​​may not generate a trigger (i.e., they may not generate a trigger in relation to a first predetermined criterion). In the context of a driving system whose function is directed towards collision avoidance, criticality may be derived for situation estimation from quantities such as time to collision (TCC), the scaled delay amount required to avoid an accident (brake threat number, BTN), and / or a time / distance-based index. If the first predetermined criterion is met by the criticality (at a point in the time series or for a part of the criticality time series), the function is triggered. Conversely, if the first predetermined criterion is not met by the criticality, the triggering is suspended. For example, the first predetermined criterion may be met if the criticality (KRIT) exceeds a first predetermined threshold at one or more sequentially consecutive points in the criticality (KRIT) time series. Criticality (KRIT) may be defined as a multidimensional numerical measure that is important for a trigger or non-trigger at each point in time in a time series, based on a first predetermined criterion.In other words, the time series of criticality (KRIT) may be multidimensional. The multidimensional numerical scale (each point in the time series) may include, for example, vectors, matrices, and / or tensors. Selectively, the multidimensional time series of criticality may include one-dimensional time series (one for each element of the multidimensional time series).

[0024] Method 100 may further include a calculation 120 of a time series of Criteria (REF), where Criteria (REF) may be a comparative criticality for triggering a function of System 200. Criteria (REF) may, at each point in the time series, be encoded by at least two numerical values ​​(e.g., binary: 0 and 1; or ternary: 0, 1 and 2; or by a number of numerical values, e.g., in MATLAB vector notation: [0.00:0.01:1.00]). In particular, Criteria (REF) may be encoded in a manner similar to that of Criticality (KRIT). Such encoding may be advantageous because, in this case, it is possible to check whether the trigger was authorized or not according to the same first predetermined criterion. Advantageously, Criteria (see below) encodes a larger system awareness, for example, in terms of time and / or by a special sensor system. Criteria (REF) may, at each point in the time series, be a multidimensional numerical measure, for example, similar to Criticality (KRIT). In other words, the reference (REF) time series may be multidimensional. The multidimensional numerical scale (each point in the time series) may include, for example, vectors, matrices, and / or tensors. Optionally, the reference multidimensional time series may include one-dimensional time series (one for each element of the multidimensional time series).

[0025] Method 100 may further include a calculation 130 of a time series of an error measure (TPI) (also known as: overall system-TPI) based on at least a time series of criticality (KRIT) and a time series of criterion (REF), wherein a non-trigger of a function of system 200 may be evaluated as a false non-trigger if a portion of the time series of the error measure (TPI) (e.g., the last time point in the time series or the last three time points in the time series) satisfies a second predetermined criterion. The error measure (technical performance indicator, TPI) may optionally also be called an evaluation measure. For example, the second predetermined criterion may be satisfied if the error measure (TPI) exceeds a second predetermined threshold (also known as: error threshold, θ0) at one or more time points in a temporally sequential time series of the error measure (TPI). Selectively or additionally, the second predetermined criterion may be satisfied if the time series of the error measure (TPI) has local extrema (e.g., a maximum value exceeding another second threshold). The error scale (TPI) can similarly be encoded at each point in the time series by at least two numerical values ​​(e.g., binary: 0 and 1; or ternary: 0, 1 and 2; or by a number of numerical values, for example in MATLAB vector notation: [0.00:0.01:1.00]).

[0026] An error metric (TPI) calculated from criticality (KRIT) and a criterion (REF) can be considered as statistically evaluable information about the system's potential error behavior. In other words, the two quantities (KRIT, REF) can be combined, here using intelligent rules, into a single error metric (TPI), which, for example, in the case of a given set of first, second, and third judgment criteria with respective thresholds, can satisfy at least the following conditions: • If no system errors exist, TPI may be small or equivalent to a selected standard value (e.g., TPI=0). For example, in this system (e.g., an AEB system), if only false non-triggers should be evaluated and false triggers should not be evaluated, then the KRIT≧REF case would not be critical and would map to TPI=0. KRIT≧REF may mean that this system evaluates the situation critically rather than objectively justifying it. Therefore, in such cases, a trigger will likely occur even if it is not justified (e.g., in AEB: emergency braking will be triggered). • When KRIT remains constant and REF increases, TPI may increase; conversely, when REF decreases, TPI may decrease. • When REF is constant and KRIT increases, TPI may decrease, and when KRIT decreases, TPI may increase. • If the TPI exceeds a second predetermined threshold, this may be a false response (i.e., a false non-trigger) of the system.

[0027] These conditions can be satisfied by a number of functions. Therefore, an appropriate function can be selected, which may be used, for example, to map the partial subjective falsity of criticality to the current situation using appropriate parameters. For example, consider an AEB system that initiates emergency braking when the KRIT value is greater than a first predetermined threshold (e.g., 0.9 or 1). In this case, it must be answered, for example, what TPI value is assigned to a situation with a criticality of 0.4 and a criterion of 0.6, and whether other situations with a criticality of 0.7 and a criterion of 0.9 should obtain equivalent values. In both cases, the difference between KRIT and REF is the same, but in the second situation, on the one hand, the system is near the trigger based on a relatively high KRIT value, which may be largely justified based on a relatively high REF value.

[0028] As one example, the time series calculation of the Error Scale (TPI) can be performed as follows:

number

[0029] Method 100 may further include the identification of at least one subcritical non-trigger (also referred to as a subcritical event or situation) 140, where a non-trigger of the function of System 200 may be evaluated as a subcritical non-trigger if a portion of the time series of the Error Scale (TPI) (e.g., the last point in the time series or the last three points in the time series) satisfies a third predetermined criterion but does not satisfy a second predetermined criterion. For example, the third predetermined criterion may be satisfied if the Error Scale (TPI) exceeds a third predetermined threshold (also known as a sub-error threshold) at one or more points in a temporally sequential time series of the Error Scale (TPI), but does not exceed a second predetermined threshold (error threshold), for example. Selectively or additionally, the third predetermined criterion may be satisfied if the time series of the Error Scale (TPI) has a local extremum (e.g., a maximum value that exceeds another third threshold).

[0030] The term “quasi-false non-trigger” can be understood as “quasi” relating to “false non-trigger” as a whole. A quasi-false non-trigger may be specifically (and by definition) a non-trigger.

[0031] Figures 1a to 1c schematically illustrate a computer-implemented method 100 for protecting the system against false non-triggers according to a first aspect (or embodiment thereof). Unlike those shown, steps 110 and 120, for example, may be performed in any order, as do the optional steps 111 and 121.

[0032] The time series of criticality (KRIT) may be assumed to be (approximately) continuous. Selectively or additionally, the time series of criterion (REF) may be assumed to be (approximately) continuous. Selectively or additionally, the time series of error measure (TPI) may be assumed to be (approximately) continuous. A time series may be assumed to be continuous if the values ​​(i.e., here: KRIT, REF, and / or TPI) can exist within a continuum, for example, within an interval of real numbers (e.g., [0,1]). Selectively, a continuum, or more precisely, a quasi-continuum, may be given by a sufficiently large number of discrete values ​​in a real interval. For example, the numerical values ​​[0.00:0.01:1.00] may be considered a (quasi)continuum in MATLAB vector notation. Typically, in a time series, another continuity is also given in time, for example, at equal intervals (e.g., a 100Hz clock) or non-equal intervals. Furthermore, in many systems, a (nearly) continuous criticality (KRIT) can be used to determine the strength of the trigger for the system's function; that is, the strength of the trigger (e.g., the brake strength in an AEB system) may sometimes be related to a portion of the (nearly) continuous criticality (KRIT) at the time of intervention decision (e.g., the last point in the time series, or the last three points in the time series). For example, a greater brake strength may be selected as the criticality increases.

[0033] At least one quasi-false non-trigger can be used in the operational diagnosis of system 200. This can affect the control of the system and / or provide information to the system user and / or issue an alarm. System 200 may be a driving system, and optionally may be a vehicle driving system. The driving system may be a driver assistance system (FAS, e.g., AEB) or an automated driving system, and in particular may be a highly automated driving system (HAF). Particularly in safety-critical systems and / or environmental detection systems (e.g., for collision avoidance), this diagnosis can enhance the safety of the system, the safety of the system user and / or the system environment.

[0034] Selectively, system 200 may be a system that does not necessarily have to be a driving system. Such a system may be, for example, an alarm system, the function of which includes an alarm trigger. This system may also be critical in terms of safety and / or environmentally sensing (via at least one sensor system).

[0035] Method 100 may further include a modeling 150 of a functional relationship to a (univariate) distribution function or an (univariate) density function of an error scale, based at least on at least one quasi-false non-trigger, as shown in Figures 1a and 1b. For example, the parameters of a parametric (univariate) distribution function can be fitted to at least one quasi-false non-trigger, and / or, in fact, to a number of quasi-false non-triggers. Figure 5 shows an example of such a model, where vertical bars represent the (relative) frequency of quasi-false non-triggers and the (continuous, univariate) distribution function is fitted. Figure 5 shows the density function of the distribution function.

[0036] Furthermore, Method 100 may include the calculation of a false non-trigger rate 160 based on a distribution function of error measures for a set of (general) error measures (TPIs) that satisfy a second predetermined criterion, as shown in Figures 1a and 1b. In other words, the distribution function of error measures for a set of critical events may be evaluated. For example, this set of error measures (TPIs) may be defined by all error measures that exceed a second predetermined threshold (i.e., above the error threshold θ0), as shown in Figure 5. The false non-trigger rate may be used, for example, to compare the system with the false non-trigger rate of other systems that have possibly already been released and / or successfully tested. The combination of modeling a functional relationship 150 for the distribution function and evaluating it for a set of critical events may be called extrapolation, because it is not necessary to measure false non-triggers, yet predictions about their probability are still possible. Extrapolation is particularly significant when the time series of error measures (TPIs) is (almost) continuous.

[0037] The functional relationship may be a parameterizable statistical model, and optionally, a parameterizable statistical model from extreme value theory. In selecting the distribution function of the error scale, i.e., the functional relationship of the statistical model, the following may be important: that both critical and subcritical events occur rarely in the system, and that both types of events can be represented, for example, by particularly large / small values ​​or extreme values ​​of a variable (here, TPI), and / or by local extreme values ​​of TPI, for example, by exceeding / falling above each appropriate high / low threshold. In such cases, a family of general, system-nonspecific statistical models from extreme value theory can be used for extrapolation. Generally, statistical models (e.g., probability distributions) for describing the distribution of system errors may be system-specific and therefore not generalizable. However, here, specifically, the aim is to perform extrapolation to particularly rare error events, so only intervals of the error scale that sufficiently approximate the critical threshold and therefore are assumed to occur with extremely low probability may be relevant (subcritical events). Under these preconditions, instead of system-specific models, general statistical properties of rare events can be utilized. In particular, the maximum value of the error scale within a sufficiently large recording interval (block maxima) or exceeding a sufficiently high selected threshold (peak-over threshold) can be described by so-called extreme value models, such as so-called extreme value distributions or generalized Pareto distributions. The effectiveness of extrapolation is coupled with the effectiveness of the assumed statistical model with respect to the error scale. If a suitable model is identified, its model parameters may be fitted to the system based on the measured course of the error scale (TPI). The fitted model in this case represents the probability that the error scale exceeds a set value. Extrapolation allows for the use of the model to make predictions about the frequency of critical errors occurring (for example, when a second predetermined threshold / error threshold θ0 is exceeded), even though such errors should not exist in the input long-term driving data.

[0038] Method 100 may further include a check 170 to determine whether the rate of false non-triggers (and / or the upper limit of the confidence interval for the rate of false non-triggers) satisfies a fourth predetermined criterion, as shown in Figures 1a and 1b. This method may further include a release 180 of the function of the system 200 if the rate of false non-triggers satisfies the fourth predetermined criterion. Optionally, the fourth predetermined criterion is satisfied if the rate of false non-triggers (and / or the upper limit of the confidence interval for the rate of false non-triggers) is less than a predetermined acceptable rate. Thus, Method 100 provides a quantifiable release criterion that can be used in system development (e.g., in the case of HAF) and / or in system operation (e.g., in the case of a vehicle with a driver assistance system).

[0039] Method 100 may further include receiving a data timesheet of system 200 (e.g., a vehicle data timesheet). Optionally, the data timesheet may include at least one time series of data based on measurements using the sensor system 210 of system 200. Receiving the time series of criticality (KRIT) 110 may include receiving the time series of criticality (KRIT) from the data timesheet, as schematically shown in Figure 2a. For example, in the case of a driving system, the time series of criticality (KRIT) can be received via at least one (CAN) interface of the driving system. Generally, the data timesheet may be a data timesheet for one or more long-term driving tests of the system, where the system may be in an active and / or inactive state. The data timesheet may further be input and processed live during operation, or it may be input at some point in the past. The advantage of receiving a time series of criticalities (KRIT) from a data timesheet is that these criticalities accurately correspond to the criticalities responsible for trigger and / or non-trigger decisions during system operation. This ensures that the actual functionality of the system is demonstrated.

[0040] Selectively or additionally, the receipt of a time series of criticality (KRIT) 110 may include the calculation of a time series of criticality (KRIT) based on a data timesheet. In this case, the criticality (KRIT) may relate to the data on the data timesheet at the same time point in the time series. Selectively or additionally, the criticality (KRIT) may relate to the data on the data timesheet at at least one earlier time point in the time series. In other words, in this case, the criticality may relate only to the present (i.e., the last time point) and / or the past.

[0041] The calculation of the reference (REF) time series 120 may similarly be based on a data timesheet, as schematically shown in Figure 2a. The reference (REF) may relate to the data on the data timesheet at the same time point in the time series. Selectively or additionally, the reference (REF) may relate to the data on the data timesheet at at least one earlier time point in the time series. Selectively or additionally, the reference (REF) may relate to the data on the data timesheet at at least one subsequent time point in the time series. In other words: in this case, the reference may relate to the present, past and / or future. The fact that the reference may relate to at least one future data point allows the reference to be calculated only with a time delay (i.e., it cannot be calculated at the time of the intervention decision). This, on the other hand, allows for a better assessment of the event / situation than criticality, because it allows for consideration of how the event / situation has progressed. Therefore, the criterion is a more meaningful comparative criticality to criticality, and can correspond to the viewpoint of a discerning observer. In many cases, a delay of a few seconds (e.g., <1s, <2s, <5s, <10s, <30s, <60s) may be sufficient to allow for a final evaluation of the event / situation. However, such time delays have also been found useful during system operation, particularly in diagnostics for informational and / or alarm purposes. Selectively, steps 110, 120, and 130 may be performed within system 200, as illustrated in Figure 2a.

[0042] Method 100 may further include receiving another data timesheet, as schematically shown in Figure 2b. Optionally, this additional data timesheet may include at least one additional time series of data based on further measurements using another sensor system 310. However, this additional sensor system does not have to be part of the system. This additional sensor system may be, for example, a special system for a system to be tested during long-distance driving. This additional sensor system may be used to ensure that the criteria correspond to the viewpoint of a knowledgeable observer. Optionally, steps 110, 120, and 130 may be performed within the system 200, as shown in Figure 2b.

[0043] Method 100 may further include the formation of an overall data timesheet from a data timesheet and / or another data timesheet. The calculation of the reference (REF) time series 120 may be based on an overall data timesheet (i.e., a vehicle data timesheet and / or another vehicle data timesheet). The reference (REF) may, at each point in the time series, relate to data in the overall data timesheet at the same point in time. Selectively or additionally, the reference (REF) may, at each point in the time series, relate to data in the overall data timesheet at at least one earlier point in time. Selectively or additionally, the reference (REF) may, at each point in the time series, relate to data in the overall data timesheet at at least one subsequent point in time. By the reference being selectively or additionally related to another sensor system, for example, as opposed to criticality, the reference can evaluate events / situations better than criticality and thus become a more meaningful comparative criticality to criticality. Advantageously, it has been found that the calculation of the reference can be automated. This allows for efficient evaluation of even long-term driving tests and enables analysis of the rate of false non-triggers.

[0044] In exemplary embodiments of AEB systems where environmental sensor data is not required, information about the actual behavior of the vehicle can be retrospectively used (e.g., via vehicle condition sensors) to evaluate whether the criticality estimation (and consequently the trigger / non-trigger) was performed correctly. For example, in a post-simulation of an actual long-distance driving test, the function is not (actually) active and therefore cannot intervene, but it is still possible to analyze what the system did. For example, if the system detects that the preceding vehicle is slightly decelerating and believes it must react, and therefore assumes that, based on a criticality of 0.4 (on a criticality scale of 0 to 1), the system determines that an intervention / trigger according to a criticality of 0.4 (e.g., braking at 40% brake pressure) is appropriate at the present time (e.g., time t=0) to avoid an accident, it is possible to analyze whether the driver actually performed a braking operation with a similar strength (e.g., corresponding to a criticality of 0.4) during the period from t=0 to approximately t=3s. In this case, such information can be considered at the criterion (criterion A) at time t=0. For example, if the driver does not react at all during this period, the 40% criticality may be deemed inappropriate and therefore incorrect. In this case, the criterion A at time t=0 (or a later time) can be set to, for example, 0. For the TPI at time t=0, a value of, for example, 0.4 may be calculated / defined here. On the other hand, if the driver reacts extremely strongly, for example, by applying emergency brakes, this criticality can be evaluated as excessively low, or, for example, an overreaction by the driver can be demonstrated. In either case, there was no excessively high criticality. In this case, the criterion A at time t=0 (or a later time) can be set to, for example, 1. For TPI at time t=0, a value of, for example, 0 may be calculated / defined here.

[0045] For example, at any point in time (e.g., t=0), criterion A can be defined as follows: For example, for future time in seconds, such as t=(0.1,0.2,…,3.0), the rate of change from time 0 to time t can be calculated. Based on this braking model (taking into account latency, maximum jerk, etc.), we can calculate the criticality the system will output to enable such a speed reduction from 0 to t. This allows us to calculate a baseline candidate at each time point t = (0.1, 0.2, ..., 3.0). In this case, for example, the reference value for reference A at time t=0 can be calculated as the maximum value of all reference candidates at time t=(0.1, 0.2, ..., 3.0). Such a method may be advantageous because it takes into account that the driver may have reacted earlier or later in some cases, and therefore acted with relatively less force or relatively more force. Furthermore, it is possible to check whether the driver reacted in a non-longitudinal direction (i.e., laterally) by, for example, switching to another lane laterally. For this purpose, it is possible to check whether the driver performed a lateral motion from t=0 to t=(0.1, 0.2, ..., 3.0). In some cases, additional curve driving must be considered, for example, based on the progression of steering angle from the (vehicle) data timesheet.

[0046] Considering the analysis of possible false non-triggers, the criterion calculated in such examples (criterion A) may have a gap, but this gap can be estimated by an additional criterion (criterion B). However, in the above example where criticality is assessed as requiring 40% intervention, the driver may react more strongly than necessary. In this case, criterion A will be higher than objectively required. If the criticality assessment is then made at this value, the function will be unfairly criticized for predicting an excessively low criticality. In other examples where the function can / should react only to vehicles, for example, the driver may apply the brakes excessively to pedestrians or radar speed enforcement devices. Thus, criterion A will receive a high value, while the criticality should naturally be 0 (because criticality should not / cannot react to these objects). In such cases, it may be beneficial to incorporate other data (in the sense of a "second opinion") into the analysis to examine whether the driver's actual actions are important to the system's function. For this purpose, a separate sensor system 310 (which may not be used in, for example, mass production systems, and is used only in long-term driving tests, for example, in relation to system release) is used, which is more expensive than the sensor system 210 that is standardly used in the system, but can detect different data (for example, a different camera system for a different field of view). Thus, steering can be evaluated more reliably, and a more reliable criterion (criterion B) can be calculated. Thus, this can, for example, resolve driver overreactions. Criterion B may be more accurate than criticality, and may be used to compare with criticality, and an error scale may be calculated. Additional sensor devices or reference sensor images obtained therefrom may be used for further calculations, for example, to collect data by so-called reverse tracking.

[0047] The following methods may arise: If the driver does not react, i.e., if criterion A is 0, this allows the entire criterion to be set to 0. Therefore, it can be determined that no steering is necessary. • If the driver reacts, the type of reaction (e.g., avoidance, braking, or a combination thereof) can be identified. In this case, the values ​​of Criterion A and Criterion B can be evaluated for a given type of reaction. Next, the baseline value can be identified by comparing baseline A and baseline B. A minimum value can be selected to identify overreactions and avoid misjudgments. However, other operations (such as averaging) could also be considered.

[0048] Criterion B may be a criterion (REF), the time series of which is calculated in step 120 of a computer-implemented method 100 for protecting the system against false non-triggers. Optionally, the calculation of the time series of the criterion (REF) (120) may include the calculation of a criterion from criterion A and criterion B.

[0049] The calculation of the time series of the error measure (TPI) 130 may include the calculation of the time series of the error measure such that the error measure (TPI) relates to different time points in the time series relating to criticality (KRIT) and criterion (REF) at at least one point in the time series.

[0050] The calculation of the time series of the Error Scale (TPI)130 may further be based on data from the data timesheet and / or the overall data timesheet. This allows for consideration of temporal effects that may cause differences in KRIT and REF. In AEB, for example, if the system identified some degree of criticality in this situation, but the driver gradually mitigated the situation with a slight delay (but more strongly), the previous KRIT was not excessively high and was justified. That is, in some cases, a temporal allocation between KRIT and REF values ​​may be necessary to correlate similar underlying events as much as possible. Such an allocation can be done, for example, by analyzing each time window around the criticality value.

[0051] Method 100 may further include the identification of at least one false non-trigger, and optionally here, a modeling 150 of a functional relationship 150 to the distribution function of the error scale or to the (univariate) density function of the error scale is further performed based on at least one false non-trigger. If false non-triggers occur despite their rarity, they can be advantageously utilized in the calculation of false non-triggers.

[0052] Furthermore, computer-implemented methods 100, 101 for protecting a system / the above-mentioned system against false non-triggers (according to a second aspect) are disclosed, wherein the system includes a function / the above-mentioned function that is triggered when a certain criticality / the above-mentioned criticality satisfies a first predetermined criterion / the above-mentioned first predetermined criterion, wherein the system 200 includes a system of subsystems 220, 221, each having at least one subsystem 220. The system 200 may include, for example, one subsystem (for example, in this case this one subsystem may be the system), one or more subsystems, two or more subsystems, three or more subsystems, four or more subsystems, five or more subsystems, ten or more subsystems, or twenty or more subsystems. Methods 100 and 101 may include receiving a time series 111 of subcriticalities (KRIT1, KRIT2) for each subsystem 220 and 221 of system 200, where the criticality (KRIT) may be related to the subcriticalities (KRIT1, KRIT2) of subsystems 220 and 221 of system 200. The criticality (KRIT) may be related to whether or to what extent at least one subcriticality (KRIT1, KRIT2) satisfies a first predetermined subcriterion (for each subsystem) to which it belongs. Selectively, the criticality (KRIT) may be related to whether or to what extent all subcriticalities (KRIT1, KRIT2) satisfy a first predetermined subcriterion to which each belongs. For example, criticality (KRIT) may relate to whether or to what extent at least two subcriticalities (KRIT1, KRIT2, KRIT3) satisfy a first predetermined subcriterion to which each belongs (for each subsystem) ("2-out-of-3-Fusion").Furthermore, for example, criticality (KRIT) may relate to whether or to what extent at least m subcriticalities (KRIT1, KRIT2, ..., KRITn) satisfy a first predetermined subcriterion (for each subsystem) to which they belong ("m-out of-n-Fusion"), where m is less than or equal to n. Furthermore, for example, criticality (KRIT) may relate to whether or to what extent all subcriticalities or combinations from subsets of subcriticalities satisfy a first predetermined subcriterion (common). For example, it may be the product of all subcriticalities, the product of the largest or smallest m subcriticalities, etc. In particular, it is not necessarily required that one or more subcriticalities themselves satisfy the first predetermined subcriterion (for each subsystem) to which they belong.

[0053] The time series of subcriticalities may be (almost) continuous, like the time series of criticalities, and / or may have the same (possible) range of values. Furthermore, in each subsystem, the time series of subcriticalities may be multidimensional (e.g., vectors, matrices, or tensors). Criticality is, for example, the product of subcriticalities. KRIT = KRIT1 × KRIT2 × ... This can arise from, where, in the case of at least one (almost) continuous subcriticality, a (almost) continuous criticality can arise. Selectively, a criticality can be, for example, an AND combination of subcriticalities (AND-Fusion, UND-Fusion in German). KRIT=KRIT1&KRIT2&··· This can arise from, and here, discrete (and binary) criticality results. In selective AND fusion, criticality is, for example, KRIT=min(KRIT1,KRIT2,···) This can result from, in which (almost) continuous criticality can occur. Selectively, criticality can be, for example, an OR fusion of subcriticalities. KRIT=KRIT1|KRIT2|··· This can arise from, and here, discrete (and binary) criticality results. In selective OR fusion, criticality is, for example, KRIT=max(KRIT1,KRIT2,···) This can result from, and in this case, (almost) continuous criticality can occur as a result.

[0054] Furthermore, "mixed forms" may also occur. Here, for example, the criticality (KRIT) may be (almost) continuous in part of its value range and discrete in other parts. On the other hand, the entire value range may also be discrete. Subcriticalities and / or sub-error measures (see below) may also each have such "mixed forms". Furthermore, the criticality (KRIT), criterion (REF), and / or error measure (TPI) from Method 100 according to the first general embodiment (or its embodiment) may also each have such "mixed forms".

[0055] Methods 100 and 101 may further include a calculation 121 of the time series of subcriteria (REF1, REF2) for each subsystem 220 and 221 of system 200. Here, the subcriteria may be one comparison subcriteria for each subcriticality (KRIT1, KRIT2) of subsystems 220 and 221. The time series of the subcriteria may be (almost) continuous, and / or have the same (possible) range of values, like the time series of the criterion. Furthermore, in each subsystem, the time series of the subcriteria may be multidimensional (e.g., vector, matrix, or tensor). In practice, at least one time series of the subcriteria (REF1, REF2) (or all time series of the subcriteria) may be the time series of the criterion (REF). This is especially true if each subsystem makes similar trigger or non-trigger decisions based on its subcriticality, for example, in an AND join.

[0056] Methods 100, 101 may further include, for each subsystem 220, 221 of System 200, the calculation 131 of a time series of sub-error scales (TPI1, TPI2) based on at least the time series of subcriticalities (KRIT1, KRIT2) of subsystems 220, 221 and the time series of subcriteria (REF1, REF2) of subsystems 220, 221, where a time series of sub-error scale vectors may be generated (from the time series of sub-error scales of subsystems). In other words, each component of the sub-error scale vector may include the sub-error scale of subsystem (also known as subsystem-TPI). The sub-error scale vector can be referred to as the overall system-TPI. The time series of sub-error scales may be (approximately) continuous, and / or have a similar (possible) range of values, like the time series of error scales.

[0057] Methods 100, 101 (according to the second aspect) can be implemented independently of Method 100 (according to the first aspect). Optionally, they can be implemented additionally, particularly following Method 100 (according to the first aspect). Exemplary embodiments of computer-implemented Methods 100, 101 for protecting the system against false non-triggers are schematically shown in Figure 1d. A system 200 comprising two subsystems 220, 221 is schematically shown in Figure 3. When Methods 100, 101 (according to the second aspect) are implemented independently of Method 100 (according to the first aspect), Methods 100, 101 may include receiving a time series of criticality (KRIT) as shown in Figure 1d, where the system 200 includes a function / function described above that is triggered when the criticality satisfies a certain first predetermined criterion / the first predetermined criterion described above. The reception of the criticality (KRIT) time series 110 may include the calculation of the criticality (KRIT) time series from the subcriticality time series for each subsystem of the system.

[0058] In a vector space spanning sub-error vectors, a critical subset corresponding to false non-triggers may be defined. In other words, here, false non-triggers may be decomposed down to the subsystem level. The critical subset may be the first Cartesian product, especially in the case of OR joins, for example,

number

number

[0059] Furthermore, a subcritical subset may be defined in the vector space spanning the sub-error scale vectors as follows: Each element of the subcritical subset is not an element of the critical subset, but each may be defined to be close to at least one element of the critical subset according to a given distance criterion (e.g., based on Euclidean distance). Selectively or additionally, a subcritical subset may be defined to correspond to quasi-erroneous non-triggers that have been decomposed to the subsystem level. Selectively or additionally, a subcritical subset is a complementary element of the second Cartesian product.

number

number

number

[0060] The integration of a critical subset and a subcritical subset, either selectively or additionally, is a second Cartesian product.

number

number

number

[0061] Selectively or additionally, the integration of critical subsets and subcritical subsets is a complementary element of the second Cartesian product.

number

[0062] Evaluation errors in subsystems 220 and 221 may exist if a portion of the time series of the sub-error scales (TPI1, TPI2) satisfies the second predetermined sub-criterion to which they belong. Each second predetermined sub-criterion can be obtained from the boundary hyperplane of the first Cartesian product (for example, θ1 ≥ θ 1.0 Here, for example, θ1 is the sub-error measure (TPI) in the first subsystem. In other words, for example, for each subsystem, the sub-error measure belongs to a second predetermined sub-threshold (θ 1.0 ,θ 2.0 If the value exceeds the specified threshold, the second predetermined sub-criterion to which it belongs may be considered satisfied. Selectively or additionally, the second predetermined sub-criterion in the subsystem may be considered satisfied if the time series to which the sub-error scale (TPI) belongs has a local extremum (for example, the maximum value that exceeds another second sub-threshold to which it belongs).

[0063] Furthermore, a preliminary evaluation error in subsystems 220 and 221 may exist when a part of the time series of the sub-error metrics (TPI1, TPI2) (e.g., the last time point or the last three time points of the time series) satisfies the third predetermined sub-judgment criterion to which it belongs but does not satisfy the second predetermined sub-judgment criterion to which it belongs. Each third predetermined sub-judgment criterion can be obtained from the boundary hyperplane of the difference between the first Cartesian product and the second Cartesian product (e.g., [Number] ). In other words, for example, for each subsystem, when the sub-error metric exceeds the third predetermined sub-threshold [Number] to which it belongs but does not exceed the second sub-threshold (θ 1.0 , θ 2.0 ) to which it belongs, it may be considered that the third predetermined sub-judgment criterion to which it belongs is satisfied. Optionally or additionally, when the time series to which the sub-error metric (TPI) belongs has a local extreme value (e.g., a maximum value exceeding another third sub-threshold to which it belongs), it may be considered that the third predetermined sub-judgment criterion in the subsystem is satisfied.

[0064] Methods 100 and 101 may further include the identification 141 of at least one preliminary evaluation error for each of the subsystems 220 and 221 of the system 200. Optionally or additionally, methods 100 and 101 may include the identification 141 of at least one evaluation error for each of the subsystems 220 and 221 of the system 200. As already explained, evaluation errors in subsystems are not necessarily as rare as incorrect non-triggers.

[0065] At least one quasi-evaluation error (or further quasi-evaluation error) of subsystems 220 and 221 can be used in the operational diagnosis of system 200, particularly in the operational diagnosis of subsystems 220 and 221 of system 200. Selectively or additionally, at least one evaluation error (or further evaluation error) of subsystems 220 and 221 can be used in the operational diagnosis of system 200, particularly in the operational diagnosis of subsystems 220 and 221 of system 200. This allows for diagnosis at the subsystem level. If necessary, subsystems can be deactivated, for example.

[0066] Methods 100, 101 may further include a modeling 151 of a functional relationship for each subsystem 220, 221 of System 200, for a (univariate) distribution function of the sub-error scales (TPI1, TPI2) of subsystems 220, 221 (i.e., each component of the sub-error scale vector), or for the (univariate) density function of the sub-error scales of the subsystems, based on at least one quasi-evaluation error and / or at least one evaluation error in subsystems 220, 221. For example, for each subsystem, the parameters of the parametric (univariate) distribution function can be aligned to at least one quasi-evaluation error and / or, in practice, to a number of quasi-evaluation errors. In such alignment of parameters, at least one evaluation error and / or (optionally) a number of evaluation errors in subsystems can be considered, selectively or additionally. As in the case of Modeling the Functional Relationship to the Distribution Function (or Density Function) of the Error Scale 150, the functional relationship to the (univariate) distribution function (or density function) of the subsystem's sub-error scale may be considered a parameterizable statistical model, and optionally, it may be considered a parameterizable statistical model derived from extreme value theory.

[0067] Methods 100, 101 may further include, for each subsystem 220, 221, the identification 153 of at least one subsystem 220, 221 that requires optimization, based on the distribution function of a sub-error scale.

[0068] Methods 100 and 101 may further include the calculation of a functional relation 152 to the (multivariate) distribution function of the sub-error scale vectors, based on the distribution functions of subsystems 220 and 221 of system 200. The calculation 152 to the functional relation 152 to the (multivariate) distribution function of the sub-error scale vectors can be performed according to Sklar's theorem. In this case, under mild assumptions, a unique multivariate distribution function of the sub-error scale vectors can be obtained (e.g., via copulas). Selectively, the (multivariate) distribution function of the sub-error scale vectors can be calculated without using pre-calculation of the (univariate) distribution function for each subsystem.

[0069] The following explanation may be interpreted as relating to the calculation of the functional relation between the sub-error scale vector and the (multivariate) distribution function 152:

[0070] Specifically, the objective is to form a model for statistical extrapolation of a vector-value error scale on a continuous scale (in contrast to the discrete counting of erroneous non-trigger occurrences by prior art, and in an extended form of the computer-implemented method 100 (according to the first embodiment), or as an option). Here, each subsystem can be represented, for example, in a coupled system, by a component in the overall system-TPI, and each vector component can itself form a one-dimensional error scale (subsystem-TPI) for the subsystem to which it belongs. Furthermore, each subsystem-TPI may be defined such that the exceedance of a specific, known (and possibly individual) threshold corresponds to the occurrence of an evaluation error in such a subsystem. In the cases considered herein, evaluation errors occur rarely even in individual subsystems, and therefore evaluation errors in subsystems are often not observable in the collected data (evaluation errors at the subsystem level are fully visible in the data, but erroneous non-triggers are rather unexpected). While evaluation errors are not necessarily expected to occur in subsystems, if they do occur, it is expected that they will not be clearly manifested as false non-triggers within the existing data material. Therefore, similarly, extreme models can be assumed for the statistical descriptions of individual sub-error measures, as in the case of statistical extrapolation of conventional method 100 (according to the first aspect). (Cases may be considered in which several subsystems frequently produce error events, and extrapolation is unnecessary for these error events, which can be done, for example, by integrating multiple subsystems into a single "virtual" subsystem with rare errors, or, for example, by first modeling the remaining subsystems as described below, and then incorporating subsystems with more frequent errors into the overall argument, for example, as additional dimensions in the overall model.) However, this does not yet require that it be possible to infer the probability (or rate of false non-triggers) of false non-triggers in the entire combined system.

[0071] For example, in the above example of OR coupling in the case of AEB, it is rare for both the radar subsystem and the video subsystem to make incorrect non-trigger decisions on their own, and such subsystem errors (i.e., evaluation errors in the subsystems) may be unobservable in the recorded data. Nevertheless, it may be expected that the two subsystems have a certain number of quasi-evaluation errors available for extrapolation, that is, that the subsystems are mistakenly not prepared for triggers. Certainly, as in the conventional methods (according to the first aspect or embodiments thereof), the evaluation error frequency of the radar subsystem or the video subsystem can be estimated by separate one-dimensional extrapolation, but this does not provide information on how often the two subsystems cause errors simultaneously, and it is only in such cases that erroneous non-triggers occur in such a coupled system. Furthermore, in order to enable inference about the rate of erroneous non-triggers in the overall system, the probabilistic relationships between individual sub-error measures (often represented by so-called copulas) are also incorporated into model formation, in which case the focus is placed on the extreme region. This may be, for example, the region in which (at least) one subsystem has a (quasi)evaluation error. Based on a mathematical-statistical set of boundary values, under weak assumptions, the asymptotic dependency structure within random vectors in the extremum can be derived. This allows for the approximate determination of multivariate extremum distributions (similar to the one-dimensional block maxima method in previous method 100), multivariate generalized Pareto distributions (similar to the one-dimensional peak-over-threshold method in previous method 100), or the distributions of such distributions in the so-called Domain of Attraction.What all these distributions have in common is that their probabilistic relationships are represented or approximated by a central model component, which, depending on the source, has various equivalent representations known by terms such as "D-Norm," "stable tail dependence function," "Pickands dependence function," "max-stable copula," "extreme value copula," "generalized Pareto copula," "exponent measure," "spectral measure," "angular measure," and "(multivariate) regular varying function." This type of model can be used to represent the dependency structure of the entire system of vector values—TPI—in the extreme value region.

[0072] Accordingly, the probabilistic overall model for the overall system-TPI includes (similar to the previous method 100) one-dimensional extrapolation models and / or dependency models for individual subsystem-TPIs (a substantial extension compared to the previous method (according to the first aspect or its embodiments)), the dependency models representing the (probabilistic) interactions between the individual subsystems, and are particularly edited to fit the extremum regions of the distribution. Such an overall model may be fitted to an existing dataset (e.g., fitted to subcritical events) and used for extrapolation from subcritical subsets to critical subsets (e.g., in step 161).

[0073] Methods 100, 101 may further include the calculation of another false non-trigger rate (i.e., another estimate of the false non-trigger rate) based on a distribution function of a sub-error scale vector for a critical subset.161 The other false non-trigger rate can be made available, for example, for comparison with other systems, by a similar method as the false non-trigger rate by Method 100 (according to the first embodiment).

[0074] Similar to method 100 (according to the first embodiment), methods 100, 101 may further include a check 171 to determine whether another false non-trigger rate satisfies another fourth predetermined criterion. Methods 100, 101 may further include a release 181 of the system 200 function if the other false non-trigger rate (and / or the upper limit of the confidence interval for the other false non-trigger rate) satisfies another fourth predetermined criterion. Optionally, here, the other fourth predetermined criterion is satisfied if the other false non-trigger rate (and / or the upper limit of the confidence interval for the other false non-trigger rate) is lower than another predetermined acceptable rate. The other predetermined acceptable rate may be a predetermined acceptable rate.

[0075] The overall model can be estimated, for example, stepwise (through its individual components) or as a whole. Even when the overall model is viewed as a unit, the described aspects usually exist at least implicitly. Selectively, the individual distribution functions or copulas of the sub-error scales can also be described by other models that do not necessarily originate from the context of extreme value theory.

[0076] In optional embodiments of methods 100 and 101, a false non-trigger may exist if a portion of the time series of sub-error scale vectors (i.e., the overall system-TPI) satisfies a certain second predetermined criterion / the second predetermined criterion described above. Here, optionally, the second predetermined criterion may arise from the boundary hyperplane of the first Cartesian product. In other words, the second predetermined criterion is, for example, that a portion of the time series of sub-error scale vectors lies within a critical subset (e.g., within the first Cartesian product). A quasi-false non-trigger may exist if a portion of the time series of sub-error scale vectors (i.e., the overall system-TPI) satisfies a certain third predetermined criterion / the third predetermined criterion described above, but does not satisfy the second predetermined criterion. Here, optionally, the third predetermined criterion may arise from the boundary hyperplane of the difference between the first Cartesian product and the second Cartesian product. In other words, the third predetermined criterion is, for example, that a portion of the time series of the sub-error scale vectors falls within a subcritical subset.

[0077] Methods 100 and 101 may further include the calculation of a functional relation to the (multivariate) distribution function of the sub-error scale vectors, based at least on a quasi-false non-trigger. Selectively or additionally, the calculation of the functional relation to the (multivariate) distribution function may be based on at least one false non-trigger (if any). Selectively or additionally, the calculation of the functional relation to the (multivariate) distribution function may be based on at least one evaluation error in the subsystem (if any). Selectively or additionally, the calculation of the functional relation to the (multivariate) distribution function may be based on at least one quasi-evaluation error in the subsystem (if any). In other words, here, for example, it is not necessary to calculate the univariate distribution for the sub-error scale for each subsystem of the system. On the other hand, for each subsystem of the system, the univariate distribution for the sub-error scale may be calculated from the multivariate distribution function (if necessary).

[0078] Furthermore, a system 200 is disclosed that is protected against false non-triggers according to a computer-implemented method 100 (according to a first aspect or embodiment thereof) and / or a computer-implemented method 100, 101 (according to a second aspect). Such protection may be performed, for example, during development within the framework of a release. Selectively or additionally, protection may be performed by diagnostics during system operation.

[0079] Disclosed is at least one computer program configured to implement a computer-implemented method 100 (according to a first aspect or embodiment thereof) or a computer-implemented method 100, 101 (according to a second aspect) that protects the system 200 against false non-triggers. This computer program may exist, for example, in an interpretable form or in a compiled form. It may be loaded into the RAM of a control unit or computer, for execution, for example, as a bit sequence or a byte sequence.

[0080] Furthermore, a computer-readable medium or signals storing and / or containing this computer program is disclosed. This medium may include, for example, one of RAM, ROM, EPROM, ... on which signals are stored.

[0081] Furthermore, a computer system configured to run this computer program is disclosed. This computer system may, in particular, include at least one processor and at least one main memory. Furthermore, the computer system may include memory.

[0082] A computer-implemented method 100 (according to the first aspect or an embodiment thereof) or a computer-implemented method 100, 101 (according to the second aspect) for protecting system 200 against false non-triggers may include one or more user input algorithms, where each of the above user input algorithms is configured to request and / or receive user input from a user (e.g., via a control device or computer user interface). User input can make the method more compatible (as opposed to hardcoded parameters), thereby improving protection. When user input is used, for example, the compatibility of the statistical model with the data may be important: for example, a threshold at which a subcritical event typically begins is usually selected according to a predetermined criterion such that the model assumptions are met as well as possible. A similar situation may arise in methods that do not involve thresholds; here, it is usually determined how much data flows from the extreme value region into the model compatibility. On the other hand, it may be conceivable that such a determination only needs to be made once for a given system. In such cases, it may be conceivable to "hard" implement precisely such a threshold for such a single system. In some cases, it may be possible to repurpose such thresholds for (very) similar systems.

Claims

1. A computer-implemented method (100) for protecting a system (200) against incorrect non-triggers, the method comprising: - receiving (110) a time series of criticality (KRIT), the system (200) including a function that is triggered when the criticality meets a first predetermined criterion; - calculating (120) a time series of a reference (REF), the reference (REF) being a comparison criticality for triggering the function of the system (200); - calculating (130) a time series of an error metric (TPI) based at least on the time series of the criticality (KRIT) and the time series of the reference (REF), wherein when a part of the time series of the error metric (TPI) meets a second predetermined criterion, the non-triggering of the function of the system (200) is evaluated as an incorrect non-trigger; - including identifying (140) at least one quasi-incorrect non-trigger, wherein when a part of the time series of the error metric (TPI) meets a third predetermined criterion but does not meet the second predetermined criterion, the non-triggering of the function of the system (200) is evaluated as a quasi-incorrect non-trigger; A computer-implemented method (100).

2. The method (100) according to claim 1, wherein the time series of the criticality (KRIT), the time series of the reference (REF) and / or the time series of the error metric (TPI) are continuous.

3. The method (100) according to claim 1, wherein the at least one quasi-incorrect non-trigger is used in a diagnosis during operation of the system (200).

4. The method (100) according to claim 1, wherein the system (200) is a driving system, optionally a driving system of a vehicle.

5. The method (100) according to claim 4, wherein the driving system is a driver assistance system (FAS) or an automated driving system, in particular a highly automated driving system (HAF).

6. Furthermore, - including modeling (150) a functional relationship for a distribution function of an error metric based at least on the at least one quasi-incorrect non-trigger; The method (100) according to claim 1, including the calculation (160) of the rate of false non-triggers based on the distribution function of the error measure for a set of error measures (TPI) for which the second predetermined criterion is satisfied.

7. Furthermore, including an inspection (170) of whether the rate of false non-triggers meets a fourth predetermined criterion, optionally including a release (180) of the function of the system (200) when the rate of false non-triggers meets the fourth predetermined criterion, and optionally, when the rate of false non-triggers is less than a predetermined acceptable rate, the fourth predetermined criterion is satisfied, the method (100) according to claim 6.

8. Furthermore, including receiving a data time sheet of the system (200), and optionally, the data time sheet includes at least one time series of data based on measurements using a sensor system (210) of the system (200), the method (100) according to claim 1.

9. Furthermore, including receiving another data time sheet, and optionally, the another data time sheet includes at least one another time series of data based on further measurements using another sensor system (310), including forming an overall data time sheet from the data time sheet and / or the another data time sheet, the calculation (120) of the time series of the reference (REF) is based on the overall data time sheet, and optionally, the reference (REF) is related to the data of the overall data time sheet at the same time point, and / or at least one previous time point, and / or at least one subsequent time point at each time point of the time series, the method (100) according to claim 1.

10. The calculation (130) of the time series of the error measure (TPI) includes calculating the time series of the error measure such that at at least one time point of the time series, the error measure (TPI) is related to various different time points of the time series with respect to the criticality (KRIT) and the reference (REF), the method (100) according to claim 1.

11. Optionally, a computer-implemented method (100, 101) for protecting a system against false non-triggers according to claim 1, comprising: The system includes a function that can be triggered when the criticality meets a first predetermined criterion, the system (200) includes a system of subsystems (220, 221) including at least one subsystem (220), and the method (100, 101) includes: - For each subsystem (220, 221) of the system (200), including receiving (111) a time series of sub-criticalities (KRIT1, KRIT2), The criticality (KRIT) is related to the sub-criticalities (KRIT1, KRIT2) of the subsystems (220, 221) of the system (200), and optionally, the criticality (KRIT) is related to whether or to what extent at least one sub-criticality (KRIT1, KRIT2) meets a first predetermined sub-criterion to which it belongs. A computer-implemented method (100, 101).

12. Furthermore, - For each subsystem (220, 221) of the system (200), including calculating (121) a time series of sub-criteria (REF1, REF2), each of the sub-criteria being a respective comparison sub-criticality for the sub-criticalities (KRIT1, KRIT2) of the subsystem (220, 221), and optionally, at least one time series of the sub-criteria (REF1, REF2) is the time series of the criterion (REF). The method (100, 101) according to claim 11.

13. Furthermore, - For each subsystem (220, 221) of the system (200), including calculating (131) a time series of sub-error metrics (TPI1, TPI2) based on at least the time series of the sub-criticalities (KRIT1, KRIT2) of the subsystem (220, 221) and the time series of the sub-criteria (REF1, REF2) of the subsystem (220, 221), where a time series of a vector of sub-error metrics is generated. The method (100, 101) according to claim 11.

14. In the vector space spanned by the sub-error measure vectors, a critical subset is defined, said critical subset corresponding to a false non-trigger, and optionally, said critical subset is a first Cartesian product, the method (100, 101) according to claim 13.

15. Each element of the sub-critical subset is not an element of the critical subset, but each is close to at least one element of the critical subset according to a predetermined distance criterion, said sub-critical subset being defined in the vector space spanned by the sub-error measure vectors, and optionally, the integration from said critical subset and said sub-critical subset is a second Cartesian product, the method (100, 101) according to claim 14.

16. An evaluation error in the subsystem (220, 221) exists when a part of the time series of said sub-error measures (TPI1, TPI2) satisfies a second predetermined sub-criterion to which it belongs, and optionally, each said second predetermined sub-criterion results from a boundary hyperplane of the first Cartesian product, the method (100, 101) according to claim 13.

17. A quasi-evaluation error in the subsystem (220, 221) exists when a part of the time series of said sub-error measures (TPI1, TPI2) satisfies a third predetermined sub-criterion to which it belongs but does not satisfy the second predetermined sub-criterion to which it belongs, and optionally, each said third predetermined criterion is obtained from a boundary hyperplane of the difference between the first Cartesian product and the second Cartesian product, the method (100, 101) according to claim 16.

18. Furthermore, ・ For each subsystem (220, 221) of said system (200), including the identification (141) of at least one quasi-evaluation error and / or at least one evaluation error, the method (100, 101) according to claim 17.

19. Using said at least one quasi-evaluation error and / or said at least one evaluation error of the subsystem (220, 221) in the diagnosis during the operation of said system (200), particularly in the diagnosis during the operation of said subsystem (220, 221) of said system (200), the method (100, 101) according to claim 18.

20. Furthermore, - For each subsystem (220, 221) of the system (200), it includes modeling (151) of a functional relationship for the distribution function of the sub-error metric (TPI1, TPI2) of the subsystem (220, 221) based on at least one pre-evaluation error and / or at least one evaluation error in the subsystem (220, 221). - Optionally, for each subsystem (220, 221), it includes identification (153) of at least one subsystem (220, 221) that requires optimization based on the distribution function of the sub-error metric, according to the method (100, 101) described in claim 17.

21. Furthermore - It includes calculation (152) of a functional relationship for the distribution function of a vector of sub-error metrics with respect to the distribution function of the subsystems (220, 221) of the system (200), according to the method (100, 101) described in claim 20.

22. Furthermore - It includes calculation (161) of another false non-trigger rate based on the distribution function of a vector of sub-error metrics for the critical subset, according to the method (100, 101) described in claim 21.

23. Furthermore, - It includes inspection (171) of whether another false non-trigger rate meets another fourth predetermined criterion. - Optionally, when the another false non-trigger rate meets the another fourth predetermined criterion, it includes release (181) of the function of the system (200). Optionally, the another fourth predetermined criterion is met when the another false non-trigger rate is less than another predetermined acceptable rate, according to the method (100, 101) described in claim 22.

24. A system (200) protected by a computer-implemented method (100, 101) for protecting the system (200) against false non-triggers, as claimed in any one of claims 1 to 23.

Images