Techniques for validating or validating closed loop test platform

Abstract

A general aspect of the present disclosure relates to a method for training a machine learning module for validating or validating a closed loop test platform for a test system. The method includes providing a plurality of data sets from two or more closed-loop test platforms, wherein the closed-loop test platforms include at least one reference test platform. Each data set contains input data and associated output data of a system under test in a respective closed-loop test platform. The method further includes determining a similarity of a plurality of segments of the input data for a pair of the plurality of data sets. Each pair of test data sets includes a data set of a reference test platform. The method further includes determining a similarity of segments of the output data associated with the segments of the input data; and estimating input and output data of a data set of a closed-loop test platform to be validated or validated and a data set of a plurality of data sets of the at least one reference test platform by using a specific similarity in view of a function of the input and / or output data as the data set of the closed-loop test platform to be validated or validated The similarity aspects train a machine learning module to generate a machine learning module for validating or validating a closed loop test platform.

Description

Background Art

[0001] The use of virtual test benches in the verification and validation of new systems (e.g. systems for automated or assisted driving) has great potential and can contribute to shortening the validation process. In particular, when releasing automated driving functions from level 3 (according to the SAE definition), the use of simulation-enabled test benches is almost indispensable. The term "verification" refers to comparing the behavior and / or configuration of a system with pre-given specifications. Verification is usually performed using test cases derived from requirements. Within the scope of the validation, it is checked whether the system meets all necessary safety goals and has all necessary quality attributes. The validation goes beyond verification. In particular, in the validation, the completeness of the requirements and the validity of the assumptions made are also checked. When "testing" is discussed within the scope of the present disclosure, it is therefore meant to check the quality requirements, safety goals and other requirements for the system.

[0002] For example, by means of virtual confirmation or validation, the scope of an on-site test program can be significantly reduced, or such a test program can even be avoided completely (e.g. test drives with the system to be tested). Additionally or alternatively, the scope and / or quality of the test program can be increased without increasing the use of resources. For example, this can help to reduce potential risks due to rare events. During a test drive, it may be difficult to test the reaction of the system to be tested to rare events. For example, a precautionary number of test kilometers may be required in order to observe a large number of specific rare events in a test drive. In a virtual confirmation or validation, rare events can be easily generated by appropriately selected scenarios.

[0003] However, virtual confirmation or verification also brings new problems. A basic challenge is to emulate the environment of the system to be tested or also emulate the system to be tested itself in a virtual environment (for example, system simulation system input signals for a vehicle system). Here, within the scope of environmental modeling or selection of environmental data, input signals from the real environment of the system are always replaced by simulated and / or synthetic input signals. Synthetic input signals can also be generated based on real data (for example, fault injection based on real fault images). The question here is whether the associated features or attributes of the real environment are changed or not shown. The lack of associated features and their changes may result in: the system to be tested is not fully tested within the scope of virtual confirmation or verification. This may have a decisive impact on the function and even operational safety of the system tested in this way.

[0004] For the "closed-loop test platform" for virtual confirmation or verification, this danger is particularly a problem. In the closed-loop test platform, input data (also referred to as "input signal" in the present disclosure) is fed to the system to be tested. In addition, the output data of the system to be tested (also referred to as "output signal" in the present invention) is also fed back to return the action of the system to be tested to the test environment. On this basis, the input signal, etc. can be adapted. With the help of the closed-loop test platform, the behavior of the system to be tested can therefore be tested when the system reacts to its environment (and a feedback loop occurs). For the closed-loop test platform, the above-mentioned problem is even more critical in some cases, because different models must be used to describe complex systems in many cases, for example, in order to form the input signal of the system to be tested (and process its output signal). With the increase of the complexity of the closed-loop test platform, the risk that the system to be tested is not fully tested within the scope of virtual confirmation or verification will usually increase (for example, because some events in the environment are not simulated or not simulated correctly). For example, in a closed-loop test platform that simulates and emulates the system environment, the occurrence of the expected system reaction in a specific case does not always mean that an effective test has been performed on the system relative to the specific case. It may thus happen that a simulated emulation of an environment may contain different triggers for the intended reactions than the real environment.

[0005] Therefore, in order to reduce the risk of insufficient virtual confirmation or verification (eg, for the operational safety of the system to be tested) and the associated consequences, measures for confirming or verifying the closed-loop test platform are needed. The technology disclosed in the present invention undertakes this task. Summary of the invention

[0006] A first general aspect of the present disclosure relates to a method for training a machine learning module for confirming or validating a closed-loop test platform, the closed-loop test platform being used to confirm or verify a system. The method includes: providing multiple data sets from two or more closed-loop test platforms, wherein the multiple closed-loop test platforms include at least one reference test platform. Each data set contains input data and associated output data of the system being tested in the corresponding closed-loop test platform. The method also includes: determining the similarity of multiple segments of input data for a pair of multiple data sets. Each pair of data sets contains a data set of a reference test platform. The method also includes: determining the similarity of segments of output data associated with segments of input data, and estimating the similarity of input data and output data of the data set of the closed-loop test platform to be confirmed or verified and the data sets in the multiple data sets of the at least one reference test platform by using a specific similarity in view of the input data and / or output data of the data set of the closed-loop test platform to be confirmed or verified as a function of the data set, so as to train the machine learning module in terms of similarity, so as to generate a machine learning module for confirming or verifying the closed-loop test platform.

[0007] A second general aspect of the present disclosure relates to a method for confirming or validating a closed-loop test platform for a test system. The method includes: receiving a data set for a closed-loop test platform for a test system, feeding at least a portion of the data set of the closed-loop test platform into a machine learning module for confirming or validating the closed-loop test platform. The machine learning module is trained in terms of estimating the similarity of the input data and output data of the data set of the closed-loop test platform to be confirmed or verified and the data set of at least one reference test platform in view of being a function of the input data and / or output data of the data set of the closed-loop test platform to be confirmed or verified. The method also includes outputting one or more estimates of the similarity of the input data and output data of the data set of the closed-loop test platform to be confirmed or verified.

[0008] A third general aspect of the present disclosure relates to a computing unit designed to perform a method according to the first or second general aspect.

[0009] In some cases, the techniques according to the first to third general aspects of the present disclosure may have one or more of the following advantages.

[0010] First, through the technology of the present disclosure, in some cases, even if the system to be tested and / or its environment is complex and, for example, must be simulated and represented by multiple (possibly a large number of) (sub) models, confirmation or verification at the level of the (overall) system can be performed. In some prior art methods, different (sub) models are confirmed or verified separately. This may be very expensive. In addition, in some cases, the confirmation or verification can only be performed in an open loop, which, as mentioned above, may not allow reliable statements of the system behavior in the field (i.e., in a closed loop). The effectiveness of the closed-loop test platform is evaluated using the similarity of the input data and output data of the (overall) system, so that it is possible to, for example, identify whether the specific behavior of the system is fully described and thus tested in a specific operating scenario. The data set used by the reference test platform can represent sufficient confirmation or verification. Therefore, the machine learning module's estimate of the similarity of the input data and output data with the data set of the reference test platform can, in some cases, enable reliable statements to be made about the following aspects: whether the behavior of the system fully tests the closed-loop test platform to be confirmed or verified.

[0011] In particular, it can be estimated whether the closed-loop test bench to be confirmed or verified tests the associated input data and whether this leads to the expected output data of the system to be tested. In some prior art, it is only checked whether, for example, certain critical operating conditions occur in the output data of the system to be tested. However, it may then happen that, for example, in a closed-loop test bench simulating the environment of the system to be tested, input data that are completely different from those in the field lead to critical operating conditions in the output data of the system to be tested. This may also mean that the closed-loop test bench cannot fully confirm or verify certain behaviors of the system to be tested.

[0012] Secondly, even if the data from different test platforms are different (for example, not exactly the same operating scenarios are simulated / performed or not all associated conditions are reconstructed in the simulation), the technology of the present disclosure can also be applied in some cases. For example, data from continuous operation (such as data recorded in the system on site) can also be used. In some cases, this can reduce the cost of confirming or verifying the test platform because a wider range of data can be used for confirmation or verification (for example, previously collected data or even data collected outside the scope of the test can continue to be used after the test protocol is changed). In some cases, test platforms from different manufacturers can also be compared (more easily). The use of a machine learning module trained in view of estimating the similarity of the input data and output data of the data set of the closed-loop test platform to be confirmed or verified and multiple data sets of the reference test platform enables reliable confirmation or verification in the above case.

[0013] Third, in some cases, the techniques of the present disclosure can reduce the cost of performing validation or verification (e.g., simulation). Input and output data are compared segment by segment, with only a portion of the input and output data being considered depending on the operational scenario being examined. In addition, in some cases, these segments can be compactly represented using techniques for reducing their dimensionality. Both can make the training and application of machine learning modules more efficient (e.g., in terms of storage space and / or computational cost).

[0014] Fourth, the disclosed techniques can be used to identify the following operating scenarios or parameter ranges for which the closed-loop test platform has not yet been fully confirmed or validated. If data appears in the closed-loop test platform data set for which no suitable data of the reference test platform can be assigned, this indicates that further data must be collected for the closed-loop test platform or the sub-model contained therein for confirmation.

[0015] Some of the terms used in this disclosure are as follows:

[0016] A "closed-loop test platform" (roughly translated in German as "Testplattform mit Rückkopplung (test platform with feedback)") is any system that forms a test environment for a system to be tested, wherein the system to be tested is fed with input data and in addition the output data of the system to be tested is fed back to the test environment (and can influence it). In other words, a "closed-loop test platform" forms a feedback loop (and is therefore suitable for confirming or verifying systems or system functions in which feedback plays a role in the behavior of the system - this applies to a large number of systems). In a "closed-loop test platform", a distinction is made between the system to be tested (in the present disclosure, unless explicitly described otherwise, it is part of the closed-loop test platform) and the environment of the system to be tested. In some cases, the closed-loop test platform can simulate not only the system to be confirmed or verified but also its environment. In other examples, the real system to be confirmed or verified can be inserted into the closed-loop test platform (in contrast to the term "simulation", the term "real" is used here - the real system is also used in this form later, the simulated system emulates the real system; the real environment is the environment in the field or in the application). The system to be tested can take any conceivable form. For example, the system to be tested can be a hardware system, a software system, or a system with hardware and software components. In the present disclosure, the system to be tested on site or on a test bench is also a closed-loop test platform (e.g., a test vehicle containing the system to be tested-the test environment here can be the real environment of the system to be tested in the application situation).

[0017] "Input data" refers to all data transmitted to the system to be tested in a closed-loop test platform (e.g., used to emulate the environment of the system to be tested or from the real environment of the system to be tested). The input data may include data simulated or synthesized using a model or otherwise, and / or real data (which may be recorded / generated in real time or retrieved from a database and / or may be modified by one or more processing steps). For example, the input data may include real data manipulated in some way. In some cases, the input data may include time records. However, the input signal may also have any other conceivable format.

[0018] "Output data" refers to all data output from the system under test in the closed-loop test platform, or all data detected or derived about the system under test (e.g., by an agent monitoring the behavior of the system under test). The output data can be at least partially fed back into the environment of the closed-loop test platform and / or provided for monitoring purposes.

[0019] The term "environment" or "test environment" refers to the context in which the system to be tested is embedded (i.e., in the informed application situation). The environment can generate input data for the system to be tested. In addition, the output data of the system to be tested may (at least partially) react to the environment. The environment can contain objects of other systems (e.g., other systems to which the system to be tested is connected in use) and / or the environment in which the system to be tested runs in the informed application situation. The environment (or the system to be tested) defines one or more interfaces through which data is transferred from the environment to the system (or vice versa). In some closed-loop test platforms, the environment is simulated. Therefore, the environment can be emulated in such a way that the data transmitted to the system in the application situation is emulated so that the functionality of the system can be tested.

[0020] A "system" can be any device or group of devices designed for a specific purpose, such as one or more regulation, control, communication and / or monitoring tasks. A system can be a component of a higher-level (further) system. A system can be software. In other examples, a system can be a hardware component (in some examples, with associated software). For example, a system can be an embedded system (German: "eingebettetes System"). In some examples, a system can be a means of transport or a module for a means of transport (e.g. for maneuvers). In other examples, a system can be a robot or a module for a robot. In yet other examples, the system can be an industrial facility or an industrial machine or a module for an industrial facility or an industrial machine. In yet other examples, the system can be a tool or a module for a tool.

[0021] According to the present disclosure, a "machine learning module" is any hardware and / or software module that can be trained / trained using machine learning for a task. To this end, the machine learning module has parameters that are changed within the scope of training using a training data set so that the desired output data is generated in response to specific input data. In the best case, the machine learning module trained in this way also processes unknown input data (i.e., those not included in the training data set) in the desired manner. The machine learning module may include one or more artificial neural networks, but is not limited to this topology. The machine learning module can be implemented in any suitable manner. For example, the machine learning module can be a software module (i.e., the functionality is defined in software that can be executed on a non-specific computing unit). In other examples, the machine learning module can be a hardware module (i.e., the functionality of the machine learning module is defined in hardware). In still other examples, the functionality of the machine learning module can be defined in both software and hardware.

[0022] In the present disclosure, a "vehicle" is any device for transporting passengers (including the driver) and / or cargo. A vehicle may be at least partially autonomous or assisted. A vehicle may be a motor vehicle, but may also be any other land vehicle. Floating, submersible and / or flying devices may also be vehicles according to the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of a method for validating or verifying a machine learning module of a closed-loop test platform for testing a system according to the present disclosure is shown.

[0024] Figure 2 The training method of the present disclosure is schematically illustrated.

[0025] Figure 3 An exemplary method for training a machine learning module of the present disclosure is schematically illustrated.

[0026] Figure 4 A closed-loop test platform is schematically shown.

[0027] Figure 5 A flow chart of a method for validating or verifying a closed-loop test platform for a test system according to the present disclosure is shown.

[0028] Figure 6 An exemplary method for validating or verifying a closed-loop test platform is schematically illustrated.

[0029] Figure 7 An exemplary output of the disclosed method for validating or verifying a closed-loop test platform is shown.

[0030] Figure 8 Another exemplary output of the disclosed method for validating or verifying a closed-loop test platform is shown. DETAILED DESCRIPTION

[0031] First, refer to Figures 1 to 4 , explaining techniques for training a machine learning module for confirming or validating a closed-loop test platform for a test system according to the present disclosure. Figures 5 to 8 Techniques for validating or verifying a closed-loop test platform for testing a system (ie, application of a trained machine learning module) in accordance with the present disclosure are discussed.

[0032] Figure 1 A flow chart of a method 100 for training a machine learning module for validating or verifying a closed-loop test platform for testing a system according to the present disclosure is shown. Figure 2 The method for training of the present disclosure is schematically illustrated.

[0033] A method for training a machine learning module for confirming or validating a closed-loop test platform for confirming or validating a system includes providing 101 a plurality of data sets 11-14 from two or more closed-loop test platforms 20, 22. The plurality of closed-loop test platforms include at least one reference test platform 22 (in the present disclosure, the term "reference test platform" is used as an abbreviation of the term "reference closed-loop test platform"). The reference test platform 22 or its data sets 13, 14 are used as reference points for similarity estimation within the scope of training. In other words, the machine learning module is trained / is trained in view of the following aspects: estimating how similar the input and output data of the closed-loop test platform to be confirmed or verified are to the input and output data of (multiple) reference test platforms 22. In some examples, (multiple) reference test platforms may be closed-loop test platforms that have been confirmed or verified or for which the closed-loop test platforms are confident for other reasons that they have adequately tested the system to be tested.

[0034] Each data set 11-14 contains the corresponding closed-loop test platform 20, 22 ( Figure 1 and Figure 2 The providing may include any operation for making the plurality of data sets 11-14 available for subsequent processing.

[0035] Data sets 11-14 may include any data generated when the closed-loop test platform is run using the system to be tested. For example, data sets 11-14 may include data generated within a specific time interval when the closed-loop test platform is run using the system to be tested. As previously mentioned, the system on site is also a closed-loop test platform according to the present disclosure. Therefore, data sets 11-14 may also include data generated during the operation of the system in the application and / or on site (even if in this case it is not intended to test the system in a more narrow sense).

[0036] Data set 11-14 can comprise the data of one or more test runs for the closed-loop test platform with the system to be tested. In some examples, the test run can be related to the specific operation scenario of the system to be tested and / or the specific operation situation (such as running state) of the system to be tested (such as the test run can emulate this operation scenario or this operation situation). In this case, input data and output data can be the data (more see below) generated during the specific operation scenario and / or the specific operation situation of the relevant test run. However, in other examples, it is also possible to record data sets when not emulating the specific operation scenario of the system to be tested and / or the specific operation situation of the system to be tested. For example, the data of data set 11-14 may be generated from the continuous operation of the closed-loop test platform with the system to be tested.

[0037] In the present disclosure, an operation scenario or an operation situation may be any configuration that occurs during the operation of the system to be tested or any state that occurs during the operation of the system to be tested (e.g., characterized by: specific values ​​of operating parameters of the system to be tested or specific values ​​of parameters of its environment). For example, an operation scenario or an operation situation may involve specific environmental conditions or environmental configurations of the system (e.g., operation in specific environmental conditions or environmental configurations). Additionally or alternatively, an operation scenario or an operation situation may include a violation of a safety goal. Further additionally or alternatively, an operation scenario or an operation situation may involve a critical state of the system to be tested or an error state of the system to be tested.

[0038] The input data may include all data that is delivered to the system to be tested during the operation of the corresponding closed-loop test platform. Depending on the closed-loop test platform and / or the system to be tested and / or the type of operation (e.g., a specific test operation or a continuous operation), the input data may be constructed and / or structured differently. In some examples, the input data includes those signals (e.g., sensor signals or output signals from other systems) obtained from the environment by the system to be tested during operation (e.g., during a specific test operation or a continuous operation), wherein the other systems are upstream of the system to be tested and / or communicate with the system to be tested. These signals are also referred to as input signals in the present disclosure. In some examples, the input data may at least partially include one or more time records (e.g., Figure 2 The time record may describe one or more parameters or quantities that vary over time (i.e., in one or more dimensions). The time record may also describe the time course of a complex data structure (e.g., a list of objects or an image).

[0039] The output data may include all data output by the system to be tested within the scope of operation of the corresponding closed-loop test platform (e.g., during a specific test run or continuous run). In some examples, the output data may include data fed back to the closed-loop test platform (e.g., within the scope of the test run or continuous run). Additionally or alternatively, the output data may include data recorded for the purpose of monitoring the system to be tested (the data, for example, characterizes the behavior of the system to be tested). Output data is also referred to as output signal in the present disclosure. Like input data, output data may also be constructed and / or formatted differently according to the closed-loop test platform and / or the system to be tested and / or the type of operation (e.g., a specific test run or continuous run). In some examples, the output data may include, at least in part, one or more time records. These time records may describe one or more parameters or quantities that vary over time (i.e., in one or more dimensions).

[0040] Output data may be associated with input data if the output data is generated (e.g., substantially simultaneously or with an upper time offset) in the same run (e.g., during a particular test run or a continuous run) that is time-correlated with the input data. In other words, the associated output data is data recorded when monitoring the system or data output by the system when the system processes the particular input data (e.g., within the context of a test run or a continuous run).

[0041] In some examples, both the input data and the output data are processed (e.g., filtered or further processed into another format). Thus, the input and / or output data need not (but may) be processed in the form in which it is generated in the closed-loop test platform. In other examples, the input and / or output data are derived from data generated in the closed-loop test platform. In some examples, the input data and / or output data include all data generated in the closed-loop test platform (e.g., within the scope of a test run or a continuous run). In other examples, the input data and / or output data are only a selection of data generated in the closed-loop test platform (e.g., within the scope of a test run or a continuous run).

[0042] In some examples, the method may also include generating one or more of the plurality of data sets 11-14 (eg, by performing corresponding test runs using the closed-loop test platforms 20, 22). Alternatively or additionally, one or more of the plurality of data sets 11-14 may already exist.

[0043] The method further comprises: determining 102 similarities of segments 31-38 of input data for pairs of a plurality of data sets 11-14. Each pair of test data sets comprises data sets 13, 14 of a reference test platform 22 (and correspondingly a second data set 11, 12 of another closed-loop test platform 20). A number of aspects of the manner in which similarities are determined (e.g. similarity metrics that may be used) are described below. In other words, for example, a similarity is determined for a first segment 31 of a first data set 11 to a second segment 35 of a second data set 13 (of a reference test platform 22). The process may then be repeated for additional (first) and additional (second) segments (e.g. more than 100, more than 1000 or more than 100000 segments). Additionally or alternatively, segments from different pairs of data sets may be compared.

[0044] A segment is here a part of the input data and / or a part of a data element of the input data (i.e., less than the complete input data and / or less than a complete data element of the input data). If a data element of the input data (or the entire input data) is a time record, a segment can in turn be a time record (i.e., a time segment of the time record) that is shorter than the duration of the entire time record. In some examples, the length of one of the time segments can be set within a predetermined (fixed or variable) duration (which can depend on the type of system to be tested and / or operation). The predetermined duration can be shorter than 2 minutes (e.g., shorter than 30 seconds or shorter than 15 seconds). Additionally or alternatively, the predetermined duration can be longer than 2 seconds (e.g., longer than 5 seconds). These durations can be of particular interest for systems used in means of transport.

[0045] In other examples, a segment may mark a portion of the input data and / or a portion of one of the input data (eg, a segment of an image of a series of image data or a segment of a list of objects) according to other criteria than time.

[0046] In some examples, the method includes: decomposing the input data or one of the input data into a plurality of segments 31-38 (eg Figure 2 The segments may be disjoint (ie, have no common data points) or overlapping. The above determination of similar segments may be performed for each of the segments.

[0047] In some examples, the method may include determining similar segments 31-38 of input data for pairs of multiple data sets 11-14 (where each pair of test data sets includes data sets 13, 14 of a reference test platform 22). In some examples, similarity is determined only for the determined similar segments. In other words, for example, a similar second segment 31 of a second data set 11, 12 is determined (and similarity is determined if necessary) for a first segment 35 of a first data set (e.g., test data sets 13, 14 of a reference test platform 22). The process may then be repeated for additional (first) and additional (second) segments. If the multiple data sets include more than two data sets, determination of similar segments (and determination of similarity if necessary) may be performed for multiple (e.g., all) pairs of the multiple data sets. For a specific (first) segment, multiple similar (second) segments may also be determined (and corresponding similarities determined). Of course, it is also possible that no other similar segments may be found for a specific segment.

[0048] In some examples, the technology of the present disclosure may also include: finding pairs of segments of input data for training a machine learning module for confirming or validating a closed-loop test platform (i.e., pairs for determining 102 the similarity of the input data). In other words, segments of input data that are paired with each other may first be searched in multiple data sets, and then their similarity may be determined as described. The search may include the above-mentioned decomposition of the input data and / or the determination of similar segments.

[0049] Finding pairs of segments of input data may include: comparing multiple different possible combinations of data pairs (i.e., two data sets) from multiple data sets and selecting one or more of the different combinations of data pairs according to predetermined criteria. These possible combinations can be generated by changing different parameters. For example, the lengths of these segments can be selected differently. Additionally or alternatively, the segments of a pair can be offset differently from each other in time. In one example, these segments of a pair of data sets can be folded (optionally additionally by using variable lengths of these segments). In each case, similarities can be determined separately for the possible combinations (e.g., using the techniques described in the present disclosure), and the combination with the greatest similarity and / or a similarity exceeding a specific metric can be selected.

[0050] Additionally or alternatively, in some examples, in order to select these pairs, these segments of the input data can be decomposed into smaller and smaller units (e.g., time segments). First, for a first variable (e.g., a first operating parameter), segments that meet specific criteria (e.g., approximate stability or similar criteria) can be identified from the input data. For the relevant segment, a second variable (e.g., a second operating parameter) is considered and the segment is decomposed into smaller parts, in which the specific criteria for the second parameter are met respectively. In some examples, the process can be repeated for one or more other parameters. Thus, there are multiple segments that are shorter (than the original segment), in which the specific criteria are met for all processed parameters (first parameter, second parameter, etc.). These segments can be determined not only in the input data of the data set of at least one reference test platform but also in other data sets, and are summarized into pairs, and the similarity of these pairs is determined as described in the present disclosure.

[0051] The method further comprises determining 103 that a segment of the input data 31-38 (ie a segment for which a similarity is determined and / or has been determined) is associated with Figure 250 of the segments of the output data of the input data 31-38 (not shown). For each pair of segments of the input data 31-38, the similarity of the associated segments of the output data is determined. The following describes the similarity metrics that can be used (different similarity metrics can be used for comparing segments of the input data and comparing segments of the output data). The segments of the output data can be designed like the segments of the input data. For example, if one of the data of the output data (or the entire output data) is a time record, the segment can be a time record that is shorter than the duration of the entire time record (i.e., a time segment of the time record). In some examples, the length of one of the time periods can be set within a predetermined (fixed or variable) duration (which can depend on the type of system to be tested and / or test run). As for the segments of the input data, the predetermined duration can be shorter than 2 minutes (e.g., shorter than 30 seconds or shorter than 15 seconds). Additionally or alternatively, the predetermined duration can be longer than 2 seconds (e.g., longer than 5 seconds). In some examples, the duration of the segments of the output data can be different from the duration of the segments of the input data.

[0052] In some examples, sections of the output data and / or input data are selected to include data related to one or more predetermined operating conditions and / or operating scenarios of a system in a corresponding closed-loop test platform.

[0053] For example, the one or more predetermined operating conditions and / or operating scenarios may be critical operating conditions and / or operating scenarios of the system (e.g., critical to the functionality or operational safety of the system). The criticality may be determined according to a predetermined criticality criterion. The predetermined criticality criterion may be designed differently depending on the type of system to be tested. In some examples, the criticality criterion may be that the operating condition of the system or the state of its environment has one or more predetermined characteristics (e.g., a specific error has occurred, a specific event (e.g., a specific anomaly) has occurred in the system or its environment, and the input signal and / or output signal of the system satisfies one or more predetermined conditions).

[0054] For example, in the case of a system for a means of transport, the critical operating conditions and / or operating scenarios of the system may be critical driving conditions and / or driving scenarios. These driving conditions and / or driving scenarios may include one or more of the following: driving under special environmental conditions (e.g., light conditions or weather conditions), critical objects or events in the system's environment (missing or incorrect lane markings, obscured objects, sharp turns), or specific driving maneuvers (e.g., driving above a certain speed). In some examples, the critical driving conditions and / or driving scenarios may include one or more of the following critical driving conditions and / or driving scenarios, which are described as "triggering events" in the standard ISO / PAS 21448:2019 (English title "Road vehicles—Safety of the intended functionality").

[0055] The method further comprises: training 104 a machine learning module in terms of estimating the similarity between the input data and output data of the data set of the closed-loop test platform to be confirmed or verified and the data sets 13, 14 in the multiple data sets of the at least one reference test platform 22 by using a specific similarity 50 in view of the input data and / or output data of the data set of the closed-loop test platform to be confirmed or verified as a function of the input data and / or output data of the data set of the closed-loop test platform to be confirmed or verified, so as to generate a machine learning module for the closed-loop test platform for confirmation or verification. In other words, the goal of the training is to enable the machine learning module to estimate: the degree of similarity between the test data set (and thus the closed-loop test platform that generates the test data set) and the one or more data sets 13, 14 used for the training of the at least one reference test platform 22.

[0056] The estimation result can be used to confirm or verify the closed-loop test platform (and / or the system contained therein). For example, if the data sets 13, 14 of the at least one reference test platform 22 fully confirm or verify the operating behavior of the system to be tested, it can be estimated whether the closed-loop test platform to be confirmed or verified shows similar behavior related to the input data and output data as the data sets 13, 14. If this is the case, it can be assumed that the closed-loop test platform to be confirmed or verified also fully confirms or verifies the operating behavior of the system to be tested.

[0057] Reference now Figure 3 Explain other aspects of training the machine learning module. Figure 3 An exemplary method for training the machine learning module 62 of the present disclosure is schematically illustrated.

[0058] The machine learning module 62 may obtain at least a portion of the output data and / or the input data 61 for the following data sets as input parameters, which data sets are not generated using the at least one reference platform ( Figure 3 In the (first) closed-loop test platform 20), and generates similarity estimates of input data and output data as output parameters. Within the scope of training, the determined similarity 50 is used as a reference truth value (i.e., the expected output parameter of the machine learning module). Here, the similarity determined using a specific similarity metric can be used or processed directly. For example, the similarity can be normalized and / or scaled. In other examples, the similarity can be divided into two or more classes (e.g., binarized into "similar" or "dissimilar" classes). Therefore, the machine learning module 62 can be trained to estimate the similarity 50 of the input data and the output data based on at least a portion of the output data and / or the input data 61. Any suitable training method for the machine learning module can be used to perform training. For example, the loss function (i.e., the difference between the true similarity and the output of the machine learning module 62) can be gradually reduced by adapting the parameters of the machine learning module (e.g., by using a backpropagation algorithm and / or a gradient descent algorithm).

[0059] In some examples, the machine learning module 62 obtains only a portion of the output data and / or input data as input parameters. In this case, the machine learning module 62 can therefore be trained (and eventually): estimate the similarity of the input data and output data based on a smaller and / or different portion of the output data and / or input data of the data set of the system to be tested than the portion used for determining the similarity within the scope of the training of the machine learning module 62 (e.g., further selection of the input data and output data within the scope of the training of the machine learning module 62 is used to determine the similarity).

[0060] For example, the machine learning module 62 may be trained (and as a result is then trained) to estimate the similarity between input data and output data based on only a portion of the output data of the system under test in the closed-loop test platform. In other examples, the machine learning module 62 may be trained (and as a result is then trained) to estimate the similarity between input data and output data based on only a portion of the input data of the system under test in the closed-loop test platform.

[0061] Additionally or alternatively, in some examples, the portion of the input data and / or output data 61 may include the following information, which represents: an operating scenario of a system included in a closed-loop test platform detected or simulated using a corresponding closed-loop test platform 20, 22 or an operating condition of a system included in a closed-loop test platform detected or simulated using a corresponding closed-loop test platform 20, 22. For example, a segment of the input data may represent an operating scenario or an operating condition (e.g., a time record during an operating scenario or an operating condition). The segment may be an input parameter of a machine learning module 62 in an application case (for which the machine learning module 62 estimates the similarity 50). Additionally or alternatively, a segment of the output data may represent an operating scenario or an operating condition (e.g., a time record during an operating scenario or an operating condition). The segment may also be an input parameter of a machine learning module 62 in an application case (for which the machine learning module 62 estimates the similarity 50).

[0062] The machine learning module 62 may have any suitable structure so that it can be trained to estimate the similarity of input data and output data. In some cases, the machine learning module 62 may include an artificial neural network (e.g., a deep neural network and / or a convolutional neural network).

[0063] The machine learning module 62 can be designed to estimate a first similarity for input data and a second similarity for output data (e.g., the similarity of the input data and associated output data of the closed-loop test platform to be confirmed or verified and the corresponding data of at least one reference test platform). The similarity can be classified by the machine learning module 62 (i.e., the machine learning module 62 outputs the similarity of the data set of the closed-loop test platform to be confirmed or verified in two or more classes, for example, binarized into "similar" or "dissimilar" classes). The machine learning module can also output probabilistic classification in multiple similarity categories. In these examples, the machine learning module 62 is a classifier. In other examples, the machine learning module 62 can output characteristic parameters or other characteristic parameters for similarity (e.g., from 0% to 100%). In these examples, the machine learning module 62 is a regressor. In each case, the output of the machine learning module 62 can be used to identify whether the data set for the closed-loop test platform adequately tests the system to be tested (i.e., in a manner similar to at least one reference test platform).

[0064] As described above, similarities are determined not only for segments of input data but also for segments of output data. In some examples, these determinations may be performed in one of the following ways.

[0065] In some examples, determining the similarity of a segment of input data may include determining the distance between a first segment of input data of a first data set of a corresponding pair and a second segment of input data of a second data set of the pair. In other words, an appropriate distance metric is used to calculate the distance between a first segment of input data of a first data set of a corresponding pair and a second segment of input data of a second data set of the pair. The greater the distance, the smaller the similarity (and vice versa). The distance metric may be selected based on the type of input data being compared. Exemplary distance metrics include Euclidean distance, Manhattan distance, or distance of a higher vector norm (for one-dimensional or multi-dimensional segments of input data spanning Euclidean space). Other examples for distance metrics are Hamming distance or Levenshtein distance (e.g., for strings). In other examples, the distance metric may include distance metrics for multiple distributions (e.g., distance metrics for probability distributions, such as bulldozer moving distances, i.e., EMD or Wassertstein distances—e.g., for point clouds detected by radar or LIDAR).

[0066] In some examples, these segments of the input data can be directly compared with an appropriate distance metric. In other examples, these segments can undergo one or more preprocessing stages. For example, determining similarity can include: compressing each segment and comparing the compressed segments of the input data. In some examples, compressing each segment can include: generating data characterizing the corresponding segment with a lower dimension than the original segment. Additionally or alternatively, compressing each segment can include: executing one or more methods for reducing the dimensionality of each segment (e.g., selecting one or more most important principal components in principal component analysis, omitting the least significant bit, etc.). In some examples, a segment of a scalar or vector time record can be represented by a single scalar or vector data (i.e., omitting the time dimension). Additionally or alternatively, vector data can be represented by lower dimensional vector data or scalar data (in the present disclosure, the vector has at least two entries). Compression and / or dimensionality reduction may be helpful because the input data in many closed-loop test platforms may be high-dimensional.

[0067] Additionally or alternatively, the segments of the input data may be normalized or otherwise unified. In this way, segments of input data having different formats (e.g., lengths or dimensions) and / or different value ranges may be compared. In some examples, the above-described methods for compression and / or dimensionality reduction may also be used to unify these segments of the input data. For example, segments (of different lengths or dimensions) may be represented by scalars, respectively. These techniques may improve the comparability of segments of input data from different closed-loop test platforms.

[0068] In the preceding paragraphs, processing aspects (particularly with respect to determining similarities) have been described primarily with respect to segments of input data. However, these techniques may also or alternatively be applied to these segments of output data. For example, determining the similarity of a segment of output data may include determining a distance between a first segment of output data of a first data set of a corresponding pair and a second segment of a second data set of the pair. In other words, a distance between a first segment of output data of a first data set of a corresponding pair and a second segment of input data of a second data set of the pair is calculated using an appropriate distance metric. A greater distance means a lesser similarity (and vice versa). The other techniques described above may also be applied to output data.

[0069] The closed-loop test platform whose data is used to train the machine learning module 62 takes different forms. Figure 4 A closed loop test platform 75 is schematically shown.

[0070] As previously mentioned, the closed-loop test platform 75 may include a system to be tested 71 and an environment 70 of the system to be tested. The environment 70 delivers input data 73 to the system to be tested 71. The input data 73 may be generated in simulation or synthesized in another way (e.g., by emulating the environment 70 through simulation). In other examples, the input data 73 may be real data from the environment of the system to be tested 71 (e.g., in a test bench or during the operation of the system to be tested 71 on site-in these examples, the environment 70 may at least partially include the real environment of the system or emulate it through hardware). In yet other examples, the input data 73 may include real data changed by one or more processing steps (e.g., error injection based on a real error image). The output data 74 of the system to be tested 71 may be fed back to the environment 70 (and new input data 73 is generated there), etc.

[0071] In some examples, the closed-loop test platform 75 can be a software-in-the-loop test platform. In this case, the system to be tested 71 is software (i.e., the system is composed of software). In the application, the software can be part of a higher-level system here (e.g., a component that contains the software and in which the software performs a predetermined function). In the software-in-the-loop test platform, input data 73 and output data 74 are conveyed or tapped via a software interface. The software is executed during the test run or in continuous operation (on a suitable computing unit) to reconstruct the operation in the application. The environment 71 can be reconstructed in simulation and / or by a real system.

[0072] In some examples, the closed-loop test platform 75 can be a model-in-the-loop test platform. In this case, the system to be tested 71 is a model of another system (e.g., a model of a system to be used in an application). In these examples, the technical system can also be software (i.e., software that models another system). In these model-in-the-loop test platforms, input data 73 and output data 74 (wherein the input data is adapted to the corresponding model) are conveyed or tapped via a software interface. The model is executed during the test run or in continuous operation (on a suitable computing unit) to reconstruct the operation in the application. The environment 71 can be reconstructed in simulation and / or by a real system.

[0073] In other examples, the closed-loop test platform 75 can be a hardware-in-the-loop platform. In this case, the system to be tested 71 is a hardware component (which itself may also contain software). For example, the system to be tested can be a hardware component specific to an application. In the hardware-in-the-loop test platform, input data 73 and output data 74 can be delivered / queried via existing interfaces of the hardware component. The environment 71 can be reconstructed in simulation and / or reconstructed by a real system.

[0074] In some other examples, the closed-loop test platform 75 may include a system to be tested 71 on site or on a test bench. In this case, the system to be tested 71 may be a hardware component (which may also include software) or a software component. For example, the system to be tested may be a specific hardware component or software component for an application. Similarly, input data 73 and output data 74 may be conveyed / queried via an existing interface or software interface of the hardware component. In these examples, the environment 71 is the real environment of the system on site or on a test bench.

[0075] As described above, the following multiple data sets are used to train the machine learning module 62, wherein the data sets are respectively generated using different closed-loop test platforms 75. The above closed-loop test platforms can be combined in any manner. In particular, any type of closed-loop test platform 75 can be a reference test platform.

[0076] In some examples, the at least one reference test platform may include a first type of closed-loop test platform 75 , and the other(s) closed-loop test platform may include a second type of closed-loop test platform 75 .

[0077] Additionally or alternatively, the at least one reference test platform may include a closed-loop test platform 75, which includes the system to be tested 71 on site or on a test bench. In other words, the reference test platform may contain a real system to be tested. This can ensure the quality of the data set from the reference test platform (because, for example, problems can be reduced by simulating and emulating the system and / or its environment). Alternatively or additionally, the at least one reference test platform may also include another of the above-mentioned closed-loop test platforms 75 (for example, a closed-loop test platform that simulates and recreates the environment and / or the system to be tested at a greater cost than other closed-loop test platforms to be confirmed or verified and / or a closed-loop test platform whose behavior has been confirmed or verified).

[0078] As described above, the technology of the present disclosure can be used for any system tested in a closed-loop test platform. In some examples, the system to be tested is a vehicle or a module for a vehicle (e.g., for a motor vehicle). The system can be configured to be installed in the vehicle itself and / or communicate with the vehicle (e.g., in a back-end or infrastructure component that provides functionality for the vehicle). In some examples, the system can provide functionality for auxiliary or autonomous driving of a vehicle.

[0079] Combination Figures 1 to 4 , basic aspects of training a machine learning module for validating or verifying a closed-loop testbench for testing a system have been described. In the following paragraphs, applications of the trained machine learning module for validating or verifying a closed-loop testbench for testing a system are discussed in more detail.

[0080] Figure 5 A flow chart of a method for validating or verifying a closed-loop test platform for a test system according to the present disclosure is shown. Figure 6 An exemplary method for validating or verifying a closed-loop test platform 81 of the present disclosure is schematically illustrated.

[0081] The method comprises: receiving 501 a data set 61a, 61b from a closed-loop test bench 81 for testing a system. The purpose of the method is to confirm or validate the data set 61a, 61b or the closed-loop test bench 81 used to generate the data set 61a, 61b. The data set 61a, 61b may be constructed in the same manner as the data set described above for training of a machine learning module (e.g., the data set contains input data and associated output data as described above). In some examples, the data set may include only input data or only output data (e.g., only selected input data or output data).

[0082] As described above, the technology of the present disclosure can be used for any system tested in a closed-loop test platform. In some examples, the system to be tested is a vehicle or a module for a vehicle (e.g., for a motor vehicle). The system can be configured to be installed in the vehicle itself and / or communicate with the vehicle (e.g., in a back-end or infrastructure component that provides functionality for the vehicle). In some examples, the system can provide functionality for auxiliary or autonomous driving of a vehicle.

[0083] The method further comprises feeding 502 at least a portion of the data set 61a, 61b of the closed-loop test platform 81 into a machine learning module 83 for validating or verifying the closed-loop test platform. For example, a portion of the test data set 61a, 61b may comprise input data and / or output data comprising information characterizing an operating scenario of a system included in the closed-loop test platform 81 detected or simulated using the closed-loop test platform 81 or an operating condition of a system included in the closed-loop test platform 81 detected or simulated using the closed-loop test platform 81. For example, a portion of the input data may characterize an operating scenario or an operating condition (e.g., a time record during the operating scenario or operating condition).

[0084] The machine learning module 83 is trained by using a specific similarity to estimate the similarity between the input data and output data of the data set of the closed-loop test platform to be confirmed or verified and multiple data sets of the at least one reference test platform as a function of the input data and / or output data of the data set of the closed-loop test platform to be confirmed or verified.

[0085] The method further comprises: outputting 503 one or more estimates 40a, 40b of similarity of input data and output data of a data set 61a, 61b of a closed-loop test platform 81 for testing the system (i.e., a closed-loop test platform to be confirmed or verified) and a data set of at least one reference test platform with a machine learning module 83. In some examples, a first similarity for the input data and a second similarity for the output data are output. As described above, the estimates 40a, 40b may be in different forms. For example, binary information may be output as to whether the input data and the output data are similar to the data set used for training. Alternatively (or additionally), characteristic parameters or other characteristic quantities for similarity may be output.

[0086] In some examples, the closed-loop test platform 81 for testing the system is a closed-loop simulation platform, that is, the environment of the system to be tested, the system to be tested, or both are created by simulation. For example, the input data delivered to the system to be tested in the closed-loop test platform can be generated by simulation (for example, by simulating other systems that provide input data to the environment of the system to be tested, the system to be tested, or both). In addition or alternatively, the output data of the system to be tested can be at least partially fed back into the simulation.

[0087] In some examples, the method includes: outputting multiple estimates of similarity of input data and output data of a data set 61, 61b of a closed-loop test platform 81 (i.e., a closed-loop test platform to be confirmed or verified) for testing the system and a data set of at least one reference test platform with a machine learning module 83. The multiple estimates can be estimates of similarity for different values ​​of operating parameters of the tested system and / or different operating scenarios or operating conditions. Based on the information, it can be identified whether the closed-loop test platform 81 to be confirmed or verified has adequately tested (or not adequately tested) a specific area.

[0088] The estimates of similarity 40a, 40b may be output in any suitable manner. In some examples, one or more similarity estimates 40a, 40b are output in a graphical user interface. In other examples, the estimates may also be output via other interfaces (in a human or machine readable form).

[0089] refer to Figure 7 or Figure 8 ,We now discuss other aspects of the exportable similarity estimates. Figure 7 An exemplary output of the disclosed method for validating or verifying a closed-loop test platform is shown. Figure 8 Other exemplary outputs of the disclosed method for validating or verifying a closed-loop test platform are shown.

[0090] In some examples, estimates may be determined and output for different points or regions of parameter space. Figure 7 and Figure 8 The example shows a two-dimensional parameter space (with a first axis 96 plotting a first parameter and a second axis 97 plotting a second parameter). In other examples, the parameter space may have more than two dimensions. In other examples, estimates may be output for two or more operating scenarios and / or operating conditions of the system.

[0091] Based on the estimation of similarity, the following areas 90 can now be identified in the parameter space, in which the closed-loop test platform to be confirmed or verified provides sufficient results (for example, the input and output data are similar to the input and output data of the reference test platform under a predetermined similarity criterion), as well as other areas 91, in which the closed-loop test platform to be confirmed or verified does not provide sufficient results (for example, the input and output data are not similar to the input and output data of the reference test platform under a predetermined similarity criterion). The evaluation can be performed automatically and output to the user (for example, via a graphical user interface). In other examples, instead of a binary classification ("similar" or "dissimilar"), a probabilistic classification can be performed and / or a value for a similarity measure can be output.

[0092] Figure 8 An output (e.g., via a graphical user interface) is shown, wherein estimates of similarity are output for a plurality of points in parameter space. A (first) estimate 94 of the similarity of the input data and a (second) estimate 95 of the similarity of the output data may be output for each point, respectively. Additionally, data points 92 or parameter ranges may be marked for which a data set of the reference test platform exists.

[0093] Based on the output of the similarity, one or more further steps may be performed (which may be performed automatically in some examples).

[0094] In some examples, data sets (e.g., for a specific test run of a closed-loop test platform to be confirmed or validated) for which data of a reference test platform (e.g., for a system to be tested in the field) are available can be checked, in particular with regard to reliability. In some cases, the trained machine learning module can also make a reliable estimate of these data sets: whether the data of the closed-loop test platform to be confirmed or validated have a sufficient degree of similarity with the expected data.

[0095] Additionally or alternatively, in some examples, the coverage of the parameter space by the closed-loop test platform to be confirmed or verified and / or the at least one reference test platform can be determined based on one or more estimates 40a, 40b. For example, areas for which data for reference test platforms are not yet available can be identified. Additional data can be requested and / or collected for these areas. For example, data can be requested and / or collected for the closed-loop test platform to be confirmed or verified and / or at least one reference test platform, where a specific operating condition or operating scenario is processed in the closed-loop test platform and / or the reference test platform (for example, if there is no similar segment of the closed-loop test platform to be confirmed or verified for a segment in the data set of the reference test platform, or vice versa).

[0096] Additionally or alternatively, a targeted new dataset may be requested (and generated) for the closed-loop testbed to be validated or verified in order to improve similarity with a dataset of a reference testbed (eg, by input data obfuscation).

[0097] If the closed-loop test platform is confirmed or verified using the technology of the present disclosure, the system to be tested contained therein can be released and further used (for example, in a product or in a downstream stage of the development process). The present disclosure also includes: deploying the system contained in the closed-loop test platform.

[0098] In the previous paragraphs, the technology of the present disclosure is mostly explained using the methods of the present disclosure.In some examples, the methods of the present disclosure can be performed in an automated (eg, computer-implemented) manner.

[0099] The present disclosure also relates to a computing unit designed to perform the method according to the present disclosure.The computing unit may take any suitable form in terms of its hardware or software (eg a standalone computing unit or a distributed computing unit, such as a computing unit in a cloud).

[0100] The present disclosure also relates to a computer program comprising instructions which, when executed on a computing unit, cause the computing unit to perform the steps of the method of the present disclosure.

[0101] The present disclosure also relates to a computer program product or a signal comprising the computer program of the present disclosure.

Claims

1. A method for training a machine learning module for confirming or validating a closed-loop test platform, wherein the closed-loop test platform is used to confirm or validate a system, the method comprising: providing (101) a plurality of data sets (20, 22) from two or more closed-loop test platforms, wherein the closed-loop test platforms include at least one reference test platform, wherein each data set (20, 22) contains input data and associated output data of the system under test in the corresponding closed-loop test platform; determining (102) similarities (50) of a plurality of segments of input data (31-38) for a pair of a plurality of data sets (20, 22), wherein each pair of data sets comprises a data set of one of the at least one reference test platform; determining (103) a similarity (50) of a segment of the output data associated with a segment of the input data (31-38); and A machine learning module is trained (104) in terms of estimating similarities between input data and output data of a data set of a closed-loop test platform to be confirmed or verified and data sets in a plurality of data sets of at least one reference test platform using a specific similarity (50) as a function of input data and / or output data of the data set of the closed-loop test platform to be confirmed or verified and similarities between the data sets, so as to generate a machine learning module for confirming or verifying the closed-loop test platform.

2. The method according to claim 1, wherein the machine learning module obtains only a part of the output data and / or the input data as input parameters and generates a similarity estimate of the input data and the output data as an output parameter.

3. The method according to claim 2, wherein: The part of the input data and / or output data includes the following information, which represents: the operating conditions of the system contained in the closed-loop test platform detected or simulated using the corresponding closed-loop test platform or the operating scenarios detected or simulated using the corresponding closed-loop test platform.

4. The method according to any one of the preceding claims 1 to 3, wherein: The segments of the output data and / or input data (31-38) include segments from a time record.

5. The method according to any one of the preceding claims 1 to 4, wherein: The sections of the output data and / or input data (31-38) are selected to contain data related to one or more predetermined operating conditions or one or more predetermined operating scenarios of the system to be tested in the corresponding closed-loop test platform.

6. The method according to claim 5, wherein the predetermined operating condition or operating scenario is a critical operating condition or operating scenario of the system, and optionally, wherein the critical operating condition or operating scenario includes a violation of a predetermined safety goal.

7. The method according to any one of claims 1 to 6, wherein determining the similarity of the segments of the input data comprises: A distance between a first section (31; 32; 33; 38) of input data of a first data set (20) of a respective pair and a second section (34; 35; 36; 37) of input data of the second data set (22) of the pair is determined.

8. The method according to any one of the preceding claims 1 to 7, wherein determining the similarity of the segments of the input data (31-38) comprises: The respective segments are compressed and the compressed segments of the input data are compared.

9. The method of claim 8, wherein compressing the respective segments comprises: Data characterizing the corresponding segments (31-38) are generated having a lower dimension than the original segments.

10. The method according to any one of the preceding claims 1 to 9, wherein the system comprises a vehicle or a module for use in a vehicle.

11. The method according to any of the preceding claims 1 to 10, wherein the at least one reference test platform comprises at least one test platform comprising a system in the field or on a test bench.

12. A method for validating or verifying a closed-loop test platform for a test system, the method comprising: Providing (501) a data set for a closed-loop test platform for testing the system; feeding (502) at least a portion of the data set of the closed-loop test platform into a machine learning module for validating or verifying the closed-loop test platform, wherein the machine learning module is trained in terms of estimating the similarity between the input data and output data of the data set of the closed-loop test platform to be confirmed or verified and the data set of at least one reference test platform as a function of the input data and / or output data of the data set of the closed-loop test platform to be confirmed or verified; and Output (503) one or more estimates of similarity between input data and output data of a data set of a closed-loop test platform for testing the system and a data set of at least one reference test platform having a machine learning module. 13 . The method according to claim 12 , wherein the closed-loop test platform for testing a system is a closed-loop test platform that simulates the system to be tested, an environment of the system to be tested, or both.

14. The method according to claim 12 or claim 13, further comprising: outputting a plurality of estimates of similarity between input data and output data of a data set of a closed-loop testbench for testing the system and a data set of at least one reference testbench having the machine learning module, The multiple estimates are estimates of similarities of different values ​​of operating parameters of the tested system and / or different operating scenarios.

15. A computing unit designed to execute the method according to any one of claims 1 to 14.

16. A computer program comprising instructions which, when executed on a computing unit, cause the computing unit to perform the method according to any one of claims 1 to 14.

17. A computer program product or a signal comprising a computer program according to claim 16.

Images