Method for validating or verifying technical system

Abstract

To provide a computer-implemented method for validating / verifying whether a technical system fulfills a desired criterion with a predefined probability, the technical system being configured to output signals based on input signals supplied thereto, a program, and a storage medium.SOLUTION: A method comprises: obtaining models M1, M2-MC for components S1, S2-SC included in a control system 40; obtaining connections between the models; obtaining, when a measurement input is supplied to a component in the control system, a plurality of verification measurements comprising a measurement input and a measurement output; obtaining test outputs for the models based on test inputs of the models and connections between the models; determining an upper boundary of an output or a lower boundary of the output by propagating upper or lower boundary of discrepancies through the models; and validating / verifying whether a technical system fulfills a criterion with a predefined probability based on the determined upper boundary and lower boundary of the output.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method, computer program, and machine-readable storage medium for verifying or confirming a technical system. [Background technology]

[0002] Conventional technology Shalev-Shwartz et al., 2018, "ON a Formal Model of Safe and Scalable Self-driving Cars," https: / / arxiv.org / pdf / 1708.06374.pdf, discloses a mathematical model for ensuring safety.

[0003] Danquah et al., 2021, "Statistical Validation Framework for Automotive Vehicle Simulations Using Uncertainty Learning," https: / / doi.org / 10.3390 / app11051983, discloses a statistical validation framework for dynamic systems involving changes in parameter configuration.

[0004] Background technology A technical system can be understood as comprising components and defining the interactions and / or interrelationships between each component. For example, a control system for a robot, such as an autonomous vehicle, typically includes a component that senses the robot's environment, a component that plans the robot's movements in that environment, and a component that determines the control signals to the robot's actuators to perform the planned movements. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Shalev-Shwartz et al., 2018, “ON a Formal Model of Safe and Scalable Self-driving Cars”, https: / / arxiv.org / pdf / 1708.06374.pdf [Non-Patent Document 2] Danquah et al., 2021, “Statistical Validation Framework for Automotive Vehicle Simulations Using Uncertainty Learning”, https: / / doi.org / 10.3390 / app11051983 [Overview of the project] [Problems that the invention aims to solve]

[0006] Modern technological systems typically involve a considerable number of components, making it extremely difficult to predict their behavior. In the example above, the component sensing the environment may make errors in the sensing process, such as overlooking an object in the environment, and the component planning may face a sensing environment where appropriate behavior is unknown to it, or there may be a discrepancy between the planned behavior and the behavior actually performed by the robot.

[0007] In general, confirming and / or verifying that a technical system exhibits desired behavior within its environment is extremely difficult, and guaranteeing such desired behavior is even more difficult. The main reason for this is that the components included in the system themselves may exhibit unknown and / or probabilistic behavior, and / or this behavior may be so complex that it can only be treated as a black box. For example, the environmental perception components of recent robots typically rely on methods from the field of machine learning for environmental sensing, particularly deep learning methods. These methods are inherently statistical, and their complexity typically prevents a direct approach to determining the precise behavior of the perception system.

[0008] Furthermore, the components of a system can themselves constitute a system; in other words, a system can actually be a system of systems. In such systems of systems, complexity increases extremely rapidly, making it extremely difficult to accurately predict the behavior of such systems, that is, to confirm and / or verify that the system is behaving as required or desired.

[0009] Advantageously, the proposed invention makes it possible to check and / or verify a technical system even when the technical system is complex, for example, when the technical system includes multiple components that are intricately linked to each other. [Means for solving the problem]

[0010] Disclosure of the invention In a first embodiment, the present invention relates to a computer-implemented method for determining the probability that a technical system that outputs a signal based on a supplied input signal satisfies a desired criterion, The step of obtaining models of the components included in the technical system and obtaining connections between the models of the components, wherein the connections characterize which model takes which input and which output passes to another model. A step of acquiring multiple verification measurements, including measurement inputs and measurement outputs, wherein, if the measurement input is supplied to a component of a technical system, the measurement output is acquired from that component in relation to the measurement input. The steps include obtaining the test output of the model based on the test inputs of the model and the connections between the models, A step of determining the upper or lower boundary of the output of a technical system by propagating the upper or lower boundary of a model mismatch through the model, wherein the model mismatch is characterized by a mismatch between the distribution of the measured output of a component and the distribution of the test output obtained for the model of that component. - A step of checking and / or verifying whether the technical system meets the criteria with a predetermined probability based on the upper boundary of the determined output, and / or checking and / or verifying whether the technical system meets the criteria with a predetermined probability based on the lower boundary of the determined output. Regarding methods including

[0011] Generally, the purpose of a method for verifying and / or validating a technical system can be understood as assisting the user of the method in determining the probability that the technical system meets a desired criterion. In this case, the determined probability can be compared with a probability threshold, and if the probability of meeting the desired criterion is equal to or greater than the probability threshold, the technical system can be considered verified and / or validated against that criterion. If the determined probability does not reach or exceed the probability threshold, the criterion can be considered unverifiable or unvalidable. For example, the technical system could be a mobile robot, and the criterion could be defined as "the robot does not deviate from its desired path by more than 50 cm." In this case, the method can statistically evaluate the probability that the criterion is met.

[0012] Selectively, this method can also be used to determine the probability of violating a desired criterion. In this case, it can be considered that confirmation and / or verification has been performed that the determined probability is below a probability threshold.

[0013] Advantageously, the inventors found that the determined probability is a guaranteed lower boundary for the probability of the criterion being met (or, conversely, a guaranteed upper boundary for the probability of the criterion being violated). This allows for a final response regarding the characteristics of the technical system, that is, enables a deep understanding of the technical system, and ultimately allows the technical system to function correctly.

[0014] In some embodiments, the desired criterion may be one or more requirements relating to some specification, such as a design specification or a legal specification. For example, in the powertrain (i.e., technical system) of a vehicle equipped with an internal combustion engine, it may be required to generate only a specified amount of emitted particles over a given operating time (e.g., the number of NOx particles must be below a given threshold). In such specific cases, the method can assist engineers performing verification in determining the probability that the powertrain will actually operate according to the specified emission amount.

[0015] The probability threshold can be selected according to the system specifications. Selectively, the probability threshold may be determined by or derived from legal regulations.

[0016] In general, this method can be used for any system including components or subsystems. For all such systems, this method enables a guided process. Verification and / or confirmation is typically a built-in component with respect to the release of a system under development, so that the system can be released, for example, when it has been verified that a criterion is met and / or is met. Optionally, this method can also be used to verify and / or confirm multiple criteria, so that the system can be released when it has been verified and / or confirmed that all criteria are met. Optionally, this method may be part of a larger test strategy for evaluating whether a system is releaseable. In all the embodiments described above, if one or more criteria cannot be verified and / or confirmed, the system may be considered, for example, not releaseable. In such cases, components of the technical system can be improved, and this method can be run again after the component improvements to verify and / or confirm the improved system with respect to one or more criteria. For example, in the embodiments described above, engine output can be limited to reduce particle emissions, and / or powertrain components can be replaced with components that are more efficient with respect to particle emissions.

[0017] In other words, this method can be understood as a human- and machine-guided process for evaluating whether a system is expected to meet certain criteria when used in the real world.

[0018] Technical systems interact with the real world by determining output signals based on input signals. In the exemplary embodiments described above, the powertrain may include sensors to measure, for example, the ambient temperature of the powertrain, and can appropriately control components of the engine and / or exhaust gas aftertreatment system, such as a catalyst. However, input signals to the system are not necessarily provided to the technical system using sensors or communication devices. Input signals may also characterize environmental conditions of the technical system that affect its operation. For example, the amount of emissions produced by an internal combustion engine typically depends on the ambient temperature of the engine. In this embodiment, the engine does not receive temperature information from sensors or communication devices, but temperature can be considered an input signal to the technical system. In other words, although the engine does not measure temperature, temperature has a physical effect on the engine. Therefore, input signals can also be understood as input stimuli.

[0019] Therefore, various components of a technical system can be connected, for example, by the exchange of information via signals (e.g., measurement data, control signals) and / or by physical interaction (e.g., exhaust gas treatment).

[0020] In general, the technical systems described in the present invention can be extremely complex, for example, with respect to the interactions of the individual components of the technical system and / or the relative complexity of the individual components of the technical system, which can make it extremely difficult to accurately evaluate the behavior of the technical system in response to various environmental conditions and various input signals. The only method known to evaluate the behavior of such a technical system is typically to treat the technical system as a black box regarding its input / output behavior, run the technical system in the real world to collect data on its input / output behavior, and extract information about the input / output behavior using statistical methods. This can also be understood as data collection to predict the input / output behavior of the technical system. However, such an approach has a major drawback: it typically requires the collection of a large amount of data to accurately evaluate the input / output behavior. In particular, in safety-critical systems and / or technical systems constrained by some form of legal regulation regarding input / output behavior (e.g., only a maximum number of emitted particles over a given operating time) where data is collected to accurately predict input / output behavior, numerous tests or verification actions are required, which are often impractical due to the enormous amount of data that needs to be collected.

[0021] Advantageously, the proposed method significantly reduces the actual amount of real-world data that needs to be collected, while also guaranteeing an approximation of the input / output behavior of the technical system. In particular, this method enables verification and / or validation of a technical system using only component-level data, without requiring end-to-end (black-box) data. Therefore, verification and / or validation can be performed during the design phase of the technical system, even before the entire technical system is assembled.

[0022] From an abstract perspective, this method can be understood as constructing a model of a technical system by modeling various components of the technical system using different models, and then using these models to simulate the behavior of the technical system. Advantageously, the model of the technical system (e.g., a combination of different models of the components of the technical system) is linked to the technical system so that a proposition of the difference between the model's output and the technical system's output can be determined. Based on this difference proposition, a probabilistic guarantee of the behavior of the technical system itself can be obtained, even if the data used to determine the model's output is based on simulations.

[0023] In the first step, a model of the components of the technical system is obtained. The model may be a physical model. In the example above, the engine of the powertrain (the powertrain is the technical system, and the engine is a component of the technical system) can be modeled by a physical model of the engine. Optionally, the model may be given by the component itself. For example, the engine control unit may be an algorithm instantiated by a software component. The software itself may be used directly as the model. Optionally, the model may be a statistical model, a machine learning model such as a neural network, etc.

[0024] Advantageously, the granularity of the modeling can be selected at the user's discretion. That is, the level of detail in the model can be chosen at the user's discretion. For example, the user can decide whether to model the engine components as a single model, or to apply finer-grained modeling by modeling the behavior of engine parts, such as pistons, valves, and crankshafts, such as fuel injection behavior, combustion behavior, and / or mechanical properties.

[0025] In other words, this method is independent of the level of component modeling (coarse granularity, fine granularity, or some intermediate granularity).

[0026] In the second step, verification measurements are obtained for the components of the technical system. In the context of this method, verification measurements are understood as pairs of input and output measurements, with output measurements obtained when input measurements are supplied to a component. In other words, output measurements are linked to a component. The output measurement of a first component can be supplied to a second component as an input measurement. The input measurements of the technical system can be used as input measurements to one or more components of the technical system. Advantageously, verification measurements for each component of the technical system can be obtained by running and measuring each component separately, for example, on a test bench. If the entire technical system is already assembled and / or executable, verification measurements can also be obtained by running one or more test processes (sometimes referred to as verification processes) of the technical system in the real world and measuring each input and output measurement of each component.

[0027] In the third step, test inputs are obtained for each model. These test inputs are used to determine the test output by running the model on the test inputs. Preferably, this can be achieved, for example, by transferring potential inputs obtained by simulation through a model chain to the technical system, and determining the test inputs and test outputs for each model in this model chain.

[0028] A pair of test inputs and test outputs can be considered equivalent to a pair of measurement inputs and measurement outputs. While measurement inputs and outputs are linked to components of the technical system, test inputs and test outputs are linked to models of those components. Test inputs and test outputs can be obtained, in particular, through simulation, for example, by synthesizing potential inputs into the technical system and then transferring these synthesized inputs through the model to determine the test inputs and test outputs. Advantageously, this allows for the simulation of the technical system's behavior and the determination of how closely the simulation results reflect the real-world behavior of the technical system. In other words, the simulation results provide the user with statistical assurance regarding the technical system's behavior.

[0029] In this method, this is achieved by comparing the distribution of component outputs with the distribution of model outputs. This comparison may, in particular, be based on a measure of discrepancy between probability distributions. The method can determine its assurance based on an upper boundary of how much the output distribution of the technical system deviates from the output distribution obtained from the model of the technical system. Model discrepancy can be understood as characterized by a discrepancy between the distribution of measured outputs obtained for a component and the distribution of test outputs obtained for the model of that component. In other words, the inventors have found that discrepancies for individual components can be propagated through the model of the technical system, where the model of the technical system includes component models and their connections, i.e., characterized by which models supply input to which other models. By propagating discrepancies, the method can determine an upper boundary or a lower boundary of the output of the technical system. This upper or lower boundary can be used for verification and / or validation tasks.

[0030] The discrepancy between two distributions (also called the discrepancy measure) can be understood as a function that maps two probability distributions (defined in the same space) to real numbers. Discrepancy can be understood as a quantification of how close the first probability distribution is to the second, although this is not technically necessary. In particular, discrepancy does not need to be positively oriented, absolutely positive, symmetric, and / or satisfy the triangle inequality.

[0031] An example of a function used to determine a mismatch is: • The maximum mean mismatch (MMD) (or its square) between distributions with respect to the kernel function in the basis space of the distributions; • Cosine similarity of kernel functions in the basis space of the distribution, • Wasserstein distance for distance measures in the basis space of a distribution; • The norm distance between two distributions derived from an arbitrary norm, e.g., the total variation norm; f-divergence, e.g., Kullback-Leibler divergence, Renyi divergence, or similar measures; • Affine combination of any mismatched scales or any real-valued function; • Relaxation and approximation of any of these discrepancies; That is the case.

[0032] Preferably, the function is used as a discrepancy measure that carries a proposition of distance in a defined space of the distribution, for example, as the Wasserstein distance, or as a measure based on a kernel that carries such a proposition of distance (for example, via a length scale or via an intermediate embedding).

[0033] Preferably, relaxation and approximation of such discrepancy measures can be used (these are again discrepancy measures). Preferably, the discrepancy can be relaxed or formulated to obtain a convex or concave function. This is advantageous because the determination of the upper or lower boundary of the output of the technical system can be achieved by an optimization problem, and the resulting optimization can then be solved by a convex solver. The use of a convex solver can result in a reduction in the time required for optimization.

[0034] Preferably, the distribution is modeled as a weighted empirical distribution. That is, given components and their corresponding models, weights can be assigned to the measurement inputs, measurement outputs, test inputs, and test outputs of each component or model. The weights for the measurement inputs and measurement outputs can be understood as enabling the construction of a distribution from measurement inputs that are sufficiently close to the distribution of test inputs, and further enabling the construction of a distribution from measurement outputs that represent the worst-case scenario in terms of achieving the desired criterion.

[0035] In a preferred embodiment of the present invention, the upper boundary of the model mismatch is propagated through the model by iteratively determining the upper boundary of the mismatch for a second model, where the mismatch is determined based on the mismatch for a first model that supplies input to the second model.

[0036] Preferably, the discrepancy with respect to the second model is expressed in the first equation, i.e.,

number

number

number

[0037] As can be seen from the first equation, the discrepancy can be determined for each model based on the discrepancy determined for the "preceding" model, where the order of the models is determined by which model supplies input to which other model. For example, the first model supplies input to the second model and thus precedes the second model. It is also possible to provide a plurality of first models that supply input to the second model. In this case, the first equation derived above obtains constraints that follow the first constraint shown in the first equation for each model that supplies input to the second model.

[0038] The distribution p α can be understood as the distribution of the measurement input of the second model, and the distribution S c+1 [p α can be understood as the distribution of the measurement output, where the weight of the elements in the measurement output is equivalent to the weight of the measurement input used to determine the measurement output.

[0039] In the case of models that directly process the inputs or synthetic inputs of a technical system, constraints on the distribution of the inputs or synthetic inputs may be provided as a starting point for propagating the upper boundary of discrepancies through the model. Such constraints may, for example, characterize the worst-case deviation between the data occurring in the real world and the synthetic data used to determine the model's test output.

[0040] The first equation above can also be used for models that use synthetic data as input, i.e., models that do not have another model preceding them. In this case, p α This can be understood as the distribution of measurement inputs acquired for the input components of a technical system, i.e., the components that correspond to a model in which no other model precedes it, and q c This is understood as the distribution of test inputs for the model corresponding to the component, for example, the distribution of synthetic data, and B c This can be understood as the maximum deviation between the distribution of the measurement inputs for the input component and the distribution of the test inputs for the model corresponding to the input component. In this case, B c This can be understood as the a priori predicted worst-case deviation between data occurring in the real world and data used to determine the output from the model, such as synthetic data. The a priori predicted worst-case deviation can be supplied to this method as an external parameter, while B from the data 0 It is also possible to estimate B. 0 This can be estimated from the measured inputs of the input components and the test inputs of the models corresponding to the input components. Preferably, this can be achieved by determining the discrepancy between the distribution of the measured inputs and the distribution of the measured outputs using a discrepancy measure such as MMD or MMD squared. For this purpose, the weights of each distribution can be selected to be uniform. Selectively, based on potential real-world inputs that have not been used as measured inputs or have not been used as measured inputs, B 0It is also possible to estimate this. For example, a verification process can be performed to collect only the potential inputs of the system without recording the individual measured outputs of the components. In this case, B 0 To determine this, a potential input can be used instead of the measured input in the above approach.

[0041] In general, the optimization problem given by the first equation may not be a convex optimization problem. Preferably, a transformed mismatch scale can be used, which can relax the optimization problem into a convex (and possibly semi-definite) optimization problem.

[0042] Preferably, the quadratic maximum mean discrepancy is used as the discrepancy measure, where the quadratic maximum mean discrepancy measure is given by the second equation, i.e.,

number

number

number

[0043] The authors argue that the relaxation of the optimization problem according to the first equation leads to the third equation, namely,

number

number

number

number

[0044] By iteratively or recursively determining the upper boundary for each model, the user ultimately arrives at the upper boundary of the model corresponding to the output component of the technical system, i.e., the component that supplies the output signal, output information, or output operation of the technical system. In the powertrain example described above, the output could characterize, for example, the amount of particles emitted by the technical system, where the output component is a measuring device that determines the particles emitted from the exhaust component of the powertrain.

[0045] Once these upper boundaries of the output components are obtained, the probability that the desired criterion is met can then be determined. For example, the output of the system (i.e., p C A sample from can be characterized by a real value, and this criterion can define a threshold θ that the value must not exceed. The lower boundary for the probability that this criterion is met is given by the equation, i.e.,

number

[0046] In other embodiments, the criterion may define a threshold below which the output must not fall. The lower boundary of the probability of satisfying the criterion is given by the formula, i.e.,

number

[0047] The probability that the criterion is violated can be obtained by using -∞ as the lower boundary of the integral and θ as the upper boundary of the integral, and by replacing minimization with maximization.

[0048] In a preferred embodiment, the output of the technical system can also be evaluated by a function to be validated against a desired criterion. The function to be validated is a function that maps any output y of the output components of the technical system to a real-valued quantity.

number

number

number

number

number

[0049] The specification can be understood as synonymous with the desired standard.

[0050] In relation to this method, it can be understood that the largest or smallest function under verification serves as a means of determining whether the system can be confirmed and / or verified against the criterion.

[0051] This is preferably expressed in the fourth equation, namely,

number

[0052] To make optimization computable, V max Preferably, this can be determined by quantifying the output space of the output component, for example, by equidistant points or a general grid, assigning weights to each point (or each point in the grid), and solving an optimization problem similar to the problem for determining the upper boundary of the model. Thus, the optimization is expressed by the equation, i.e.,

number

[0053] In this case, V max This can be used as a statistical guarantee regarding the desired criterion. For example, using the example of a powertrain, the output can be a count of an emission amount, such as the number of NOx particles emitted from the powertrain, in which case V maxThis can represent this quantity.

[0054] Failure to verify and / or validate the criteria may simply depend on the models used in this method being too inaccurate to obtain sufficiently tight boundaries for the outputs of the models that model the output components. Loose boundaries may result in the criteria not being met, where the failure to meet the criteria may simply be due to inaccuracies in one or more models for each component, resulting in an overly pessimistic view of the output boundaries. Therefore, one or more models can be favorably improved, i.e., adapted to more accurately reflect the behavior of their respective components. This can be achieved by using the mismatch of each model as a loss function and optimizing the models to reduce the mismatch. If the models are differentiable, this can be achieved by gradient descent algorithms. Selectively, or for non-differentiable models, evolutionary algorithms can be used for optimization.

[0055] Advantageously, model improvements result in tighter model boundaries corresponding to output components, leading to a more accurate assessment of the criteria. These improvements should preferably be evaluated using measurement data not used in the model improvements (i.e., measurement inputs and output), otherwise, such data could lead to overfitting and / or information leakage.

[0056] In any one of the embodiments described above, if the criteria cannot be confirmed and / or verified, the components of the technical system can be improved.

[0057] This should be understood as meaning that the technical system has been determined to be unable or highly unlikely to be able to meet the desired standards, and therefore improvements must be made to meet those standards. Using the powertrain as an example, improvements to components may include, for example, reducing engine power to achieve lower fuel consumption and thus lower particle emissions, and / or replacing the powertrain catalyst to remove more particles.

[0058] Embodiments of the present invention will be described in more detail below with reference to the drawings. [Brief explanation of the drawing]

[0059] [Figure 1] This is a diagram of the control system. [Figure 2] This figure shows a control system for controlling a robot that is at least partially autonomous. [Figure 3] This is a diagram showing a control system for controlling manufacturing machinery. [Modes for carrying out the invention]

[0060] Description of the Embodiment Figure 1 shows, as an example, multiple components (S 1 ,S 2 ,S C It is shown how a single technical system (40) having ) can be confirmed and / or verified. The control system (40) can be understood as one embodiment of the technical system referred to in this application. For each component, the respective model (M 1 M 2 M C ) is available. Model (M 1 M 2 M C ) is a component (S) indicated by a dashed arrow. 1 ,S 2 ,S C ) has a one-to-one correspondence with .

[0061] The control system (40) includes an input component (S) configured to accept input from a sensor. 1 ) includes. The sensor may preferably be part of the control system (40). The sensor supplies the input measurement to the input component. The input measurement (input signal) is distributed (p 0 It can be understood that it follows the distribution (p 0 ) can preferably be modeled by a weighted empirical distribution. The samples for constructing the weighted empirical distribution can be determined by performing a verification process to collect input signals. In this case, the weighted empirical distribution is given by the equation, i.e.,

number

number

number

[0062] The control system (40) is configured to determine an output or operation based on an input measurement. For this purpose, the input measurement is transferred through each component. Thus, each component receives an input that is based on an input measurement but has been processed by a component of the control system (40) (unless that component is an input component). For example, an input component (S 1 ) receives an input measurement value, determines some output from this input measurement value, and sends the determined output to another component (S) of the control system (40). 2 It is then forwarded to the other component (S). 2 ) processes the input supplied to it and forwards its output to another component (not shown).

[0063] Following this chain, each component (S 1 ,S 2 ,C C ) receives an input (also called a measurement input) and supplies an output (also called a measurement output). Component (S 1 ,S 2 ,C C The output of ) is preferably some distribution (p) that can be modeled by a weighted empirical distribution. 1 ,p 2 ,p C-1 ,p C It can be understood that it follows the formula: Such a weighted empirical distribution is given by the formula, namely,

number

number

number

number

number

number

number

Number

[0064] For each component, there are models (M 1 , M 2 , M C ). Each model is connected according to the connection state of the corresponding components (S 1 , S 2 , C C ). The input model (M 1 ) corresponding to the input component (S 1 ) is supplied with data from the test dataset, i.e., test input. The test input can preferably be determined synthetically based on, for example, a synthetic model configured to model the input measurement values of the control system (40). The synthetic model may preferably be a model from the field of machine learning, such as a generative model like a adversarial generative network.

[0065] According to the modeling chain, each model (M 1 , M 2 , M C ) receives an input (also referred to as test input) and supplies an output (also referred to as test output). The output of the models (M 1 , M 2 , M C ) can preferably be understood to follow some distribution (q 1 , q 2 , q C-1 , q C ) that can be modeled by a weighted empirical distribution. Such a weighted empirical distribution can be characterized by the formula, i.e.,

Number

number

number

number

number

number

number

[0066] Output component (S) of the control system (40) CThe output of the control system (40) may be a signal characterizing an action to be performed and / or a signal characterizing a real number. For example, if the control system (40) is configured to determine control signals for a robot (e.g., a partially autonomous vehicle), the output may characterize an advanced action to be performed by the robot (e.g., performing a lane change) or a numerical value used to control the robot (e.g., steering angle position, acceleration and / or braking force). The output may be understood as being subjected to verification and / or validation by the proposed method against some desired criterion. Using the robot example, the criterion may be that any action performed by the robot does not cause the robot to deviate from a predetermined or determined path by a predetermined allowable distance. In other embodiments (not shown), the output component of the control system (40) may be a “special” component that is not normally used when operating the control system (40) but is used for evaluating the behavior of the control system (40). For example, the output component may be, for instance, an emission amount measuring device that determines the amount of particles emitted from the components of the control system (40), if the control system (40) is a powertrain.

[0067] Preferably, there may be a function to be verified that maps the output of the output component to a real value. This function to be verified is a function that maps any output y of the output component of the control system (40) to a real value quantity.

number

number

number

number

number

[0068] Preferably, the square of MMD can be used as the mismatch scale D. The model corresponding to the input component (M 1 To determine the discrepancy (D) of ), discrepancy B 0 This is required. In this embodiment, this is achieved by determining the square of the MMD between the uniformly weighted empirical distribution of the input measurements and the uniformly weighted empirical distribution of the test dataset (of the inputs).

[0069] After discrepancies in the models corresponding to the output components are obtained, the upper (or lower) boundary of the probability that the specifications are met can be determined.

[0070] Figure 2 shows an embodiment in which a control system (40) is used to control a robot that is at least partially autonomous, for example, a vehicle (100) that is at least partially autonomous.

[0071] The vehicle (100)'s sensors (30) may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors. Some or all of these sensors are preferably mounted on the vehicle (100), although this is not mandatory. The control system can be configured, for example, to automatically maintain the vehicle (100) so that its distance from other objects in the environment exceeds a predetermined distance, or to ensure that the time until the vehicle collides with another vehicle in the environment is not shortened below a predetermined threshold. A desired criterion to be verified may be, for example, that the probability of the vehicle (100) approaching another vehicle at a distance shorter than a predetermined distance is below a predetermined percentage threshold.

[0072] The control system (40) preferably includes components, such as input components (S 1The system may include an image classifier. The image classifier can be configured to detect objects in the vicinity of at least partially autonomous robots based on input images. The measured output of the input images may include information characterizing where objects are located in the vicinity of at least partially autonomous robots. In this case, subsequent components can determine a suitable driving path through the environment such that a predetermined distance or threshold for time to collision is maintained for all recognized objects. Control signals can then be determined from the output components of the control system (40). Control signals can be used to control the actuators (10) of the vehicle (100). Control signals can be configured to drive the vehicle according to the driving path.

[0073] Preferably, the actuator (10) mounted on the vehicle (100) may be provided by the vehicle's (100) brakes, propulsion system, engine, powertrain, or steering system.

[0074] In other embodiments, the at least partially autonomous robot may be provided by another mobile robot (not shown) capable of moving, for example, by flight, swimming, submersion, or walking. The mobile robot may be, in particular, an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In all of the above embodiments, control signals can be determined to control the propulsion unit and / or steering unit and / or brakes of the mobile robot so that the mobile robot can avoid collisions with the identified object.

[0075] In other embodiments, a robot that is at least partially autonomous may be provided by a horticultural robot (not shown) using sensors (30), preferably optical sensors, that determine the state of plants in the environment (20). An actuator (10) can control a nozzle that sprays liquid and / or a pruning device, such as a blade. Depending on the identified species and / or the state of the identified plant, the actuator (10) can determine a control signal to spray an appropriate amount of liquid onto the plant and / or prune the plant.

[0076] In yet another embodiment, the robot, at least partially autonomous, may be provided by a household appliance (not shown), such as a washing machine, stove, oven, microwave oven, or dishwasher. A sensor (30), such as an optical sensor, can detect the state of an object being processed by the household appliance. For example, if the household appliance is a washing machine, the sensor (30) can detect the state of the laundry inside the washing machine. In this case, a control signal can be determined according to the material of the detected laundry.

[0077] Figure 3 shows an embodiment in which the control system (40) is used to control a manufacturing machine (11) of a manufacturing system (200) as part of a production line, such as a punch cutter, cutter, gun drill, welding robot, or gripper. The manufacturing machine (11) may include a conveying device for moving the manufactured product (12), such as a conveyor belt or assembly line. The control system (40) further controls actuators (10) that control the manufacturing machine (11).

[0078] The sensor (30) used to supply input measurements to the control system (40) may be provided, for example, by an optical sensor that captures the characteristics of a manufactured product (12). The control system (40) receives input components (S 1 It can be equipped with an image classifier.

[0079] The image classifier can determine the position of the product (12) relative to the transport device. The actuator (10) can then be controlled according to the determined position of the product (12) for subsequent manufacturing processes of the product (12). For example, the actuator (10) can be controlled to cut or weld the product (12) along a specific path. In this embodiment, the path can be determined by a component of the control system (40), and then another component of the control system (40) determines the control signal for the actuator (10) of the manufacturing machine (11). A desired criterion may be that the cutting or welding does not deviate from the planned path by a predetermined threshold.

[0080] It is also conceivable that the image classifier selectively classifies whether a manufactured product is damaged or defective. In this case, the actuator (10) may be controlled to remove the manufactured product (12) from the transport device. In this case, the desired criterion may be that the amount of manufactured products (12) that are incorrectly classified and removed (e.g., products that are not actually damaged, or products that are not defective but are classified and removed) falls below a predetermined threshold.

Claims

1. 1. A computer-implemented method for verifying and / or verifying whether a technical system (40) that outputs an output signal based on an input signal provided thereto satisfies a desired criterion with a predetermined probability, comprising: The components (S) included in said technical system (40) １ , S ２ , S Ｃ ) model (M １ , M ２ , M Ｃ ) and obtain the component (S １ , S ２ , S Ｃ ) model (M １ , M ２ , M Ｃ ) and the connection is obtained between any model (M １ , M ２ , M Ｃ ) as input and which output is to be used by another model (M １ , M ２ , M Ｃ ) characterizing the passage of the step, - obtaining a plurality of validation measurements including a measurement input and a measurement output, said measurement input being connected to a component (S) of said technical system (40); １ , S ２ , S Ｃ ), the measurement output is supplied to the component (S １ , S ２ , S Ｃ ) and ・The above model (M １ , M ２ , M Ｃ ) test input and the model (M １ , M ２ , M Ｃ ) based on the connections between the models (M １ , M ２ , M Ｃ ) obtaining test output; ・The above model (M １ , M ２ , M Ｃ ) mismatch (D) upper or lower boundary (B １ , B ２ , B Ｃ－１ , B Ｃ ) into the model (M １ , M ２ , M Ｃ determining upper and lower bounds for the output of the technical system (40) by propagating the model (M １ , M ２ , M Ｃ ) mismatch (D) is caused by a component (S １ , S ２ , S Ｃ ) distribution of measured output (p １ , p ２ , p Ｃ－１ , p Ｃ ) and the component (S １ , S ２ , S Ｃ ) model (M １ , M ２ , M Ｃ ) the distribution of test outputs obtained for １ , q ２ , q Ｃ－１ , q Ｃ characterizing a discrepancy (D) between - checking and / or verifying whether the technical system (40) satisfies the criteria with the predetermined probability based on the determined upper output boundary and / or checking and / or verifying whether the technical system (40) satisfies the criteria with the predetermined probability based on the determined lower output boundary; A method comprising:

2. The model (M １ , M ２ , M Ｃ ) discrepancy (D) upper boundary (B １ , B ２ , B Ｃ－１ , B Ｃ ) is the upper bound (B) of the discrepancy (D) for the second model. １ , B ２ , B Ｃ－１ , B Ｃ ) by iteratively determining the model (M １ , M ２ , M Ｃ 2. The method of claim 1, wherein the discrepancy (D) is determined based on the discrepancy (D) for a first model that provides input to the second model.

3. 3. The method of claim 2, wherein the discrepancy (D) for the second model is determined by maximizing the discrepancy (D) of distributions of measurement outputs obtained for the component, the second model corresponding to a distribution of test outputs obtained for itself, and the discrepancy is maximized over possible distributions of measurement outputs obtained for the component.

4. The method of claim 3 , wherein the maximization of the discrepancy is a convex optimization problem under convex constraints.

5. The method of claim 3 , wherein the distribution of measurement outputs obtained for the component is a weighted empirical distribution.

6. The discrepancy for the second model is determined by the first equation, i.e., [Equation 1] The method of claim 5 , wherein the formula is determined according to:

7. The method of claim 6 , wherein the discrepancy between the distribution of the measurement outputs and the distribution of the test outputs is determined by a weighted sum of kernel estimates for elements of each distribution.

8. The method of claim 7 , wherein the discrepancy for the second model is determined according to a relaxation of the first equation that characterizes a convex optimization problem.

9. The method of claim 1 , wherein the test inputs and the test outputs are determined by synthesizing inputs of the technical system (40) and routing the synthesized inputs through the model.

10. The method of claim 1 , further comprising improving the model if the criteria cannot be confirmed and / or verified.

11. The method of claim 1 , further comprising improving each component of the technical system (40) if the criteria cannot be confirmed and / or verified with the predetermined probability.

12. 2. The method according to claim 1, wherein the technological system (40) is configured to provide control signals to manufacturing machines and / or robots.

13. A computer program configured, when executed by a processor, to cause the computer to carry out all the steps of the method according to any one of claims 1 to 12.

14. A machine-readable storage medium storing the computer program according to claim 13.

Images