Apparatus and computer-implemented method for testing technical systems, particularly systems for determining virtual sensor signals.
The method enhances virtual sensor signal testing by using a surrogate model and adaptive training data refinement to ensure accurate and reliable predictions, addressing the limitations of existing systems in evaluating virtual sensor signals.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2025-10-23
- Publication Date
- 2026-05-12
AI Technical Summary
Existing systems lack effective methods for testing virtual sensor signals using artificial neural networks in safety-related manufacturing, particularly in evaluating the reliability and accuracy of virtual sensor predictions.
A computer-implemented method and apparatus that utilizes a surrogate model to determine virtual sensor signals by mapping input values to predictions using a machine learning function, such as an artificial neural network or Gaussian process, and evaluates the distance between predictions and actual output values to assess the model's reliability, employing a simplex-based approach with probability distributions and tolerances to refine the training data set.
Enhances the reliability and accuracy of virtual sensor signal testing by adaptively updating the training data set based on probability thresholds, ensuring the model's predictions align closely with real-world physical processes, thereby improving the model's predictive capability.
Smart Images
Figure 2026076985000001_ABST
Abstract
Description
[Technical Field]
[0001] Conventional technology The present invention relates to a technical system, in particular an apparatus and a computer-implemented method for testing a system for determining virtual sensor signals. [Background technology]
[0002] Solutions that enable the use of artificial neural networks in safety-related manufacturing are attracting considerable interest. [Overview of the Initiative] [Means for solving the problem]
[0003] Disclosure of the invention According to the technical system described in independent claim 1, in particular a computer-implemented method for testing a system for determining, for example, a virtual sensor signal of a virtual sensor, the system includes a model that models a physical process, in which a physical input quantity of the physical process yields a physical output quantity of the physical process, the input space includes input values of the physical input quantity, and the output space includes output values of the physical output quantity, a surrogate model is provided for the model, the surrogate model is configured to describe a real physical process, input values are detected, and in the input space, a simplex stretched by the detected input values is determined, and each simplex A method is envisioned in which, for each input space, at least one input value is determined in particular in human-machine interaction with a user, or randomly drawn on a simplex according to a predetermined probability distribution, particularly a uniform distribution, and an output value for the input value is determined using a surrogate model, and the input value is mapped to a prediction for the output value using the model, particularly to determine a virtual sensor signal, and the distance between the prediction and the output value is determined, and the probability that the prediction deviates by more than a predetermined deviation from the physical output value of a physical process for a physical input quantity having an input value when the model maps the input value of a physical input quantity from the input space to a prediction is included in the test results. This allows for a qualitative evaluation of the region defined by the simplex in the system's input space. In this case, input values in the input space, i.e., data points, are detected, and training data that can be used as the basis for constructing the simplex is detected, for example.
[0004] The model may include a machine learning predictive function, such as an artificial neural network or a Gaussian process.
[0005] It can be assumed that the distance is determined by the absolute value of the difference between the prediction and the output value.
[0006] It can be assumed that the probability is determined by the ratio of the number of distances greater than the tolerance to the number of distances determined during the test. This allows for evaluation when distances outside the tolerance range exist.
[0007] The probability is determined using a generalized Pareto distribution, which can be assumed to be fitted to distances smaller than the tolerance determined for the simplex. This makes evaluation possible when there are no distances outside the tolerance range.
[0008] During the test, it can be assumed that a check is performed to determine whether at least one distance greater than the tolerance has been determined, and that the probability is determined by the ratio if at least one distance is greater than the tolerance, and by a generalized Pareto distribution if it is not.
[0009] The surrogate model is configured to describe real-world physical processes with tolerances, which can be assumed to depend on the acceptable deviation, specifically determined as part of the deviation, for example, half of the deviation. This allows the evaluation to be fitted to the tolerance.
[0010] The probability can be assumed to be determined with the same tolerance for multiple simplexes, particularly in a temporally overlapping or simultaneous manner, or to be determined with different tolerances for at least two simplexes. Using the same tolerance allows for rapid evaluation. With different tolerances, the acceptable deviations can be adapted to the domain of the input space.
[0011] For example, for each simplex, multiple distances to multiple different input values are determined. This increases the persuasiveness of the probabilities.
[0012] The model is learned based on the measured training data set, which includes pairs of input values from the input space and output values from the output space. If the probability for at least one simplex is greater than a predetermined threshold value, particularly when having a probability greater than the threshold value, new training data having input values from the simplex with a particularly high probability is measured and added to the training data set. The model is learned based on the training data set supplemented with the training data. Or, if the probability for any simplex is not greater than the predetermined threshold value, it can be assumed that the model is approved for reliable prediction of the physical process.
[0013] According to an apparatus for testing a system, particularly a system for determining virtual sensor signals of a virtual sensor for example, the apparatus includes at least one processor and at least one memory. The at least one processor is configured to execute instructions stored in the memory. When the instructions are executed by the at least one processor, it is assumed that the apparatus implements the method.
[0014] It is possible to assume a computer program that includes computer-executable instructions, and when the computer-executable instructions are executed by a computer, the method is implemented.
[0015] Further advantageous embodiments can be taken from the following description and the drawings.
Brief Description of the Drawings
[0016] [Figure 1] It is a schematic diagram of an apparatus for testing. [Figure 2] It is a flowchart having steps of a method for testing. [Figure 3] It is a schematic diagram of an input space. [Figure 4]This is a schematic diagram of the output values of a model used to determine the output values of a physical process. [Figure 5] This is a schematic diagram of a histogram showing the distance between the prediction and the output value. [Modes for carrying out the invention]
[0017] Figure 1 shows a schematic representation of the apparatus 100.
[0018] The apparatus 100 includes at least one processor 102 and at least one memory 104. The apparatus 100 is configured to test a technical system 106. In this example, the technical system 106 is located inside the apparatus 100 for testing. The technical system 106 may be located outside the apparatus 100 for testing.
[0019] Technical system 106 is, for example, a system for determining a virtual sensor signal. Technical system 106 is, for example, a virtual sensor.
[0020] The technical system 106 includes a model. The model includes, for example, a machine learning function, such as an artificial neural network f or a Gaussian process. The technical system 106 is configured to detect an input value 108 from a machine 110. The machine 110 is, for example, an internal combustion engine or an electric motor.
[0021] At least one processor 102 is configured to execute instructions stored in memory 104, and when those instructions are executed by at least one processor 102, the device 100 performs a method for testing the technical system 106.
[0022] The model is configured, for example, to predict output values for determining a virtual sensor signal.
[0023] The technical system 106 is configured, for example, to determine and output a virtual sensor signal based on a prediction.
[0024] It can be assumed that the technical system 106 is configured for human-machine interaction with the user.
[0025] The input quantity 10⁸ can be assumed to be either one-dimensional or multi-dimensional.
[0026] The apparatus 100 for testing the technical system 106 is connected, for example, to a test bench. The test bench is configured to measure a real-world physical system as intended. The test bench is configured to determine the output values of a physical process performed on the physical system for input values from a given range in the input space, particularly for simplex, and particularly for a given input value. The test bench is configured to generate new pairs of input and output values as intended. The test bench is configured to detect pairs for expanding a data set, for example, a training data set required to test the technical system 106.
[0027] The apparatus 100 for testing the technical system 106 is configured to perform tests in machine learning (ML) according to, for example, the following procedure.
[0028] 1. A model, e.g., an ML function, is trained based on a measured training data set that includes pairs of input values from the input space and output values from the output space. The model can be represented, for example, by an artificial neural network f or a Gaussian process.
[0029] 2. Test the model based on this method. The output of this method is the probability that the model makes a prediction that deviates significantly from the expected physical output value, and more specifically, the probability for each individual simplex.
[0030] 3. Repeat the test. If the probability for at least one simplex is greater than a given boundary value p*, the testbench requests new training data, particularly input values from simplexes with probabilities p>p*. These new training data are added to the existing training data set, and steps 1 and 2 are repeated.
[0031] 4. Approve the model. The model is approved for reliable prediction of the physical process if the probability for any simplex is not greater than the boundary value p*.
[0032] 5. Suspension Criteria. If step 4 is not achieved, for example, after a predetermined time in the testbench, or if the training data set reaches or exceeds a predetermined amount, the tested model is discarded and another model is tested.
[0033] This alternative model, for example, has an alternative architecture. In the case of a neural network, for example, more neurons are added to the neural network f. In the case of a Gaussian process, for example, another kernel is added to the Gaussian process.
[0034] This alternative model, for example, uses one or more alternative input quantities.
[0035] Next, the procedure is started anew using this different model.
[0036] Figure 2 shows a flowchart illustrating the steps of this method. This method is used in the example of an artificial neural network f and is also applicable to other models, such as models using Gaussian processes.
[0037] The artificial neural network f models the physical process g:X→Y.
[0038] In a physical process, the physical input quantity x ∈ X of a physical process g from the input space X of the physical process yields the physical output quantity y ∈ Y of the physical process in the output space Y of the physical process.
[0039] The input space X contains the input value of the physical input quantity x. The output space Y contains the output value of the physical output quantity y.
[0040] The artificial neural network f maps the input value of a physical input quantity x to a prediction f(x) about the output value of a physical output quantity y.
[0041] The input quantity 108 is an example for the physical input quantity x. The virtual consumption is an example for the physical output quantity y.
[0042] For testing purposes, a surrogate model is provided for the model, i.e., for the artificial neural network f in this example. In this example, the surrogate model is the function h:X→Y.
[0043] The surrogate model h is configured to describe the real physical process g. For example, the function describes the real physical process with tolerances.
number
[0044] This method includes step 202.
[0045] In step 202, the input value is detected.
[0046] This method includes step 204.
[0047] In step 204, the simplex stretched in the input space X is determined by the detected input value.
[0048] The one - dimensional simplex S is spanned, for example, by two detected input values as follows. x(λ ij ) = λ ij x j +(1 + λ ij )x i where λ ij ∈[0,1]
[0049] For the provided surrogate model h, the following holds.
Number
[0050] This means that the surrogate model h describes the real physical process g well inside all simplices, that is, [[ID=k]]
Number
[0051] [[ID=k]] This method includes step 206. <k>
[0052] In step 206, for each simplex, a) at least one input value x k is determined or randomly extracted from the input space, b) the output value h(x k ) for the input value x k is determined using the surrogate model h, c) the input value x k is mapped to a prediction for the output value using the artificial neural network f.
[0053] Each input value x k is determined, for example, in a human - machine interaction with the user.
[0054] Each input value x kFor example, x is randomly drawn on a simplex according to a given probability distribution U(S), particularly according to a uniform distribution. k ~U(S).
[0055] This method includes step 208.
[0056] In step 208, for each prediction, the prediction f(x k ) and the output value h(x) of the surrogate model. k The distance d between ) k This will be decided.
[0057] The distance is determined, for example, by the absolute value of the difference between the prediction and the output value, for example, d k =|f(x k )-h(x k )| is.
[0058] For each simplex, multiple different input values x k It can be assumed that multiple distances to a given point are determined.
[0059] This method includes step 210.
[0060] In step 210, when the artificial neural network f maps the input values of the physical input quantities from the input space to predictions, the predictions are based on a predetermined deviation (for example,
number
[0061] For example, test results include probability.
[0062] The probability p is determined, for example, by the ratio of the number of distances greater than the tolerance to the number of distances determined in the test.
number
[0063] The probability p is determined, for example, using a generalized Pareto distribution with a density function γ, which is fitted to a distance smaller than the tolerance determined for the simplex.
number
[0064] For example, tolerance is deviation
number
number
number
[0065] During the test, it can be assumed that a check is performed to determine whether at least one distance greater than the tolerance has been determined. The probability is determined, for example, by the ratio if at least one distance is greater than the tolerance. The probability is determined, for example, by a generalized Pareto distribution if this is not the case.
[0066] This method assumes that probabilities are determined with the same tolerance for multiple simplexes. For example, probabilities may be determined for multiple simplexes in a time-overlapping manner or simultaneously.
[0067] This method assumes that the probability is determined with different tolerances for at least two simplexes.
[0068] Figure 3 schematically shows an example input space.
[0069] The input space includes the detected input value 302. The detected input value 302 is, for example, the result of measuring the input quantity 108 in machine 110.
[0070] Figure 3 shows simplex 304. In this example, three simplexes are shown.
[0071] In Figure 3, the input values 306 determined or extracted during testing are shown, for example, within one of the multiple simplex 304s.
[0072] Figure 4 shows a schematic diagram of an exemplary one-dimensional simplex stretched by two exemplary input values 302, and an exemplary output value y = f(x) determined using the function f for an exemplary input value x from this one-dimensional simplex. In Figure 4, these output values are denoted by reference numeral 402. In this example, the output value y is the tolerance.
number
number
[0073] Figure 5 schematically shows a histogram of 500, representing the distance between the predictions of the artificial neural network and the output values of the function.
[0074] Beyond the boundary value u, the generalized Pareto distribution 502 is shown. The generalized Pareto distribution 502 is the tolerance determined for the simplex.
number
[0075] One exemplary embodiment relates to a surrogate model h provided and configured to describe a real physical process.
[0076] The surrogate model h, for example, takes the input value x from step 202 in each of the simplex steps. k The measured output value y corresponding to k It is formed by determining the linear interpolation.
[0077] A single or given simplex is input with values x1, x2, ..., x s It is stretched by such that any point x from these simplex x = λ1*x1 + λ2*x2 + ... + λ s *x s Assume that the coefficients λ1, λ2, ..., λ are given. s These are non-negative real numbers that sum to 1. In that case, the surrogate model h is h(x) = λ1*y1 + λ2*y2 + ... + λ s *y s It is defined as follows, where y1, y2, ..., y s The input values are x1, x2, ..., x s This is the physically measured output value corresponding to [the specified value].
[0078] Another surrogate model h, such as a Gaussian process or a physical model, can also be used.
Claims
1. Technical systems (106), in particular a computer-implemented method for testing a system for determining virtual sensor signals of, for example, a virtual sensor, The aforementioned system includes a model that models physical processes, In the aforementioned physical process, the physical input amount of the physical process yields the physical output amount of the physical process. The input space includes the input value of the physical input quantity, and the output space includes the output value of the physical output quantity. A substitute model for the aforementioned model is provided. The aforementioned surrogate model is configured to describe real physical processes, The input value (302) was detected (202), In the input space, the simplex (304) stretched by the detected input value (302) is determined (204), For each simplex, at least one input value (306) from the input space is determined in particular in human-machine interaction with a user, or randomly selected on the simplex according to a predetermined probability distribution, particularly according to a uniform distribution, and the output value for the input value is determined using the surrogate model. In particular, in order to determine the virtual sensor signal, the input value is mapped to a prediction of the output value using the model (206), The distance between the prediction and the output value is determined (208), When the model maps the input values of the physical input quantities from the input space to predictions, the probability that the prediction deviates from the physical output values of the physical process with respect to the physical input quantities having the input values by more than a predetermined deviation is included in the test results (210). A method characterized by the following:
2. The distance is determined depending on the absolute difference between the prediction and the output value. The method according to claim 1.
3. The aforementioned probability is determined by the ratio of the number of distances greater than the tolerance to the number of distances determined during the test (210). The method according to claim 1 or 2.
4. The aforementioned probability is determined using a generalized Pareto distribution (502) (210), The generalized Pareto distribution (502) is fitted to a distance smaller than the tolerance determined for the simplex. The method according to any one of claims 1 to 3.
5. During the aforementioned test, it is checked whether at least one distance greater than the tolerance has been determined (210), If at least one distance is greater than the tolerance, the probability is determined depending on the ratio. Otherwise, the probability is determined using the generalized Pareto distribution. The method according to claims 3 and 4.
6. The aforementioned proxy model represents the actual physical process, tolerance [Math 1] It is structured to be described using the following: The aforementioned tolerance depends on the acceptable deviation, and in particular as part of the deviation, for example, half of the deviation. [Math 2] It is determined as (210), The method according to claim 3, 4, or 5.
7. The aforementioned probabilities are determined with the same tolerance for multiple simplexes, in particular, so as to overlap in time or be determined simultaneously (210), or The aforementioned probability is determined with different tolerances for at least two simplexes (210). The method according to any one of claims 3 to 6.
8. For each simplex, for example, multiple distances are determined for multiple different input values (208). The method according to any one of claims 1 to 7.
9. The model is trained based on a measured set of training data, the set of training data includes pairs of input values from the input space and output values from the output space, If the probability for at least one simplex is greater than a predetermined boundary value, new training data having input values from simplexes having probabilities greater than the boundary value is measured and added to the training data set, and the model is trained based on the training data set supplemented by the training data, or If the probability for any simplex is not greater than the predetermined boundary value, the model is approved for reliable prediction of the physical process. The method according to any one of claims 1 to 8.
10. A technical system (106), in particular an apparatus (100) for testing a system for determining virtual sensor signals of, for example, a virtual sensor, The aforementioned device (100) At least one processor (102), At least one memory (104), Includes, The at least one processor (102) is configured to execute instructions stored in the memory (104), When the instruction is executed by the at least one processor (102), the apparatus (100) implements the method according to any one of claims 1 to 9. Apparatus (100) characterized by the following.
11. It is a computer program, The computer program includes a computer executable instruction, and when the computer executable instruction is executed by a computer, the method according to any one of claims 1 to 9 is performed. A computer program characterized by the following features.