VERIFICATION OF A DATA SET FOR USE IN A COMPUTER-BASED MACHINE LEARNING SYSTEM FOR A TECHNICAL SYSTEM, IN PARTICULAR FOR A VEHICLE

Abstract

One aspect of the present disclosure relates to a computer-implemented method for checking a data set containing measurement data points for use in a computer-based machine learning system deployed in a technical system. The method aims to classify the data set as suitable or unsuitable for subsequent training and / or validation of the computer-based machine learning system based on a predefined criterion.

Description

Technical field

[0001] The present invention relates to techniques for verifying a data set containing measurement data points for use in a computer-based machine learning system deployed in a technical system. Related aspects include a computer-implemented method for training and / or validating a computer-based machine learning system used in a technical system, a computer-implemented method for applying a computer-based machine learning system, a computer program, a computer system, a computer-readable medium or signal, and a sensor system. background

[0002] Sensor systems (e.g., based on cameras, radar, lidar, or other sensor systems) are increasingly used in various technical systems (e.g., in vehicles) to enable the execution of functions within these systems (e.g., in connection with the control, regulation, or monitoring of the technical system or a part thereof). To simplify the design of a technical system, optimize its size, and / or reduce subsequent costs associated with the cost and integration of a sensor system into a technical system, it can, in some cases, be advantageous to replace some (real) sensor systems with corresponding computer-based machine learning systems (so-called virtual sensors) that are, for example, capable of generating data of interest, at least partially, from the measurements of other sensors (i.e.,to calculate) and thus provide the necessary data (which may be required for the executability of a function) instead of the measurement data from the missing (real) sensors.

[0003] However, the use of such machine learning systems (e.g., those that use artificial neural networks) instead of sensor systems may be limited (especially in safety-critical applications, e.g., in connection with autonomous or assisted driving), because if the machine learning system operates on different datasets than those on which it was trained and / or validated, some of the existing state-of-the-art methods may produce results with insufficient accuracy or even unexpected (e.g., incorrect) results.

[0004] The reason for this may be that this training and / or validation dataset, which contains, for example, measurement data from a sensor system (e.g., a test sensor system) that is to be replaced by a corresponding computer-based machine learning system, contains measurement data that a priori cannot guarantee the required accuracy for the computer-based machine learning system after it is trained and / or validated with this dataset (e.g., because the measurement data was not generated by the sensor with sufficient accuracy and / or because the measurement data contains a significant amount of measurement data that are statistically identical and only differ from each other due to measurement noise).

[0005] Therefore, there is a need to develop new techniques for computer-based machine learning systems that can solve some or all of the problems mentioned above. Summary of the invention

[0006] A first general aspect of the present disclosure relates to a computer-implemented method for verifying a data set containing measurement data points for use in a computer-based machine learning system deployed in a technical system. The method comprises receiving a data set containing a plurality of data points, wherein each data point comprises an input element and a corresponding output element for the computer-based machine learning system, the input element and the output element of the data point representing a measurement relating to the technical system. Furthermore, the method comprises varying an input element of a first data point of the data set within a predetermined range, thereby generating a plurality of varied input elements for the first data point.Furthermore, the first aspect involves determining a characteristic pairwise error between an output element of the first data point and an output element of a second data point in the dataset that differs from the first data point. This determination of the characteristic pairwise error is based on i) the multitude of varied input elements for the first data point, ii) an input deviation between an input element of the second data point and the input element of the first data point, and iii) an output deviation between the output element of the second data point and the output element of the first data point.Finally, the procedure includes classifying the data set as suitable for subsequent training and / or validation of the computer-based machine learning system if the characteristic pairwise error meets a predefined criterion, and otherwise, if the characteristic pairwise error does not meet the predefined criterion, classifying the data set as unsuitable for subsequent training and / or validation of the computer-based machine learning system.

[0007] A second general aspect of the present disclosure relates to a computer-implemented method for training and / or validating a computer-based machine learning system used in a technical system. The method of the second aspect comprises receiving a data set that, according to the first general aspect, has been classified as suitable for subsequent training and / or validation of the computer-based machine learning system. Furthermore, the method comprises training and / or validating the computer-based machine learning system with the received data set in order to obtain a trained and / or validated computer-based machine learning system.

[0008] A third general aspect of the present disclosure relates to a computer-implemented method for applying a computer-based machine learning system. The method of the third aspect comprises receiving a trained and / or validated computer-based machine learning system according to the second general aspect. Furthermore, the method comprises processing application data by the received computer-based machine learning system.

[0009] A fourth general aspect of the present disclosure relates to a computer program designed to execute the computer-implemented methods according to one of the first to third general aspects.

[0010] A fifth general aspect of the present disclosure relates to a computer system designed to execute the computer-implemented methods according to one of the first to third general aspects and / or the computer program according to the fourth aspect.

[0011] A sixth general aspect of the present disclosure relates to a computer-readable medium or signal that stores and / or contains the computer program according to the fourth general aspect.

[0012] A seventh general aspect of the present disclosure relates to a sensor system of a technical system designed to generate at least partially a data set of measurement data points that is suitable for being classified as suitable or unsuitable for subsequent training and / or validation of the computer-based machine learning system in the process according to the first general aspect.

[0013] The techniques described in the first through seventh general aspects may have one or more of the following advantages.

[0014] Firstly, the techniques of the present disclosure can be used to check whether a data set with measurement data points (e.g. generated by a sensor system) is suitable (or not) for training and / or validating the computer-based machine learning system, faster and more efficiently than in some cases of the prior art.

[0015] In particular, unlike some state-of-the-art techniques, this decision can essentially be made in a single computational step without any computational effort (e.g., by appropriately varying the data points of the data set and determining the resulting characteristic pairwise errors; more on this below).

[0016] Secondly, unlike some prior art techniques, the techniques presented here offer a more efficient way to estimate the accuracy that a computer-based machine learning system will achieve when trained and / or validated with this dataset of measurement data points. In some cases, this eliminates the need for unnecessary tests to improve the machine learning system (e.g., by varying its hyperparameters), since the accuracy of the machine learning system (in other words, its performance) cannot fall below a required threshold according to the predictions of the available techniques anyway.

[0017] Thirdly, based on knowledge of the estimated accuracy of the computer-based machine learning system that is trained and / or validated with the data set of measurement data points, the techniques presented can provide a meaningful assessment of whether this machine learning system has sufficient accuracy to replace a given sensor system.

[0018] Some terms are used in the present revelation in the following way: In the present disclosure, “a technical system” can be a vehicle, a robot, a household appliance (e.g., a refrigerator), or a medical device. It is also conceivable that one or more components of a vehicle (i.e., vehicle components), a robot, a household appliance, or a medical device can be considered "a technical system".

[0019] In this disclosure, the term "dataset with measurement data points" for use in a computer-based machine learning system can be understood as a collection of data associated with real-world measurements (in other words, observations of the real world). (For the sake of simplicity, the term "data points" will be used in some instances below instead of "measurement data points.") The "dataset with measurement data points" can be generated by a "sensor system" of a "technical system" (e.g., an internal or external sensor system). In some examples, the data points can then be processed to obtain a suitable format for feeding into the computer-based machine learning system. In some instances, the data points of the "dataset with measurement data points" will be referred to below as "input" or "process" data points.The term "output elements" refers to input and output elements, which can each form a pair so that these pairs of input and output elements can be used for training and / or validation in the machine learning system. (In this context, the output elements can sometimes be referred to as target variables.) Furthermore, in some examples, other data can also be used alongside the data points of the "measurement data point set" for training and / or validation in the machine learning system: for example, measurement data points generated by a different sensor in the technical system than the sensor used to generate the "measurement data point set."

[0020] Depending on the nature of the task for which a computer-based machine learning system is used, the data points of a dataset containing measurement data points can contain different types of information. For example, each data point in an image dataset can contain a single image (or a section thereof) or a video. An image element might contain a number of pixels (e.g., 1024x2048 pixels), with each pixel having a number of color values ​​(e.g., three color values ​​with 16-bit color depth). In some cases, the images may contain features that are processed by a computer-based machine learning system. In the example of a vehicle-integrated sensor system (in the case of a vehicle or vehicle component, which is considered a "technical system"), features in the images related to street scenes with one or more objects, such as...Traffic signs, lane markings or other road or pedestrian zone markings, trees, buildings, and road users such as pedestrians, cyclists, or other vehicles are conceivable examples. In other examples, "data points" of a data set can contain data series (e.g., time series). These data series (e.g., time series) can be generated using various sensor systems (e.g., cameras, radar, lidar, ultrasonic or thermal sensors, sensor systems for a vehicle's engine control unit).

[0021] In this disclosure, the term "computer-based machine learning system" means any device that can be trained using machine learning to perform one or more tasks. During training, training data sets, such as one or more (e.g., all) data points of the "data set with measurement data points" defined above (under the conditions introduced below), can be supplied as input data to the computer-based machine learning system. The properties (e.g., corresponding parameters) of the computer-based machine learning system can be adapted in response to the processing of the training data sets (e.g., by analyzing the "output elements") to solve the one or more tasks in a defined manner (e.g., with a certain accuracy). A computer-based machine learning system may contain a model, which may be parameterized.Adapting the properties of the machine learning system during learning can be achieved, for example, through an optimization procedure with respect to (unknown) parameters of the machine learning system (e.g., a corresponding model), which can be represented as the minimization of a loss function (within a predetermined numerical accuracy and / or until a predetermined termination criterion is reached). Subsequently, the adapted computer-based machine learning system can be used to predict the responses to the observations in another dataset, the so-called validation dataset. Thus, the validation dataset can provide an evaluation of the machine learning system's adaptation to the training datasets. In the present disclosure, in some cases, one or more data points from the "dataset of measurement data points" can be used to validate the computer-based machine learning system; more on this below.

[0022] Within the scope of the present disclosure, it may be conceivable that a trained “computer-based machine learning system” trained on the basis of a “data set of measurement data points” (or part thereof) can be used in a “technical system” instead of a sensor system that (at least partially) generated this “data set of measurement data points”.

[0023] In some cases, a computer-based machine learning system can perform a classification or regression task. In a non-restrictive example, the computer-based machine learning system (e.g., its model) can comprise an artificial neural network with a specific topology and a number of neurons with corresponding connections. According to some embodiments, the neural network can be a convolutional neural network (CNN), defined, for example, by the number of filters, filter sizes, step sizes, and so on. A convolutional neural network can be used, for example, for image classification and perform one or more transformations on digital images based, for example, on convolution, nonlinearity (ReLU), pooling, or classification operations (e.g., using fully connected layers).The neural network can also be designed as a multi-layered feedforward or recurrent network, as a neural network with direct or indirect feedback, or as a multi-layered perceptron. Machine learning systems based on neural networks can be used in a "technical system," for example, in a vehicle computer or another component of a vehicle, or in a semi-autonomous robot (e.g., to evaluate the operating state of the vehicle or robot and / or to control (or regulate) the vehicle, a function of the vehicle or robot based on state and / or environmental data of the vehicle or robot as input data). For example, computer-based machine learning systems can be implemented in any suitable form, i.e., in software, in dedicated hardware, or in a hybrid form of software and dedicated hardware.Therefore, computer-based machine learning systems can be a software module (even integrated into a higher-level software system) that can be executed on a general-purpose processor. In other cases, a computer-based machine learning system can be implemented (at least partially) using circuitry.

[0024] In some examples, a "sensor system" of a "technical system" can be used to provide at least a partial function for that "technical system" (e.g., a function related to the control and / or regulation of the "technical system"). In this context, a "function" can be any task (or subtask) performed during the operation of the "technical system." A function can involve the control, regulation, or monitoring of the device or a part of the device (e.g., a component of the device). Additionally or alternatively, a function can involve data or signal processing within the "technical system" (e.g., a communication function). For example, a vehicle's sensor system can be used in the context of semi-autonomous, autonomous, or assisted driving to provide functionality for the vehicle (e.g.,...in terms of safety features, the implementation of driver information and comfort features, such as intelligent headlight control and traffic sign information).

[0025] The term "sensor system" encompasses any system that can contain one or more sensors, such as cameras, radar, lidar, ultrasonic, or thermal sensors, as modules. A non-exhaustive list of other sensor system modules includes CPU cores, system-on-a-chip (SoC) hardware, RAM, hard drives, processing modules (e.g., image processing modules), and communication interfaces to other devices. The sensors mentioned above can detect their environment and generate data, for example, in the form of images or data sequences. A sensor system can be used, for example, in the context of semi-autonomous, autonomous, or assisted driving to provide functionality for the vehicle (e.g., regarding safety functions, the provision of driver information, and comfort features such as intelligent headlight control and traffic sign recognition).In other examples, it is conceivable that the sensor system can be used for a monitoring task (e.g., a manufacturing process and / or for quality assurance) (e.g., to monitor the environment of an at least semi-autonomous robot).

[0026] A "vehicle" can be any device that transports passengers and / or cargo. A vehicle can be a motor vehicle (for example, a car or a truck), but also a rail vehicle. A vehicle can also be a motorized two- or three-wheeler. However, floating and flying devices can also be vehicles. Vehicles can operate autonomously, semi-autonomously, or with assistance. Brief description of the characters Fig. Figure 1a is a flowchart that illustrates an example of a procedure for checking a data set with measurement data points for use in a computer-based machine learning system, according to the first aspect. Fig. 1b is a flowchart showing further possible process steps according to the first aspect. Fig. Figure 2 schematically shows an illustrative example of the construction of a linear interpolation model for calculating pairwise errors 20 (Δz ik ) between an initial scalar 3 (z i ) of the first data point [1, 3] ([v i , z i ]) and an initial scalar 4 (z k ) of the second data point [2, 4] ([v k , z k ]). In this example, an input element of the first data point is a first two-dimensional vector 1 (v i = [v i,1 v i,2 ] T) and an input element of the second data point, a second two-dimensional vector 2 (v k = [v k,1 v k,2 ] T ).

[0027] Fig. Figure 2 shows, by way of example, how a length 10 (Δa) of a variation 11* (Δv) of the varied two-dimensional vector 1* (vi*) to a pairwise error 20 (Δz ik ) can lead to. Other designations in Fig. 2: Difference 43 between the output scalars 4 and 3; input absolute value 21 of a subtraction between the vectors 2 and 1. Detailed description

[0028] First, using Fig. 1a and Fig. 1b Techniques for checking a dataset with measurement data points for use in a computer-based machine learning system are described. Then, an exemplary construction of a linear interpolation model is presented based on… Fig. 2 discussed.

[0029] As in the Fig. 1a and Fig. As outlined in 1c, a first general aspect concerns a computer-implemented method for checking a dataset with measurement data points for use in a computer-based machine learning system (e.g., with regard to possible subsequent training and / or validation using this dataset in the machine learning system, see further discussions) that is used in a technical system (e.g., in a vehicle or in another technical system mentioned above). The method steps of the corresponding independent claim are shown in the boxes drawn with solid lines in Fig. 1a and Fig. 1b is shown, while the process steps of some dependent claims are shown in the boxes represented by dashed lines.

[0030] The first step of the process involves receiving a dataset containing a multitude of data points. Each data point (e.g., each data point in the multitude) comprises an input element and a corresponding output element for the computer-based machine learning system. The input and output elements of the data point represent a measurement related to the technical system. For example, each data point in the multitude can represent a specific measurement of the technical system. These measurements can be generated, for example, by a sensor system, as discussed in detail above.

[0031] Furthermore, the techniques presented here include varying (200) an input element 1 (v i ) of a first data point [1, 3] ([v i ,z i]) of the data set within a given range (W), resulting in a variety of varied input elements 1* (vi*) is created for the first data point.

[0032] (The reference symbols refer to a non-restrictive example found in Fig. 2 is shown and explained in more detail below.)

[0033] Furthermore, the procedure of the first aspect includes determining 300 a characteristic pairwise error (r). i,k ) between an output element 3 (z i ) of the first data point and an output element 4 (z k ) of a second data point [2, 4] ([v k , z k ]) of the data set that differs from the first data point. Furthermore, the characteristic pairwise error is determined based on i) the multitude of varied input elements 1* (vi*) for the first data point, ii) an input deviation 21 between an input element 2 (v k ) of the second data point and the input element 1 (v i ) of the first data point, and iii) an output deviation 43 between the output element 4 (z k ) of the second data point and the output element 3(z i ) of the first data point.

[0034] (The first and second data points are named as such for further discussion - it is conceivable that any pair of data points from the dataset could be selected as the first and second data points.)

[0035] In the present revelation, the input element 1 (v i ) of the first data point [1,3] ([v i ,z i ]) an input scalar or an input vector of the first data point and the output element 3 (z i) of the first data point can be an output scalar of the first data point. Accordingly, the input element 2 (v k ) of the second data point an input scalar or an input vector of the second data point [2, 4] ([v k , z k ]) and the output element 4 (z k The second data point's input scalars can be an output scalar of the second data point. In one of the simplest, non-restrictive examples, the input scalars of the data points can be vehicle speeds, while the output scalars of the data points can be aerodynamic drag forces corresponding to those speeds.

[0036] In the example of Fig. 2 is an input element of the first data point [1, 3] ([v i , z i ]) as the first two-dimensional vector 1 (v i = [v i,1 v i,2 ] T ) and an input element of the second data point [2, 4] ([v k , z k]) as the second two-dimensional vector 2 (v k = [v k,1 v k,2 ] T ) outlined. Furthermore, the initial scalar 3 (z i ) of the first data point and the output scalar 4 (z k ) of the second data point [2, 4] ([v k , z k ]) is represented. This figure is also a variation 11* (Δv) of the varied two-dimensional vector 1*. (vi*) to be taken.

[0037] In the present techniques, the procedure step “Determine 300” of the characteristic pairwise error (r) can be i,k ) further calculating 310 of a multitude of pairwise errors 20 (Δz ik ) between the initial scalar 3 (z i ) of the first data point [1, 3] ([v i ,z i ]) and the initial scalar 4 (z k ) of the second data point [2, 4] ([v k , z k ]) include. Each pairwise error can be 20 (Δz ik) given by: i) a length 10 (Δa) of a variation 11* (Δv) of a corresponding varied input scalar or input vector 1* (vi*) of the first data point, ii) a difference value 43 between the output scalar 4 (z k ) of the second data point and the output scalar 3 (z i ) of the first data point, and iii) an input absolute value of a subtraction between the input scalar of the second data point and the input scalar of the first data point, or between the input vector of the second data point and the input vector of the first data point. (Note that the subtraction of input scalars or input vectors depends on whether the input elements were selected as input scalars or input vectors.)

[0038] Here, the difference value of the (above-defined) output deviation 43 between the output element 4 (z) can be calculated. k) of the second data point and the output element 3(z i ) of the first data point. For example, the difference value 43 between the output scalar of the second data point and the output scalar of the first data point can be given by the following expression: z k - z i Furthermore, the input absolute value can be assigned an absolute value of the (above-defined) input deviation 21 between the input element 2 (v k ) of the second data point and the input element (1; v i ) of the first data point. For example, the absolute input value 21 can be determined by subtracting the input vector of the second data point from the input vector of the first data point as follows: |v k - v i |

[0039] In some cases, the pairwise errors can be 20 (Δz ik ) as a linear interpolation between the output scalar 3 (z i) of the first data point [1, 3] ([v i , z i ]) and the initial scalar 4 (z k ) of the second data point [2, 4] ([v k , z k ]) can be given. In an example, the pairwise errors 20 (Δz) can be given. ik ) define as follows: Δzik=Δa|vk−vi|⋅(zk−zi), where (in accordance with the definitions above) Δa is the length of the variation 11* (Δv) of the varied input vector 1* (vi*) of the first data point is, |v k - v i | the input absolute value 21 of the subtraction between the input vector of the second data point and the input vector of the first data point is and z k - z i The difference value is 43 between the output scalar of the second data point and the output scalar of the first data point. In some cases, the pairwise error can be 20 (Δz). ik ) should be written as follows: Δzik=ΔvT⋅(vk−vi)|vk−vi|⋅(zk−zi), where (in accordance with the definitions above) Δv is the variation of the varied input vector 1* (vi*) of the first data point, T denotes the transposition operation and v k - v i The subtraction is between the input vector of the second data point and the input vector of the first data point.

[0040] In other cases, it is also conceivable that the pairwise errors 20 (Δz ik ) as a nonlinear interpolation between the output scalar 3 (z i ) of the first data point [1, 3] ([v i , z i ]) and the initial scalar 4 (z k ) of the second data point [2, 4] ([v k , z k ]) can be given.

[0041] Subsequently, the procedure step “Determine 300” of the characteristic pairwise error (r) can be performed. i,k) The selection (320) of a pairwise error from the plurality of pairwise errors as the characteristic pairwise error includes, wherein the selected pairwise error satisfies a predefined characteristic criterion. In an example, the predefined characteristic criterion may include that the selected pairwise error has a maximum absolute value of all pairwise errors from the plurality of pairwise errors. In this case, the characteristic pairwise error may, for example, be defined as follows: r i,k = max (|Δz ik |). In another example, the predefined characteristic criterion can include the requirement that the selected pairwise error has a minimum and a maximum value among all pairwise errors from the set of pairwise errors. In this case, the characteristic pairwise error can be defined, for example, as follows: r i,k = min (Δz ik ) together with r i,k= max (Lz ik ).

[0042] In the present techniques, the procedure step "Calculate 310" of the multitude of pairwise errors between the output scalar of the first data point and the output scalar of the second data point can only be performed if the input absolute value 21 (e.g. |v) k - v i |) a first input value criterion is met. In some cases, the first input value criterion may include the input absolute value being greater than a predefined first threshold for the input absolute value. For example, the first input value criterion can be defined as follows: |v k - v i | > d v , where d v the specified first threshold. In some cases, fulfilling the first input amount criterion can exclude a situation in which the first [1, 3] ([v i ,z i ]) and the second data point [2, 4] ([vk , z k ]) statistically speaking, these are the same (measured) data points, differing only due to different noise levels (during these measurements).

[0043] In some cases of the techniques presented, the procedure step “Calculate 310” can then be performed to handle the multitude of pairwise errors between the output scalar of the first data point and the output scalar of the second data point, if the input absolute value 21 (e.g. |v) k - v i |) the first input amount criterion is met, and, if the input absolute value does not meet the first input amount criterion, are only executed if the input absolute value meets a second input amount criterion and an output absolute value (e.g. |z) k - z i|) between the output scalar of the first data point and the output scalar of the second data point, an output magnitude criterion is satisfied. For example, the second input magnitude criterion may include that the input absolute is greater than a predefined second threshold for the input absolute, which is less than the predefined first threshold, and where the output magnitude criterion may include that the output absolute is greater than or equal to the predefined threshold for the output absolute. In an example, the first input magnitude criterion can be defined as follows: |v k - v i | > ε, where ε is the given second threshold, ε < d v In this case, the initial amount criterion can be defined as follows: |z k - z i | > d z .

[0044] Furthermore, in some cases the present techniques can be used when the input element (v i) of the first data point [1,3] ([v i ,z i |) the input vector is the length (Δa) of the variation (Δv) of the corresponding varied input vector of the first data point ([v i ; z i ]) by a projection of the variation (Δv) of the corresponding varied input vector of the first data point ([v i ; z i ]) be given a difference between the input vector of the second data point and the input vector of the first data point.

[0045] If in the present disclosure the input element 1 (v i ) of the first data point [1,3] ([v i , z iIf the input scalar of the first data point is ]), the specified range (W) can be an interval containing the variation of the corresponding varied input scalar of the first data point. In some cases, the interval can be defined as follows: W ∈ [-w0,w0] with w0 > 0. Otherwise, if the input element 1 (v) is i ) of the first data point [1,3] ([v i , z i If the input vector of the first data point is , the specified range (W) can comprise a plurality of intervals, where the variation of each component of the corresponding varied input vector of the first data point lies in an interval of the plurality of intervals associated with that component, and where a number of intervals of the plurality of intervals is a dimension of the input vector of the first data point. In some cases, the number of intervals can be defined as follows: W ∈ ([-w 10 , w 10 ], [-w 20 , w20 ], ..., [-w m0 , w m0 ]), mit w i0 > 0 (i = 1, ..., m), where m corresponds to the dimension of the input vector of the first data point. In some cases, the variation of the corresponding varied input scalar or the variation of each component of the corresponding varied input vector in the respective interval can be generated using a random number generator. Within the scope of this disclosure, it is conceivable that the values ​​w0 and / or w i0 (i = 1, ..., m) can be selected on the basis of a physical process that is linked to the measurements (as explained above) (which are generated, for example, by a sensor system).

[0046] Finally, the procedure includes classifying the dataset as suitable for subsequent training and / or validation of the computer-based machine learning system if the characteristic pairwise error satisfies a predefined criterion. Otherwise, if the characteristic pairwise error does not satisfy the predefined criterion, the procedure includes classifying the dataset as unsuitable for subsequent training and / or validation of the computer-based machine learning system. In the techniques of the present disclosure, the predefined criterion may include the absolute value of the characteristic pairwise error falling below a predefined threshold. In this case, the predefined criterion may be defined in some examples as follows: |r i,k | < r th , where r thThe predefined threshold is a specific value. In another case, the predefined criterion may include the characteristic pairwise error falling below a predefined maximum threshold and / or exceeding a predefined minimum threshold. In this case, the predefined criterion may be defined as follows in some examples: r i,k < r th,max , r i,k > r th,min oder r th,min < r i,k < r th,max , where r th,max der vordefinierte maximale Schwellenwert und r th,min der vordefinierte minimale Schwellenwert ist.

[0047] In the present disclosure, the process step “Determine 300” of the characteristic pairwise error (r) i,kThe procedure step 330 further includes selecting two or more data points from the plurality of data points. Furthermore, the "Determine" step 300 of the characteristic pairwise error can include selecting a reference data point from the two or more data points, where the reference data point corresponds to the first data point and another data point from the two or more data points corresponds to the second data point. The plurality of data points can, for example, comprise N data points, where N equals two or more, equals 10 or more, equals 50 or more, equals 10 2 or more, like 10 3 or more, like 10 4 or more. The two or more points can, for example, comprise M data points, where M ≤ N. Furthermore, the procedure step "Determine 300" of the characteristic pairwise error (r) i,k ) determining 350 of the characteristic pairwise error (r) k,i) with respect to the reference data point using any other data point of the two or more data points, resulting in a variety of characteristic pairwise errors (r) i,k ) is formed for the reference data point. In some cases, the multitude of characteristic pairwise errors for a reference data point i of two or more data points, i ∈ [1, ...,M], can be written as follows: R i = [r i,1 ,r i,2 , ...,r i,i-1 , r i,i+1 , r i,i+2 , ..., r i,M ].

[0048] Furthermore, the procedure step "Select 340" of the reference data point of the two or more data points can include the successive selection 360 of one or more data points (e.g., each data point) of the two or more data points as the reference data point, thereby determining a multitude of characteristic pairwise errors with respect to each data point of the one or more data points of the two or more data points. In some cases, this can mean that the R i for one or more data points i from [1, ..., M], e.g. for each data point i ∈ [1, ..., M], can be determined.

[0049] In accordance with the above discussions, it is also conceivable within the scope of the present disclosure that the data set can be classified as suitable for subsequent training and / or validation of the computer-based machine learning system if the multitude of characteristic pairwise errors (R) i) with respect to each data point of the one or more data points of the two or more data points, the predefined criterion is met.

[0050] A second general aspect of the present disclosure relates to a computer-implemented method for training and / or validating a computer-based machine learning system used in a technical system. The method of the second aspect comprises receiving a data set that, according to the first general aspect, has been classified as suitable for subsequent training and / or validation of the computer-based machine learning system. Furthermore, the method comprises training and / or validating the computer-based machine learning system with the received data set in order to obtain a trained and / or validated computer-based machine learning system.

[0051] A third general aspect of the present disclosure relates to a computer-implemented method for applying a computer-based machine learning system. The method of the third aspect comprises receiving a trained and / or validated computer-based machine learning system according to the second general aspect. Furthermore, the method comprises processing application data by the received computer-based machine learning system.

[0052] As discussed in detail above, the computer-based machine learning system can be used for a variety of applications (e.g., for providing autonomous or assisted driving functions, or other applications mentioned above). Furthermore, in some cases, the computer-based machine learning system (which, according to the second aspect, has been trained and / or validated using the appropriate dataset) can be used in the technical system instead of a sensor system that (at least partially) generated the dataset with measurement data points.

[0053] A fourth general aspect of the present disclosure relates to a computer program designed to execute the computer-implemented methods according to one of the first to third general aspects.

[0054] A fifth general aspect of this disclosure relates to a computer system designed to execute the computer-implemented methods according to any one of the first three general aspects and / or the computer program according to the fourth aspect. The computer system may include at least one processor, at least one memory (which may contain programs that, when executed, perform the methods of this disclosure), and at least one interface for inputs and outputs. The computer system may be a stand-alone system or a distributed system that communicates via a network (e.g., the Internet).

[0055] A sixth general aspect of the present disclosure relates to a computer-readable medium (e.g., a machine-readable storage medium such as an optical storage medium or solid-state storage, e.g., FLASH memory) or signal that stores and / or contains the computer program according to the fourth general aspect.

[0056] A seventh general aspect of the present disclosure relates to a sensor system (see above definitions) of a technical system designed to generate at least partially a data set of measurement data points suitable for being classified as suitable or unsuitable for subsequent training and / or validation of the computer-based machine learning system in the procedure according to the first general aspect.

Claims

[1] Computer-implemented method for checking a data set of measurement data points for use in a computer-based machine learning system deployed in a technical system, wherein the method comprises the following steps: - Receiving (100) a data set containing a plurality of data points, wherein a data point comprises an input element and an output element corresponding to that input element for the computer-based machine learning system, wherein the input element and the output element of the data point represent a measurement relating to the technical system; - Varying (200) an input element (1; v i ) of a first data point ([1, 3]; [v i , z i ]) of the data set within a given range (W), resulting in a variety of varied input elements (1*;vi*) is formed for the first data point; - Determining (300) a characteristic pairwise error (r i,k ) between an output element (3; z i ) of the first data point and an output element (4; z k ) of a second data point ([2,4]; [v k , z k ]) of the data set that differs from the first data point, where determining the characteristic pairwise error is based on i) the multitude of varied input elements (1*;vi*) for the first data point, ii) an input deviation (21) between an input element (2; v k ) of the second data point and the input element (1, v i ) of the first data point, and iii) an output deviation (43) between the output element (4; z k ) of the second data point and the output element (3; z i ) of the first data point; and - Classifying (400) the data set as suitable for subsequent training and / or validation of the computer-based machine learning system if the characteristic pairwise error meets a predefined criterion, and otherwise, if the characteristic pairwise error does not meet the predefined criterion, classifying (420) the data set as not suitable for subsequent training and / or validation of the computer-based machine learning system. [2] Method according to claim 1, wherein the determination of the characteristic pairwise error (r i,k ) further includes: - Select (330) two or more data points from the multitude of data points; - Selecting (340) a reference data point from the two or more data points, wherein the reference data point corresponds to the first data point and another data point from the two or more data points corresponds to the second data point; and - Determining (350) the characteristic pairwise error (r) k,i ) with respect to the reference data point using any other data point of the two or more data points, resulting in a variety of characteristic pairwise errors (r) i,k ) is formed for the reference data point. [3] Method according to claim 2, wherein selecting the reference data point of the two or more data points comprises: - successive selection (360) of one or more data points of the two or more data points as the reference data point, thereby determining a variety of characteristic pairwise errors with respect to each data point of the one or more data points of the two or more data points. [4] Method according to any one of the preceding claims 1 to 3, wherein the input element (1, v i ) of the first data point ([1,3]; [v i ,z i]) is an input scalar or an input vector of the first data point and the output element (3, z i ) of the first data point is an output scalar of the first data point; and where the input element (2, v k ) of the second data point an input scalar or an input vector of the second data point ([2,4]; [v k ,z k ]) is and the output element (4; z k ) of the second data point is an output scalar of the second data point. [5] The method of claim 4, wherein the predefined criterion comprises that the absolute value of the characteristic pairwise error falls below a predefined threshold; or the characteristic pairwise error falls below a predefined maximum threshold and / or is above a predefined minimum threshold. [6] Method according to claim 4 or 5, wherein the determination of the characteristic pairwise error (ri,k ) further includes: - Calculating (310) a multitude of pairwise errors (20; Δz ik ) between the initial scalar (3; z i ) of the first data point ([1, 3]; [v i ,z i ]) and the initial scalar (4; z k ) of the second data point ([2,4]; [v k ,z k ]), where each pairwise error is given by: i) a length (10; Δa) of a variation (11*; Δv) of a corresponding varied input scalar or input vector (1*;vi*) of the first data point, ii) a difference value (43) between the initial scalar (4; z k ) of the second data point and the output scalar (3; z i ) of the first data point, and iii) an input absolute value (21) of a subtraction between the input scalar of the second data point and the input scalar of the first data point or between the input vector of the second data point and the input vector of the first data point, where the difference value of the output deviation (43) between the output element (4; z k ) of the second data point and the output element (3; z i ) of the first data point corresponds to, and wherein the input absolute value is an absolute value of the input deviation (21) between the input element (2; v k ) of the second data point and the input element (1; v i ) of the first data point; and - Selecting (320) one pairwise fault from the multitude of pairwise faults as the characteristic pairwise fault, wherein the selected pairwise fault satisfies a predefined characteristic criterion. [7] Method according to claim 6, wherein the calculation of the plurality of pairwise errors between the output scalar of the first data point and the output scalar of the second data point is performed only if the input absolute value (21) satisfies a first input magnitude criterion; or is performed if the input absolute value (21) satisfies the first input magnitude criterion, and, if the input absolute value does not satisfie the first input magnitude criterion, is performed only if the input absolute value satisfies a second input magnitude criterion and an output absolute value (d z ) an output magnitude criterion is met between the output scalar of the first data point and the output scalar of the second data point. [8] Method according to claim 7, wherein the first input value criterion comprises that the input absolute value is greater than a predetermined first threshold for the input absolute value; wherein the second input value criterion comprises that the input absolute value is greater than a predetermined second threshold for the input absolute value, which is less than the predetermined first threshold value; and wherein the output value criterion comprises that the output absolute value is greater than a predetermined threshold for the output absolute value. [9] Method according to any one of the preceding claims 6 to 8, wherein, when the input element (v i ) of the first data point ([1,3]; [v i , z i ]) the input vector is the length (Δa) of the variation (Δv) of the corresponding varied input vector of the first data point ([v i ; z i]) by a projection of the variation (Δv) of the corresponding varied input vector of the first data point ([v i ; z i ]) is given as a difference between the input vector of the second data point and the input vector of the first data point. [10] Method according to any one of the preceding claims 6 to 9, wherein, when the input element (1; v i ) of the first data point ([1, 3]; [v i , z i ]) the input scalar of the first data point, the specified range (W) is an interval in which the variation of the corresponding varied input scalar of the first data point lies; or, if the input element (1; v i ) of the first data point ([1,3]; [v i ,z i]) the input vector of the first data point, the specified range (W) comprises a plurality of intervals, wherein the variation of each component of the corresponding varied input vector of the first data point lies in an interval of the plurality of intervals that is associated with that component, and wherein a number of intervals of the plurality of intervals is a dimension of the input vector of the first data point. [11] Method according to any one of the preceding claims 6 to 10, wherein the predefined characteristic criterion comprises that the selected pairwise error has a maximum absolute value of all pairwise errors from the plurality of pairwise errors; or that the selected pairwise error has a minimum value and a maximum value of all pairwise errors from the plurality of pairwise errors. [12] Computer-implemented method for training and / or validating a computer-based machine learning system used in a technical system, wherein the method comprises the following steps: Receiving a data set that has been classified as suitable for subsequent training and / or validation of the computer-based machine learning system according to any one of the preceding claims 1 to 11; Training and / or validating the computer-based machine learning system with the received data set to obtain a trained and / or validated computer-based machine learning system. [13] Computer-implemented method for applying a computer-based machine learning system, wherein the method comprises the following steps: Receiving a trained and / or validated computer-based machine learning system according to claim 12; and Processing of application data by the received computer-based machine learning system. [14] Computer program designed to perform the method according to any one of the preceding claims 1 to 13. [15] Computer system designed to execute the computer-implemented method according to any one of the preceding claims 1 to 13 and / or the computer program according to claim 14. [16] Computer-readable medium or signal that stores and / or contains the computer program according to claim 14. [17] A sensor system of a technical system designed to generate at least partially a data set with measurement data points that is suitable to be classified as suitable or unsuitable for subsequent training and / or validation of the computer-based machine learning system in the method according to any one of the preceding claims 1 to 11.

Images