Method and apparatus for evaluating data set
By using a binning method to evaluate dataset quality, identify and improve weak areas, the problem of insufficient dataset evaluation in existing technologies is solved, thereby improving the robustness and reliability of machine learning systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2025-11-14
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies cannot fully reflect the labeling quality in specific regions of a dataset when evaluating dataset quality. This can lead to labeling errors in critical areas of machine learning systems, affecting the robustness and reliability of the system.
The binning method is used to divide the input space of the dataset into discrete regions, calculate the quality value of each bin, identify the weak links in the dataset by the lowest quality value, and improve the quality by expanding the dataset.
This enables differentiated evaluation of dataset quality, identifies and improves weak areas, ensures that the machine learning system has well-labeled data throughout the input space, and improves the system's robustness and reliability.
Smart Images
Figure CN122045720A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to determining the quality value (Gütewert) of a dataset used for training and / or testing a machine learning system, methods for training a machine learning system, a training system, a computer program, and a machine-readable storage medium. Background Technology
[0002] The paper “Virtual Sensor for Real-Time Bearing Load Prediction Using Heterogeneous Temporal Graph Neural Networks” by Zhao et al., published on April 2, 2024, at https: / / arxiv.org / pdf / 2404.02304, discloses the average global metrics used to test virtual sensors.
[0003] "Breck et al.'s 'The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction,' published in 2017 at the IEEE International Conference on Big Data, disclosed some issues in the deployment of reliable production-grade machine learning systems."
[0004] Advantages of the present invention High-quality datasets for training can further improve the quality of technical machine learning systems used to process real-world measurements (Messungen) (also known as virtual sensors). Here, "quality" can be understood specifically as the quality of the annotations on the inputs by the desired output; in other words, the quality of the dataset annotations. Therefore, evaluating the quality of a dataset is crucial for developing robust and reliable machine learning systems. Current methods for validating dataset quality typically focus on global metrics that generalize to the entire dataset, such as the average error across all data points. However, these global metrics may not adequately reflect the quality of annotations in specific regions of the dataset. For example, a dataset may have good average annotation quality but be mislabeled in specific regions critical to the application. Therefore, a more detailed evaluation is needed that considers the quality of the dataset across different segments of the input space.
[0005] This invention proposes a method for evaluating dataset quality, addressing the shortcomings of existing methods. Advantageously, this method enables the evaluation of a dataset's quality based on its "weakest link," ensuring that machine learning systems trained on this dataset acquire well-labeled data across all regions of the dataset. Advantageously, the inventors have found that identifying and evaluating the dataset's "weakest link" by considering the lowest quality within sub-regions of the input space achieves a more convincing assessment of dataset quality.
[0006] This invention is based on a "binning" method, in which the input space, expanded (aufspannen) of the sensor values of the dataset, is divided into discrete regions (bins) according to the values of the sensor signals. For each bin, a quality value is calculated that quantifies the consistency between the existing output in the dataset and the expected output in that bin. Therefore, the quality value can be understood as follows: a smaller quality value indicates a better quality dataset than a larger quality value. In other words, the smaller the quality value, the "better" the dataset. The quality value of the entire dataset is determined as the maximum quality value among all bins, or as the quantile of the quality values of these bins. This method enables differential evaluation of dataset quality and identifies the weakest regions of the dataset in the input space. Advantageously, the regions with the lowest quality values indicate where additional data is needed, or where the quality of existing data needs to be improved. Especially when developing machine learning systems for specific technical applications, such as image processing, this method can be advantageously used to ensure that the training dataset contains the necessary data, or that the test dataset contains a sufficient amount of data to allow for convincing results.
[0007] Another advantage of this method is that the same approach can also be used to evaluate the quality of the output of machine learning systems. Summary of the Invention
[0008] In a first aspect, the present invention relates to a computer-implemented method for determining quality values of a dataset used for training and / or testing a machine learning system, wherein the dataset includes a plurality of inputs and outputs corresponding to respective inputs, wherein the inputs characterize at least one value of a sensor signal, and the outputs include classification and / or regression results with respect to the inputs, wherein the method includes the following steps: • Determine bins for the values of multiple input sensor signals, wherein the inputs are assigned to different bins of the bins based on the values of the sensor signals contained in the inputs; • Assign the output to bins based on the assignment of the inputs corresponding to the output; • Determine a first mass value for each of the multiple boxes in the sub-box, preferably determining a first mass value for all boxes in the sub-box, wherein the first mass value represents the average or median of the second mass values, wherein the second mass values respectively represent the deviation of the output assigned to the box from the corresponding desired output; • Determine the quality value of the dataset, wherein the quality value corresponds to the largest quality value among a plurality of the first quality values or the quantile of the plurality of the first quality values.
[0009] In the context of this invention, the term "machine learning system" can be understood as a computer system capable of learning from data. It is based on algorithms that identify patterns and relationships in the data and utilize them to make predictions or decisions. Machine learning systems are optimized by training with example data, whereby system parameters are adapted to maximize prediction accuracy. Implementations of machine learning systems can include, in particular, neural networks, support vector machines, decision trees, Bayesian networks, or k-nearest neighbor classifiers / regressors. The choice of a specific machine learning system depends on the particular application and the characteristics of the data. In the context of this invention, machine learning systems are preferably neural networks, particularly deep neural networks with multiple layers, because they are capable of learning complex relationships in sensor data and achieving high prediction accuracy. However, it is not limited to this and may include any other suitable machine learning system that fulfills the desired functionality.
[0010] In the context of this invention, the term "sensor signal" can be understood as a digital representation of information generated by a sensor that reflects the sensor's physical measurement parameters. Since this method is performed by a computer, the sensor signal must be mandatory to exist in digital form, typically as an ordered sequence of binary values. The digitization of analog sensor signals can be achieved, for example, by an analog-to-digital converter (ADC).
[0011] Digital sensor signals can be in various formats, including: Time series: In this case, it is a series of discrete measurements detected over time. Each measurement is represented as a numerical value that reflects the parameter measured at a specific point in time.
[0012] A digitized image can be viewed as a two-dimensional array of digital values that spatially represent the measured parameters. Examples of this include pixel values in grayscale or color images.
[0013] Individual numerical values: These individual numerical values represent discrete numerical measurements that represent parameters measured at a specific point in time.
[0014] Eigenvectors: These eigenvectors can be understood as vectors of numerical values that represent different characteristics of the measured signal, such as the mean, standard deviation, or frequency components.
[0015] This invention is not limited to a specific sensor type and can be used with a variety of sensors whose output signals can be digitized. Examples of sensors applicable within the scope of this invention (considering signal digitization) are: Temperature sensor: measures temperature and provides digital temperature values.
[0016] Pressure sensor: measures pressure and provides a digital value.
[0017] Light sensor: measures light intensity and provides a digital value.
[0018] Accelerometer: Measures acceleration and provides a digital value.
[0019] Gyroscope: measures rotational speed and provides a digital value.
[0020] Magnetometer: Measures magnetic fields and provides digital values.
[0021] Proximity sensor: Determines the presence of an object and provides a digital status value (e.g., 0 or 1).
[0022] Image sensor: Detects images as digital pixel values.
[0023] Acoustic sensor (microphone): detects sound waves as digital audio data.
[0024] Position sensor: measures position and provides digital coordinates.
[0025] Chemical sensors: measure the concentration of chemical substances and provide digital values.
[0026] Biosensors: measure biological parameters and provide digital values.
[0027] This list is not exhaustive, and the invention can be used in conjunction with any sensor that provides a digitizable signal representing a physical parameter to be measured. The invention is particularly advantageous in applications where digitized sensor data is noisy, incomplete, or unevenly distributed. The method according to the invention enables the training of robust and accurate machine learning systems even under these challenging conditions. In particular, the input can also characterize multiple sensor signals from the same type of sensor and / or different types of sensors.
[0028] The term "the input characterizes the sensor signal" can be understood as meaning that the input contains or is composed of sensor signals. Alternatively, the input may represent the result of preprocessing the sensor signal using appropriate preprocessing methods.
[0029] The term "quantile of the first quality value" can be understood as: sorting the first quality values of the bins and determining the quality value of the dataset corresponding to a pre-given quantile, such as the 90th quantile.
[0030] The term "binning" can be understood as a method for dividing sensor signals or inputs derived therefrom into bins. This is a method of discretizing the value of an input parameter. This can be understood as dividing the range of values of the input parameter into a finite number of intervals, or "bins." Each bin represents a specific range of values.
[0031] Binning can be performed in different ways, including: Equal-spaced binning: In this case, bins with the same width are selected. This is the simplest form of binning and is particularly suitable for uniformly distributed data.
[0032] Quantile-based binning: In this case, the bin boundaries are chosen such that each bin contains the same number of data points. This method is robust to outliers and is very well suited for unevenly distributed data.
[0033] The number of bins can be understood as a hyperparameter of binning. The optimal number of bins depends on the application and data characteristics and can be determined through experimentation or cross-validation.
[0034] In the context of this invention, binning can be applied to the raw values of sensor signals or to features derived therefrom. It can also be multidimensional, meaning that multiple input parameters can be binned simultaneously. Advantageously, this enables the detection of complex relationships between input parameters. For example, in the case of two-dimensional binning, temperature and pressure can be separately assigned to their respective bins, thereby generating a bin matrix.
[0035] In the context of this invention, the term "quality value" can be understood as a numerical measure that characterizes the suitability or quality of a dataset or a portion thereof (e.g., a bin) for training and / or testing a machine learning system. In particular, a quality value can be understood as a measure of the extent to which a dataset or a portion thereof represents the desired relationship between the input (x) and the output (y). A smaller quality value may indicate higher quality, suitability, or representativeness.
[0036] Quality values can reflect different aspects of data quality, including: Representativeness: How well the dataset or a portion thereof covers the relevant input space. For example, high-quality values may indicate insufficient coverage.
[0037] Consistency: The degree of deviation between the actual output (y) in the dataset and the expected output. High-quality values may indicate inconsistent or noisy data.
[0038] Variance: How much the actual output (y) varies within a bin of a dataset.
[0039] Prediction accuracy: How well a model trained on a dataset predicts the output (y) for new, unseen inputs (x). High-quality values may indicate limited generalization ability.
[0040] The specific calculation of quality values can be based on different metrics and can refer to the entire dataset or a portion thereof, such as bins.
[0041] The following example illustrates how a mass value can be calculated for a box: Determine the deviation: For each input (x) within the bin, calculate the deviation between the actual output (y) and the expected output. The deviation can be measured, for example, using the absolute value of the difference, the squared difference, or other suitable functions.
[0042] Deviation aggregation: Aggregates the deviations of all inputs (x) within the bin to obtain a single value representing the overall quality of the bin. Possible aggregation functions are: Average value: the sum of deviations divided by the number of inputs in the box.
[0043] Median: The middle value of the sorted deviations.
[0044] The value calculated in this way represents the mass value for the corresponding box.
[0045] This invention is not limited to a specific quality metric or calculation method, and can be used in conjunction with a variety of metrics and methods capable of evaluating the quality of datasets used to train and / or test machine learning systems. The choice of metrics and calculation methods can depend on a variety of factors, such as the type of machine learning problem (e.g., classification or regression), the properties of the dataset, and the specific requirements of the application, and can be understood as hyperparameters of the method.
[0046] However, the following possibilities can be considered to determine different expected values: Existing Reference Data: In some cases, reference data containing the desired output for a specific input may already exist. This reference data can be derived, for example, from measurements, simulations, or manual annotations. In particular, the input to the dataset can also be fed to other trained machine learning systems, which then determine the desired output. In this case, the other machine learning system can be understood as an oracle or reference system.
[0047] Expert knowledge: The desired value can also be defined through expert knowledge. For example, an expert can determine the expected output for a specific input based on their expertise.
[0048] Physical models: In some applications, physical models can be used to calculate the desired output. For example, a physical model can predict the ideal temperature of an engine under specific operating conditions.
[0049] Calculations can be made based on other parameters: The desired value can also be calculated based on other parameters available in the dataset. For example, the desired speed of a vehicle can be calculated based on its current location and the desired time of arrival.
[0050] Combining different methods: It is also conceivable to combine different methods to determine the desired value. For example, a combination of reference data and expert knowledge can be used.
[0051] Compared to traditional global metrics, this approach offers an advantageous, more differentiated, and detailed assessment of data quality, particularly by considering individual bins. This enables a better evaluation of the dataset's suitability for training or testing machine learning systems.
[0052] In various advantageous implementations, a loss function can be used to determine the second quality value, wherein the loss function is the squared error loss function, the cross-entropy loss function, the hinge loss function, the Huber loss function, the L1 loss function, or the L2 loss function.
[0053] The advantages of each loss function are as follows: • Cross-entropy loss function: Especially ideal for classification problems, particularly in the case of multiple classes. It measures the difference between the predicted probability distribution and the true distribution, thus promoting convergence to the accurate classification probability.
[0054] Hinge Loss: Commonly used in Support Vector Machines. It focuses on the data points closest to the decision boundary and thus maximizes the threshold between classes.
[0055] • L1 Error (Absolute Error): Shows high robustness to outliers because it penalizes larger errors less than it does to MSE. This is particularly advantageous for noisy data or data containing outliers.
[0056] • L2 Error (Squared Error - MSE): Particularly suitable for regression problems where the target variable is continuous. The quadratic term penalizes large errors more severely, leading to a focus on minimizing significant biases. Furthermore, it is continuously differentiable, making it easier to apply gradient-based optimization methods.
[0057] In summary, each loss function offers specific advantages for different problems and data characteristics.
[0058] In different preferred embodiments, it is possible that, for binning, the value of the sensor signal can be normalized to a predetermined numerical range, preferably -1 to 1, during the binning step, or the value of the sensor signal may already be normalized to a predetermined numerical range when the input is provided.
[0059] This is advantageous because sensor data can have different units and measurement ranges. Normalizing to a uniform numerical range ensures that all input data are considered on the same scale. This prevents features with larger values from dominating features with smaller values and improves the comparability of the data. Normalization allows model parameters to be learned faster and more efficiently, leading to better performance and generalization ability of machine learning systems.
[0060] In various preferred embodiments, it is possible to divide the numerical range into equally spaced bins.
[0061] Advantageously, equidistant binning is computationally efficient not only in creating the bins but also in assigning data points to them. This accelerates the training process of machine learning systems and makes the method usable even in repeated iterations of iterative optimization methods, such as stochastic gradient descent.
[0062] In various implementations, the input may characterize multiple sensor signals.
[0063] Advantageously, in this case, the range of values for each input parameter derived from a single sensor signal can be divided into multiple bins. The combination of bins for all input parameters is then unfolded into a multidimensional space, in which each range is represented by a unique combination of bins. The number of bins for each input dimension can also be chosen differently.
[0064] In various implementations, the machine learning system may be a virtual sensor.
[0065] In the context of this invention, the term "virtual sensor" can be understood as a software-based model that simulates the function of a physical sensor. Instead of directly measuring a physical parameter, a virtual sensor estimates that parameter based on measurements from other physically present sensors and / or other available data. It uses a trained machine learning system for this purpose, which has learned the relationship between available data and the parameter to be estimated. A specific example of a virtual sensor in the context of this invention is estimating engine temperature in a vehicle. Instead of directly mounting an expensive and potentially inaccessible temperature sensor on the engine, a virtual sensor can estimate engine temperature based on more readily available measurement parameters. Exemplary inputs to a virtual sensor are engine speed, coolant temperature, ambient temperature, engine load conditions (e.g., torque), and / or the time since the engine was last started.
[0066] In various embodiments of the invention, it is possible to input a combination representing the position of the robot arm and output a combination representing the position of the actuators of the robot arm, wherein the robot arm occupies these positions when the actuators are positioned.
[0067] Instead of using complex inverse kinematics to calculate the robot's joint angles, machine learning systems can advantageously learn the relationship between the desired position of the robot arm and the necessary settings of the actuators. For example, the desired position, defined by the robot arm's endpoint (e.g., the orientation of the gripper) and Cartesian coordinates, is used as input to a virtual sensor. The output of the virtual sensor can consist, in particular, of a combination of actuator positions or settings, such as the angles of the joints that bring the robot arm to the desired position, or include combinations of actuator positions or settings, such as the angles of the joints that bring the robot arm to the desired position. Thus, the virtual sensor acts as an inverse kinematics model trained by machine learning. To train this virtual sensor, training data can first be collected by having the robot arm occupy different positions and recording the corresponding actuator settings. Subsequently, the input (desired position) and / or the output data (actuator settings) can be normalized and binned as described above. If the robot arm's position is multidimensional, multidimensional binning is used. Then, this prepared data is used to train a machine learning system, such as a neural network. By using a maximum loss value during training, it is ensured that the virtual sensor reliably estimates actuator settings even under unfavorable input data conditions, such as at the edge of the robot's workspace. During operation, the virtual sensor takes the desired position of the robot arm as input and provides the corresponding actuator settings as output, which are then used to control the robot arm. The advantage of this approach is simplified control of the robot arm, as complex inverse kinematics are not required. Additionally, training with a maximum loss value improves the virtual sensor's robustness to inaccuracies and variations in the input data. The virtual sensor can also be retrained using new data to adapt to changes in the robot system.
[0068] Advantageously, estimating actuator settings via virtual sensors is faster than analytically calculating inverse kinematics.
[0069] The arm of a Bagger excavator can also be understood in particular as a robotic arm, and the above example can be similarly applied to the arm of the excavator.
[0070] In a preferred embodiment, it is also possible that the output ( ) is processed by a machine learning system (60) to process the input ( The result of ).
[0071] These implementations can be understood as tests of the machine learning system. In other words, the output of the machine learning system is considered as the basis for evaluation, and thus the behavior of the machine learning system is tested. Advantageously, this testing does not derive a global metric for the machine learning system on the entire dataset, but rather allows for a significantly more granular exploration of where the machine learning system (in the sense of the bins) fails to perform sufficiently good predictions by dividing the dataset into multiple bins.
[0072] In other preferred embodiments, in the step of determining the quality value of the (104) dataset, the data may be derived from an input that has been assigned at least one predefined number of values. For example, at least 10 inputs ( Determine the maximum mass value or quantile in the box.
[0073] In other words, in these implementations, only bins that have been assigned "sufficient" inputs are considered, where "sufficient" is defined by a pre-given number. This pre-given number can be understood as a hyperparameter of the method. However, the inventors can determine that bins with at least 10 assigned inputs provide good results.
[0074] Advantageously, by selecting boxes that are sufficiently filled, quality can be better assessed, since boxes with individual outliers are irrelevant.
[0075] In another aspect, the present invention relates to a method for training a machine learning system, the method comprising the following steps: a. Determine the quality value of the dataset according to any one of the preceding claims; b. If the quality value reaches or exceeds a predefined threshold: • Use datasets to train machine learning systems; c. Otherwise: • Identify the box corresponding to the determined mass value; • Identify at least one other input and a corresponding output, wherein the input is located within the box; • Determine the expanded dataset by adding inputs and outputs to the dataset; • Using the expanded dataset as the dataset, repeat steps a and b or c.
[0076] Advantageously, the methods used to train machine learning systems allow for incremental improvement of the training dataset until the dataset is “good enough” to enable the training of machine learning systems.
[0077] At least one additional combination of input and output can be achieved, in particular, by selectively recording the desired input in the real world. Specifically, a testing environment can be provided for this purpose, and further inputs can be recorded accordingly. Then, for example, further outputs can be produced by annotating said inputs with the inputs of a human expert.
[0078] Alternatively, computer-aided simulation can be used to determine at least one additional input and output. For example, a virtual environment can be generated using a physical model of the input and output, through which the input and output can be determined or synthesized.
[0079] On the other hand, the present invention relates to a training device configured to perform a training method.
[0080] On the other hand, the present invention relates to a computer program configured to perform one of the above-described methods when executed by a processor.
[0081] On the other hand, the present invention relates to a machine-readable storage medium on which a computer program is stored. Attached Figure Description
[0082] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. In the drawings: Figure 1a schematically illustrates the method used to determine the quality value; Figure 1b schematically illustrates a method for training a machine learning system; Figure 2 shows the training system; Figure 3 schematically illustrates the structure of the control system used to manipulate the actuator; Figure 4 schematically illustrates an embodiment for controlling at least a semi-autonomous robot. Detailed Implementation
[0083] Figure 1a schematically illustrates a flowchart (105) of a computer-implemented method for determining the quality values of a dataset (the black dot in the figure symbolizes the start of the method). The dataset contains multiple inputs and corresponding outputs. These mutually corresponding inputs and outputs can be understood as data pairs. The inputs of each data pair represent at least one sensor signal, while the outputs represent the classification or regression result assigned to that sensor signal, i.e., the result of regression analysis. The inputs and outputs are preferably in numerical and mathematical structure form, such as vectors, matrices, or tensors. The numerical values of the inputs represent the values of the sensor signals represented by those inputs.
[0084] In the first step (101), the input values are binned. Preferably, the input is binned at equal intervals. Preferably, the upper and lower bounds of the bins can also be determined based on the dataset. For example, in the corresponding input dimension, the minimum value of the input can be used as the lower bound of the bins, and the maximum value of the input can be used as the upper bound of the bins.
[0085] In the second step (102), the outputs are assigned to the following boxes respectively, and the inputs corresponding to the outputs are also assigned to the boxes.
[0086] In the third step (103), a first quality value is determined for each of the bins. To determine the first quality value, a second quality value is determined, whereby the second quality value characterizes the deviation between the output assigned to that bin and the desired output. The desired output can be determined as described above. The deviation can be numerically represented by a suitable index, whereby the numerical value can be understood as the second quality value. One of the aforementioned indices can be used as such an index. The second quality values thus determined for the bins can then be aggregated to determine the first quality value. As an aggregation, in particular, the average or median of the second quality values can be provided as the first quality value.
[0087] In step four (104), the quality value of the dataset is determined. For this purpose, the first quality value with the smallest value across the different bins is given as the quality value of the dataset. The quality of the dataset is output as a result of this method (the last arrow in 104).
[0088] Figure 1b schematically illustrates a flowchart of a computer implementation method (110) for training a machine learning system (the black dots in the figure symbolize the start of the method). The machine learning system is preferably configured to receive input from a dataset and determine outputs representing classification and / or regression results. The machine learning system is preferably a neural network, such as a multilayer perceptron, convolutional neural network, or transformer. Common architectures of neural networks can be used in this method.
[0089] In the first step of method (105), preferably according to Figure 1 The embodiments in a determine the quality of the dataset. In other words, the quality value is determined according to one of the methods described above.
[0090] If the determined quality value reaches or exceeds a predefined threshold, the machine learning system uses the dataset for training in a further step (109). This can be understood in particular as follows: the dataset can be used as a training dataset for the machine learning system. Since the dataset contains pairs of inputs and outputs, these pairs can be used directly for supervised training. In particular, stochastic algorithms, such as SGD or Adam, can be used as training algorithms.
[0091] If the determined quality value is below the threshold, then in Figure 1 The method shown in b expands the dataset to improve quality. To this end, in step (106), bins are identified for which a first quality value matches the determined quality value. In a further step (107), other inputs located within these bins are identified. This can be done by examining the bin boundaries and identifying inputs located within those boundaries. The identified inputs can then be labeled according to one of the corresponding methods described above. This allows for the identification of another input-output pair. In a further step (108), this pair is added to the dataset, thus determining the expanded dataset. Quality values can then be determined again for the expanded dataset. Therefore, the dataset can be iteratively expanded more and more. It is also possible that in at least one iteration, multiple input-output pairs are identified and added to the dataset.
[0092] Figure 2 illustrates an embodiment of a training system (140) for training a machine learning system (60) using a training dataset (T). The training dataset (T) includes multiple inputs for training the machine learning system (60). ), where, for each input ( The training dataset (T) also includes the expected output (). The desired output is related to the input. ) corresponds to and characterizes the input ( The classification and / or regression results of ).
[0093] For training purposes, the training data unit (150) accesses a computer-implemented database (St2) that provides a training dataset (T). The training data unit (150) preferably randomly selects at least one input from the training dataset (T). ) and the input ( The expected output corresponding to ) ), and input ( The data is transmitted to the machine learning system (60). The machine learning system (60) is based on the input ( Determine the output ( ).
[0094] Expected output ( ) and the determined output ( The data is transmitted to the change unit (180).
[0095] Then, based on the desired output ( ) and the determined output ( The change unit (180) determines new parameters for the machine learning system (60). Therefore, the change unit (180) uses a loss function to compare the expected output (). ) and the determined output ( The loss function determines a first loss value, which characterizes the determined output. ) and the expected output ( The degree of bias is considered. In this embodiment, the negative log-likelihood function is chosen as the loss function. In alternative embodiments, other loss functions may also be considered.
[0096] Furthermore, it can be envisioned that the determined output ( ) and the expected output ( Each contains multiple sub-signals, for example, in the form of tensors, where the desired output ( The sub-signals of ) correspond to the determined output ( The first sub-signal represents the target object relative to the input. For example, one could imagine a machine learning system (60) designed for object detection, where the first sub-signal represents the target object relative to the input. The probability of a portion of the signal occurring is given by the first sub-signal, and the second sub-signal characterizes the precise location of the object. For the determined output (…), ) and the expected output ( In cases where there are multiple corresponding sub-signals, it is preferable to use a suitable loss function to determine a second loss value for each corresponding sub-signal, and then appropriately combine the determined second loss values into a first loss value, for example, through weighted summation.
[0097] The change unit (180) determines new parameters based on the first loss value. In this embodiment, training is implemented using gradient descent, preferably stochastic gradient descent, Adam, or AdamW. In other embodiments, training may also be based on evolutionary algorithms or second-order optimization.
[0098] The determined new parameters ( The determined new parameters are stored in the model parameter memory (St1). Preferably, the determined new parameters are stored in the model parameter memory (St1). ) as a parameter ( ) is provided to machine learning systems (60).
[0099] In other preferred embodiments, the training is iteratively repeated for a predefined number of iterations, or iteratively repeated until a first loss value falls below a predefined threshold. Alternatively or additionally, it is also conceivable that training terminates when the average first loss value on the test or validation dataset falls below a predefined threshold. In at least one of these iterations, new parameters determined in previous iterations ( ) is used as a parameter of the machine learning system (60) ).
[0100] Furthermore, the training system (140) may include at least one processor (145) and at least one machine-readable storage medium (146) containing instructions that, when executed by the processor (145), cause the training system (140) to perform a training method according to one aspect of the invention.
[0101] Figure 3 illustrates a control system (40) that utilizes a machine learning system (60) to determine the control signal (A) for the actuator. At preferred regular time intervals, the environment (20) of the actuator (10) is detected by a sensor (30), particularly an imaging sensor, such as a camera sensor, which may also be derived from multiple sensors, such as a stereo camera. The sensor signal (S) of the sensor (30)—or, in the case of multiple sensors, each sensor signal (S)—is transmitted to the control system (40). The control system (40) thus receives the sequence of sensor signals (S). The control system (40) thereby determines the control signal (A), which is transmitted to the actuator (10).
[0102] The control system (40) receives a sequence of sensor signals (S) from the sensor (30) in an optional receiving unit (50), wherein the receiving unit converts the sequence of sensor signals (S) into an input ( The sequence of (or alternatively, the sensor signal (S) can be directly used as input) (The input is received). This can be, for example, a fragment of the sensor signal (S) (Ausschnitt) or further processed. In other words, the input ( The input (S) is determined based on the sensor signal (S). The sequence was fed into the machine learning system (60).
[0103] The machine learning system (60) is preferably composed of parameters ( The parameters are parameterized and stored in and provided by the parameter memory (P).
[0104] The machine learning system (60) is based on the input signal ( Determine the output ( The output ( The signal is sent to an optional conversion unit (80), from which a control signal (A) is determined, which is then sent to the actuator (10) to control the actuator (10) accordingly.
[0105] The actuator (10) receives the control signal (A), is controlled accordingly, and performs the corresponding action. In this case, the actuator (10) may include (not necessarily structurally integrated) control logic, which determines a second control signal from the control signal (A), which is then used to control the actuator (10).
[0106] In other embodiments, the control system (40) includes a sensor (30). In a further embodiment, the control system (40) also alternatively or additionally includes an actuator (10).
[0107] In other preferred embodiments, the control system (40) includes at least one processor (45) and at least one machine-readable storage medium (46) having instructions stored thereon that, when executed on at least one processor (45), cause the control system (40) to perform the method according to the invention.
[0108] In an alternative implementation, a display unit (10a) is provided instead of or attached to the actuator (10).
[0109] Figure 4 illustrates how a control system (40) can be used to control at least a semi-autonomous robot, which here refers to at least a semi-autonomous motor vehicle (100).
[0110] For example, the sensor (30) may be a sensor preferably arranged within the robot (100) that performs measurements on components of the robot (100). This component may, for example, be the fuel injection valve of the internal combustion engine of the vehicle (100), wherein the sensor is a piezoelectric sensor fixed to the valve. The voltage measured by the piezoelectric sensor can be used as an input to a machine learning system (60). At least a portion thereof, wherein the machine learning system (60) is configured to predict the amount of fuel injected into the valve based on the voltage. To train the machine learning system (60), a corresponding pair consisting of the voltage of the piezoelectric sensor and the amount of fuel injected can be measured on a test device.
[0111] Alternatively, the component can be the electric motor of the robot (100), for example, a drive motor for propelling the robot (100) to move. Machine learning systems can, in particular, predict the temperature of the engine's stator or rotor as an output (…). Inputs can include engine load, engine voltage, engine current consumption, and temperature measured at locations other than the rotor or stator.
[0112] The actuator (10) preferably arranged within the robot (100) can be, for example, the robot's braking mechanism, drive mechanism, or steering mechanism, or it can be a valve as mentioned above. The control signal (A) can be selected, for example, such that it is based on the output ( The valve, representing the amount of fuel injected, controls the injection of more or less fuel into the engine based on the actual amount of fuel injected. If the output ( If the temperature of the stator or rotor is characterized, the robot (100) can be put into a safe state when the temperature reaches or exceeds a predetermined threshold. The safe state may include, for example, shutting down the robot or keeping the robot in a safe area (e.g., in the case of at least semi-autonomous motor vehicles, keeping it in an emergency parking lane).
[0113] Alternatively or additionally, the display unit (10a) can be controlled using a control signal (A), and, for example, displayed the identified injection quantity or temperature. It is also conceivable that the display unit (10a) can be controlled using the control signal (A) to output an optical or acoustic warning signal upon determining a risk of overheating in the stator or rotor. Warnings delivered via warning signals can also be delivered via tactile warning signals, for example, through vibration of the steering wheel of a motor vehicle (100).
[0114] The term "computer" includes any device used to perform pre-given computational methods. These computational methods can exist in software, hardware, or a combination of both.
[0115] Generally, "multiple" can be understood as indexed, meaning each element in the multiple is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the multiple. Preferably, if multiple Contains multiple elements, among which If the number of elements in the plurality is given, then these elements are assigned from 1 to... Integers.
Claims
1. A computer-implemented method (105) for determining the quality value of a dataset (T) used for training and / or testing a machine learning system (60), wherein, The dataset (T) includes multiple inputs ( ) and respectively with the corresponding input ( The corresponding output () ), wherein the input ( The output (S) represents at least one value of the sensor signal (S). The method includes classification and / or regression results regarding the input, wherein the method comprises the following steps: • Determine (101) multiple inputs ( The binning of the sensor signal (S) value, wherein the binning is based on the input ( The value of the sensor signal (S) contained in the input () will be input to () ) are allocated to different boxes in the sub-boxes; • Based on the output ( The corresponding input () The allocation of the output ( Distribute (102) to the box; • Determine (103) a first mass value for multiple boxes in a sub-box, preferably determining a first mass value for all boxes in the sub-box, wherein the first mass value represents the average or median of a second mass value, wherein the second mass value represents the output assigned to the box ( The deviation from the corresponding expected output; • Determine the quality value of the dataset, wherein the quality value corresponds to the largest quality value among a plurality of first quality values or the quantile of a plurality of first quality values.
2. The method according to claim 1, wherein, The second quality value is determined using a loss function, wherein the loss function is a squared error loss function, a cross-entropy loss function, a hinge loss function, a Huber loss function, an L1 loss function, or an L2 loss function.
3. The method according to claim 1 or 2, wherein, For binning, the value of the sensor signal is normalized to a predetermined numerical range during the binning step, preferably -1 to 1, or the value of the sensor signal is already normalized to a predetermined numerical range when the input is provided.
4. The method of claim 3, wherein the numerical range is divided into equally spaced boxes.
5. The method according to any one of the preceding claims, wherein the input represents a plurality of sensor signals (S).
6. The method according to any one of the preceding claims, wherein the machine learning system (60) is a virtual sensor.
7. The method according to any one of the preceding claims, wherein, The input represents the position of the robot arm, and the output represents a combination of the positions of the actuators of the robot arm, wherein the robot arm occupies these positions when the actuator positions are set.
8. The method according to any one of the preceding claims, wherein the output ( The input is processed by the machine learning system (60). The result of ).
9. The method according to any one of the preceding claims, wherein in the step of determining the quality value of the dataset (104), the data is derived from an input (which is assigned at least one predefined number of inputs) For example, at least 10 inputs ( Determine the maximum mass value or quantile in the box.
10. A computer-implemented method (110) for training a machine learning system (60), the method comprising the following steps: a. Determine the quality value of the (105) dataset (T) according to any one of the preceding claims; b. If the quality value reaches or exceeds a predefined threshold: • The machine learning system (60) is trained (109) using the dataset. c. Otherwise: • Determine (106) the box corresponding to the determined mass value; • Determine (107) at least one other input ( ) and the corresponding output ( ), wherein the input ( Located inside the box; • By the input ( ) and output ( Add to dataset ( ) to determine the (108) expanded dataset; • Using the expanded dataset as dataset (T), repeat steps a and b or c.
11. A training device (140) configured to perform the method according to claim 10.
12. A computer program configured to perform the method according to any one of claims 1 to 10 when executed by a processor (145).
13. A machine-readable storage medium (146), wherein the computer program of claim 12 is stored on the machine-readable storage medium.