Method and device for providing a trained machine learning model for detecting a hands-on / hands-off state of a steering wheel of a vehicle

Abstract

The invention relates to a method for providing a trained machine learning model for detecting a hands-on / hands-of state (40) of a steering wheel of a vehicle (50), having the steps of: training at least one teacher model (20) using a training data set (10); training a smaller student model (21) using the at least one teacher model (20) in the form of a teacher-student training process, wherein for this purpose, a loss function is used which is composed of a weighted sum of a student loss and a distillation loss, and a weighting factor (λ) is ascertained, while taking into consideration at least one specified criterion (11), for each piece of training data or group of training data of a training data set (10) used in the teacher-student training process; and providing the trained student model (21). The invention additionally relates to a device (1) for providing a trained machine learning model for detecting a hands-on / hands-off state (40) of a steering wheel of a vehicle (50) and to a method and a device for detecting a hands-on / hands-off state (40) of a steering wheel of a vehicle (50).

Description

[0001] Description

[0002] Method and apparatus for providing a trained machine learning model for detecting a hands-on / hands-off state on a steering wheel of a vehicle

[0003] The invention relates to a method and a device for providing a trained machine learning model for detecting a hands-on / hands-off state on a steering wheel of a vehicle. Furthermore, the invention relates to a method and a device for detecting a hands-on / hands-off state on a steering wheel of a vehicle.

[0004] Sensors such as a capacitive steering wheel are used in vehicles to monitor driver activity. Such a steering wheel detects when the driver has touched (“hands-on”) or not touched (“hands-off”) the steering wheel using a capacitive sensor. The result is transmitted to the functions being used, such as a longitudinal and / or lateral guidance assistance system. The contact of the hands on the steering wheel can be used to determine driver activity and the driver’s level of attention. For example, the system can remind the driver to put their hands on the steering wheel if it is detected that their hands have not been on the steering wheel for a specified time during lateral guidance.

[0005] To save additional costs for a capacitive sensor in the steering wheel, it is known to monitor driver activity using machine learning models, in particular artificial neural networks, based on a torque (hand torque) detected on the steering wheel. Such a method is known, for example, from DE 102019211 016 A1.

[0006] A major challenge with steering torque-based detection is identifying the driver-induced torque in the measured (noisy) torque (steering torque). Many factors can lead to noisy torque, in particular the position of the sensor (this is usually part of the steering gear or steering assistance, which, via the elasticity of the steering column in conjunction with the steering wheel, creates a system capable of torsional vibration, whose inherent dynamics make precise measurement of the driver-induced torque difficult); the strength of friction in the steering system; re-excitation from the road surface due to unevenness; the weight of the steering wheel / steering system; and vibration of the steering wheel due to an assistance function (e.g., due to haptic feedback when leaving the lane).

[0007] In addition, the characteristics (used for hands-on / hands-off detection) of the measured torque can change due to external influences, e.g. temperature, vehicle load, the presence of a trailer, tire type and / or tire condition, steering system changes over the lifetime, road gradient / inclination / banking, etc.

[0008] Due to the number of influences, it is necessary to equip the machine learning model, especially the neural network, with enough degrees of freedom, i.e., weights, to allow the model to represent all influences. However, this leads to a large model and thus greater memory and computing resource requirements. In particular, the relationship between the number of weights and the quality of representation of the desired objective function is not linear, but increases almost exponentially.

[0009] The use of a teacher-student training is, for example, from the

[0010] DE 11 2020 001 663 T5 or DE 102021 200643 B3. Furthermore, the use of teacher-student training is known from G. Hinton et al., Distilling the Knowledge in a Neural Network, arXiv: 1503.02531 v1 [stat.ML], March 9, 2015.

[0011] The invention is based on the object of improving a method and a device for providing a trained machine learning model for detecting a hands-on / hands-off state on a steering wheel of a vehicle. Furthermore, the invention is based on the object of improving a method and a device for detecting a hands-on / hands-off state on a steering wheel of a vehicle.

[0012] The object is achieved according to the invention by a method having the features of patent claim 1 and a device having the features of patent claim 11.

[0013] The latter object is achieved by a method having the features of patent claim 10 and a device having the features of patent claim 12. Advantageous embodiments of the invention emerge from the subclaims.

[0014] In particular, in a first aspect, a method for providing a trained

[0015] A machine learning model for detecting a hands-on / hands-off state on a steering wheel of a vehicle is provided, comprising: training at least one teacher model using a training data set, training a smaller student model using the at least one teacher model by means of teacher-student training, using a loss function that is composed of a weighted sum of a student loss and a distillation loss, a weighting factor being determined for each training date or group of training data of a training data set used in the teacher-student training, taking into account at least one predetermined criterion, and providing the trained student model.

[0016] Furthermore, in particular in a second aspect, a device is provided for providing a trained machine learning model for recognizing a hands-on / hands-off state on a steering wheel of a vehicle, comprising a data processing device, wherein the data processing device is configured to train at least one teacher model using a training data set, to train a smaller student model using the at least one teacher model by means of teacher-student training, wherein a loss function is used for this purpose which is composed of a weighted sum of a student loss and a distillation loss, and to define a weighting factor for each training data item or group of training data items of a training data set used in the teacher-student training, taking into account at least one predetermined criterion.

[0017] Furthermore, in particular in a third aspect, a method for detecting a hands-on / hands-off state on a steering wheel of a vehicle is provided, comprising: detecting at least one steering variable on the steering wheel of the vehicle, using a student model trained according to an embodiment of the first or second aspect to detect the hands-on / hands-off state, wherein the detected at least one steering variable is supplied to an input of the trained student model for this purpose, outputting the hands-on / hands-off state estimated by the trained student model.

[0018] Finally, in a fourth aspect, a device for detecting a hands-on / hands-off state on a steering wheel of a vehicle is provided, comprising at least one sensor for detecting at least one steering variable on a steering wheel, a control unit, wherein the control unit is configured to receive the detected at least one steering variable, to provide and use a student model trained according to an embodiment of the first or second aspect to detect the hands-on / hands-off state, and to supply the detected at least one steering variable to an input of the trained student model for this purpose, and to output the hands-on / hands-off state estimated by the trained student model.

[0019] The method and device according to the first and second aspects make it possible to provide a machine learning model that is improved in terms of memory and computing resource requirements. For this purpose, a student model is provided which, in terms of structural design, is particularly simpler or less complex than the at least one teacher model. For example, the student model can have fewer neurons or fewer inner layers and thus fewer parameters (in particular weights) than the teacher model.To prevent a reduction in functional quality, the training of the student model is planned to use a loss function composed of a weighted sum of a student loss and a distillation loss. The weighting factor is determined for each training data point or group of training data points of a training data set used in teacher-student training, taking into account at least one predefined criterion. The predefined at least one criterion defines, in particular, the factors that are considered when determining the weighting factor or the factors on the basis of which the weighting factor is determined.The ability to individually define the weighting factor for each training datum or group of training datums allows for each training datum in the training data set to influence the extent to which the student model learns from the at least one teacher model during teacher-student training. This allows for a focus to be set during teacher-student training, for example, resulting in higher quality when recognizing the hands-off state than when recognizing the hands-on state. The student model trained in this way is provided, in particular output. The trained student model is used, in particular, in a vehicle to recognize a hands-on / hands-off state. Based on this, an assistance system can be operated, for example.

[0020] A particular advantage of the method and device is that a student model with improved functional quality can be provided. This allows a machine learning model with reduced memory and computing resource requirements to be provided, particularly with improved functional quality. The models (the at least one teacher model and the student model) are or will be trained in particular to estimate a hands-on / hands-off state based on at least one steering variable.

[0021] The at least one teacher model and the student model are designed in particular as artificial neural networks, in particular as deep neural networks, which in particular comprise several inner layers. The models are in particular artificial recurrent neural networks, which at each time point t process the input data X t process and at least a hands-off probability y toutput in [0,1]: y t = p(x t | x 0:ti). Preferably, the neural networks output a probability for a hands-on state and a hands-off state at each time t. In particular, the neural networks have a so-called memory h, in which information from previous journals is stored and which can be used for the output of the current journal. The output is further processed, for example, by filtering, before the decreasing functions (e.g., a lateral guidance assistant) process it. In particular, it can be provided that, based on a comparison of the hands-off probability with a predetermined threshold value, a binary hands-off signal is provided (with the two states "hands-off detected" and "hands-off not detected"). The same can be done for the hands-on state, or a joint signal can be provided with the two signal states "hands-on" or "hands-off".

[0022] During training (i.e. during the respective training phase), the models are trained using training data comprising pairs in which data of the steering variable, in particular torque data, are each paired with a hands-on / hands-off state as ground truth. The data of the at least one steering variable, in particular the torque data, are in particular time series of the at least one steering variable detected at the steering wheel, in particular time series of torques detected at the steering wheel. The training data is obtained in particular with the aid of test drives and / or in simulators. In principle, the provision of training data can be carried out in particular according to the method described in DE 102019211 016 A1. Training is carried out in a manner known per se, in particular by means of supervised learning. The detection and / or collection of the training data of the training data set can be part of the first and second aspects.The hands-on / hands-off state is detected using additional sensors (e.g., a capacitive steering wheel). During training, the loss function calculates an error per weight as part of backpropagation and thus optimizes it. If the hands-on / hands-off state is detected, binary cross entropy can be used.

[0023] L = -y * log(Y) + (1-y)*log(1-Y), where Y is the ground truth and y is the output of the model.

[0024] According to the invention, an adapted loss function is used for student training within the framework of teacher-student training:

[0025] L = A * student-loss + (1-A) * distillation-loss, where student-loss is the binary cross-entropy between the output of the student model and the ground truth, and distillation-loss is the binary cross-entropy between the output of the student model and the output of the teacher model. The weighting factor A is the weighting factor that is determined according to the invention for each training data item or group of training data of a training data set used in teacher-student training, taking into account at least one predefined criterion. The weighting factor A is therefore not a constant, but is determined taking into account at least one predefined criterion.

[0026] In teacher-student learning, the student model is trained, in particular, with the help of at least one teacher model. The same training data is fed to both the teacher model and the student model as input data. In addition to a ground truth, an output from the teacher model is used to adjust parameters (particularly weights) of the student model and thus train the student model. These two variables are taken into account using the adjusted loss function described above.

[0027] The method for providing a trained machine learning model for detecting a hands-on / hands-off state on a steering wheel of a vehicle can be implemented, in particular, as a computer-implemented method. A steering variable is, in particular, a variable that represents and / or describes a current state of the steering wheel. A steering variable is, in particular, a torque that is detected, in particular, by means of a torque sensor on the steering wheel. In principle, however, a steering variable can also be another variable detected directly or indirectly on the steering wheel. For example, it can be provided to detect a current at an electrical machine on the steering wheel and use it as a steering variable. The hands-on / hands-off state can be detected exclusively based on the steering variable detected on the steering wheel, in particular a detected torque.However, it is also possible, in particular, to use additional (steering) variables that are measured at the steering wheel (e.g., a steering wheel angle and / or a steering wheel angular velocity, etc.), and for the models to recognize the hands-on / hands-off state taking these additional variables into account. Furthermore, variables that are not measured at the steering wheel can also be taken into account, such as vehicle speed, lateral acceleration, yaw rate, wheel ticks, damper information, and / or other driving dynamics variables, etc. In particular, however, no capacitive sensor is provided on the steering wheel.

[0028] A hands-off state is, in particular, a state in which the driver does not touch the steering wheel. In particular, none of the driver's fingers are in contact with the steering wheel. Detecting the hands-off state can, in particular, comprise providing a hands-off state signal. This comprises, for example, a hands-off probability or coded signals for the states "hands-off detected" and "hands-off not detected." A hands-on state is, in particular, a state in which the driver touches the steering wheel. In particular, a hands-on / hands-off state is provided, for example as a hands-on / hands-off state signal with, in particular, two signal states (e.g., "hands-on detected" or "hands-off detected").

[0029] Parts of the devices, in particular the data processing units, can be implemented individually or collectively as a combination of hardware and software, for example as program code executed on a microcontroller or microprocessor. However, it can also be provided that parts are implemented individually or collectively as an application-specific integrated circuit (ASIC) and / or a field-programmable gate array (FPGA). In one embodiment, it is provided that the trained student model is loaded into a memory of at least one control unit of a vehicle after training as part of the provision process. This allows the trained student model to subsequently be used in the vehicle to provide hands-on / hands-off recognition. Loading into the memory includes, in particular, loading a structural description and parameters (weights, hyperparameters, etc.).) of the student model into memory.

[0030] In one embodiment, a value of the weighting factor is determined based on the respective training data criterion. This allows the content of the respective training data to be taken into account. In particular, the weighting factor can be determined, for example, taking into account semantic content represented in the training data.

[0031] In one embodiment, the at least one criterion relates to a value of the ground truth of the respective training data item, wherein the ground truth includes at least the values ​​"hands-on" or "hands-off." This allows the weighting factor to be determined depending on the ground truth of the respective training data item. For example, different values ​​for the weighting factors can be determined depending on whether the ground truth includes the state "hands-on" or "hands-off."

[0032] In one embodiment, the weighting factor is selected such that the student model for the "Hands-Off" state learns more strongly from the at least one teacher model. This allows for an emphasis to be placed on the knowledge already distilled for the "Hands-Off" state in the at least one teacher model. In particular, for a "Hands-Off" ground truth, a value of A < 0.5 is then selected for the weighting factor according to the equation given above.

[0033] In one embodiment, it is provided that the at least one teacher model is more specialized and / or robustified for recognizing the hands-off state during training of the at least one teacher model. This allows an emphasis to be placed on achieving high functional quality in recognizing the hands-off state already during training of the teacher model. Since recognizing the hands-off state is more critical in the application case than recognizing the hands-on state, this can further increase robustness when using assistance systems. In one embodiment, it is provided that the weighting factor is set such that the student model for the "hands-on" state learns more from the training data than from the at least one teacher model. This can mitigate the effect of specializing the teacher model for the "hands-off" state.For this purpose, a value of A > 0.5 is chosen for the weighting factor for this state of the ground truth according to the equation given above.

[0034] In one embodiment, the at least one criterion relates to a performance metric of the at least one teacher model. This allows a functional quality of the teacher model to be directly taken into account when selecting the weighting factor. In particular, a specialization of the teacher model can be responded to flexibly. This means, for example, that a greater weight can always be placed on the teacher model where it is specialized and where it delivers a better result in terms of functional quality. The performance metric can, for example, be a binary cross-entropy between an output of the teacher model and the ground truth.

[0035] In one embodiment, it is provided that the at least one teacher model comprises a plurality of teacher models, each of which is trained for a specific context. This allows a teacher model to be used to train the student model depending on the context. For example, specialized teacher models can be provided for one or more of the following contexts: vehicle with trailer, vehicle with snow chains, poor roads (cobblestones, potholes, speed bumps, etc.), maximum assistance interventions (e.g. steering wheel vibration as haptic feedback), extreme outside temperatures (extremely hot or extremely cold), etc. In particular, it is provided that, as part of the training of the teacher model, training data sets tailored to the respective context are generated and used.For this purpose, for example, an original training data set is enriched and / or thinned out with regard to the training data it contains in such a way that the desired context is more closely represented by the training data contained in the training data set. The specialised teacher models are initialised in particular using parameters (weights etc.) of a fully trained (non-specialised) (standard) teacher model. This makes it possible to provide basic knowledge at the beginning of training the specialised teacher model without it already being specialised for a specific context. This can particularly accelerate the training of the specialised teacher models. During teacher-student training, the teacher model is then either the standard model, one of the specialised teacher models, or a mixture of these. Depending on the respective data point (orDepending on the context (or domain) in the training dataset used, a corresponding specialized teacher model is then selected (e.g., for a data point that corresponds to a context of an attached trailer, the teacher model specialized for the trailer context is selected, etc.). If there is an overlap between the contexts (or domains), all teacher models specialized for the context are selected. The multiple specialized teacher models can then be considered in the loss function using weightings.

[0036] Further features of the device design will become apparent from the description of embodiments of the method. The advantages of the device are the same as those of the embodiments of the method.

[0037] The invention will be explained in more detail below using preferred embodiments with reference to the figures.

[0038] Fig. 1 is a schematic diagram illustrating embodiments of the device for providing a trained machine learning model for detecting a hands-on / hands-off state on a steering wheel of a vehicle;

[0039] Fig. 2 is a schematic flow diagram illustrating embodiments of the method for providing a trained machine learning model for detecting a hands-on / hands-off state on a steering wheel of a vehicle.

[0040] Fig. 1 shows a schematic representation to illustrate embodiments of the device 1 for providing a trained machine learning model for detecting a hands-on / hands-off state on a steering wheel of a vehicle 50. The device 1 is configured to carry out the method described in this disclosure.

[0041] The device 1 comprises a data processing device 2. The data processing device 2 comprises at least one computing device 2-1 and at least one memory 2-2. The computing device 2-1 can perform computing operations on data stored in the memory 2-2 to implement the method.

[0042] The data processing device 2 is configured to train at least one teacher model 20 using a training data set 10 and to train a smaller student model 21 using the at least one teacher model 20 by means of teacher-student training. For this purpose, a loss function is used, which is composed of a weighted sum of a student loss and a distillation loss. A weighting factor A for each training data item or group of training data of a training data set 10 used in the teacher-student training is determined taking into account at least one predefined criterion 11.

[0043] Figure 2 shows a schematic flowchart illustrating one embodiment of the method for providing a trained machine learning model for detecting a hands-on / hands-off state on a steering wheel of a vehicle. The method is executed, for example, on a device 1 as shown in Figure 1.

[0044] In a measure 100, at least one teacher model is trained using a training data set. This is done in a conventional manner, for example, using supervised learning.

[0045] In an action 101, a smaller student model is trained using the at least one teacher model by means of teacher-student training. For this purpose, action 101 comprises actions 101a to 101d. In action 101a, a training date or a group of training data is selected from the training data set for a run. In action 101b, a weighting factor of a loss function used during training is specified for the training date or the group of training data, taking into account at least one specified criterion. In action 101c, the student model is trained by means of teacher-student training, whereby the loss function is used with the specified weighting factor. Subsequently, action 101d checks whether a specified functional (target) quality of the student model has already been achieved or not.If this is not the case, the process continues with step 101a, starting with the next training date or group of training data. If this is the case, the trained student network is provided, specifically output, in step 102.

[0046] It can be provided that the trained student model 21 (Fig. 1) is loaded into a memory 53 of at least one control unit 52 of a vehicle 50 after training as part of the provision process (measure 103 in Fig. 2). The control unit 52 is part of a device 51 for detecting a hands-on / hands-off state 40 on a steering wheel of a vehicle 50. The device 51 further comprises at least one sensor 54 for detecting at least one steering variable 55 on the steering wheel. The at least one steering variable 55 includes, in particular, a hand torque. The control unit 52 is configured to receive the detected at least one steering variable 55, to provide and apply the trained student model 21 to detect the hands-on / hands-off state 40, and to supply the detected at least one steering variable 55 to an input of the trained student model 21 for this purpose, and to output the hands-on / hands-off state 40 estimated by the trained student model 21.The hands-on / hands-off state 40 can then be fed to another control unit 56 of the vehicle 50 for further processing.

[0047] In particular, it may be provided to operate an assistance system of the vehicle 50 based on the hands-on / hands-off state 40. This may, for example, be a lateral guidance assistant.

[0048] It can be provided that a value of the weighting factor A is determined depending on the criteria based on the respective training date.

[0049] It can be provided that the at least one criterion 11 relates to a ground truth value of the respective training data item, wherein the ground truth includes at least the values ​​"hands-on" or "hands-off." In particular, two possible values ​​for the weighting factor A can then be provided, with either one or the other being selected depending on the ground truth value.

[0050] It can be provided that the weighting factor A is selected such that the student model 21 for the “Hands-Off” state learns more strongly from the at least one teacher model 20.

[0051] It can be provided that the at least one teacher model 20 is more specialized and / or robustified for recognizing the hands-off state as part of the training of the at least one teacher model 20.

[0052] It can be provided that the weighting factor A is set in such a way that the student model 21 for the “Hands-On” state learns more strongly from the training data 10 than from the at least one teacher model 20.

[0053] It can be provided that the at least one criterion 11 relates to a performance variable of the at least one teacher model 20. In particular, a binary cross-entropy between an output of the teacher model 20 and the ground truth can be used as the performance variable. For example, it can be provided that the greater the value of this performance variable, the greater the weighting is placed on the teacher model 20. In other words, the better the at least one teacher model 20 can estimate the hands-on and / or hands-off state, the more the student model 21 learns from the at least one teacher model 20.

[0054] It can be provided that at least one teacher model 20 has several

[0055] It includes 20 teacher models, each trained for a specific context.

[0056] List of reference symbols

[0057] 1 device

[0058] 2 Data processing facility

[0059] 2-1 Calculation device

[0060] 2-2 memory

[0061] 10 training data sets

[0062] 11 specified criterion

[0063] 20 Teacher model

[0064] 21 Student Model

[0065] 40 Hands-On / Hands-Off state

[0066] 50 vehicles

[0067] 51 Device

[0068] 52 Control unit

[0069] 53 storage

[0070] 54 Sensor

[0071] 55 steering size

[0072] 56 additional control unit

[0073] 100-103 Measures of the procedure

[0074] A weighting factor

Claims

Patent claims 1. A method for providing a trained machine learning model for recognizing a hands-on / hands-off state (40) on a steering wheel of a vehicle (50), comprising: training at least one teacher model (20) using a training data set (10), training a smaller student model (21) using the at least one teacher model (20) by means of teacher-student training, using a loss function composed of a weighted sum of a student loss and a distillation loss for this purpose, a weighting factor (A) being determined for each training data item or group of training data items of a training data set (10) used in the teacher-student training, taking into account at least one predetermined criterion (11), providing the trained student model (21).

2. Method according to claim 1, characterized in that the trained student model (21) is loaded into a memory (53) of at least one control unit (52) of a vehicle (50) after the training as part of the provision.

3. Method according to claim 1 or 2, characterized in that a value of the weighting factor (A) is determined as a function of criteria based on the respective training date.

4. Method according to one of the preceding claims, characterized in that the at least one criterion (11) relates to a value of the ground truth of the respective training data, wherein the ground truth comprises at least the values ​​“hands-on” or “hands-off”.

5. The method according to claim 4, characterized in that the weighting factor (A) is selected such that the student model (21) for the "hands-off" state learns more strongly from the at least one teacher model (20).

6. Method according to one of the preceding claims, characterized in that the at least one teacher model (20) is used as part of the training of the at least one Teacher model (20) is more specialized and / or robustified to recognize the hands-off state.

7. Method according to one of the preceding claims, characterized in that the weighting factor (A) is determined such that the student model (21) for the "hands-on" state learns more strongly from the training data (10) than from the at least one teacher model (20).

8. Method according to one of the preceding claims, characterized in that the at least one criterion (11) relates to a performance variable of the at least one teacher model (20).

9. Method according to one of the preceding claims, characterized in that the at least one teacher model (20) comprises a plurality of teacher models (20), each of which is trained for a specific context.

10. A method for detecting a hands-on / hands-off state (40) on a steering wheel of a vehicle (50), comprising: Detecting at least one steering variable (55) on the steering wheel of the vehicle (50), using a student model (21) trained according to one of claims 1 to 9 to detect the hands-on / hands-off state, wherein the detected at least one steering variable (55) is fed to an input of the trained student model (21) for this purpose, Output the hands-on / hands-off state (40) estimated by the trained student model ()21.

11. Device (1) for providing a trained machine learning model for recognizing a hands-on / hands-off state (40) on a steering wheel of a vehicle (50), comprising: a data processing device (2), wherein the data processing device (2) is configured to train at least one teacher model (20) using a training data set (10), to train a smaller student model (21) using the at least one teacher model (20) by means of teacher-student training, wherein a loss function is used for this purpose, which is composed of a weighted sum of a student loss and a distillation loss, and to determine a weighting factor (A) for each training data item or group of training data of a training data set (10) used in teacher-student training, taking into account at least one predetermined criterion (11).

12. Device (51) for detecting a hands-on / hands-off state (40) on a steering wheel of a vehicle (50), comprising: at least one sensor (54) for detecting at least one steering variable (55) on a steering wheel, a control unit (52), wherein the control unit (52) is configured to receive the detected at least one steering variable (55), to provide and use a student model (21) trained according to one of claims 1 to 9 in order to detect the hands-on / hands-off state (40), and to supply the detected at least one steering variable (55) to an input of the trained student model (21) for this purpose, and to output the hands-on / hands-off state (40) estimated by the trained student model (21).

Images