Method and device for recognising a hands-off state on a steering wheel

Abstract

The invention relates to a method for recognising a hands-off state (6) on a steering wheel (51), wherein: a steering variable (4) on the steering wheel (51) is detected; the detected steering variable (4) is supplied to a trained machine learning model (5) as input data (10); the machine learning model (5) is trained to recognise a hands-off state (6) based on at least the detected steering variable (4) and to output this as output data (20); the trained machine learning model (5) comprises at least one core model (8) and a plurality of sub-models (9-x); and a current context (7) is detected and / or recognised and the sub-models (9-x) are activated or deactivated taking into consideration the detected and / or recognised current context (7). The invention also relates to a device (1) for recognising a hands-off state (6) on a steering wheel (51).

Description

[0001] Description

[0002] Method and device for detecting a hands-off state on a steering wheel

[0003] The invention relates to a method and a device for detecting a hands-off state on a steering wheel.

[0004] Sensors such as a capacitive steering wheel are used in vehicles to monitor driver activity. Such a steering wheel detects whether the driver has touched or not touched the steering wheel ("hands-off") using a capacitive sensor. The result is transmitted to the functions being used, such as a longitudinal and / or lateral guidance assistance system. The contact of the hands on the steering wheel can be used to determine driver activity and the driver's level of attention. For example, the system can indicate to the driver to place their hands on the steering wheel if it is detected that their hands have not been on the steering wheel for a specified period of time during lateral guidance.

[0005] To save additional costs for a capacitive sensor in the steering wheel, it is known to monitor driver activity using artificial neural networks based on a torque (hand torque) detected on the steering wheel. One such method is known, for example, from DE 102019211 016 A1. Another method is known from CN 115782892 A.

[0006] The invention is based on the object of improving a method and a device for detecting a hands-off state on a steering wheel.

[0007] The object is achieved according to the invention by a method having the features of patent claim 1 and a device having the features of patent claim 8. Advantageous embodiments of the invention emerge from the subclaims.

[0008] In particular, a method for detecting a hands-off state on a steering wheel is provided, wherein a steering variable on the steering wheel is detected, wherein the detected steering variable is fed to a trained machine learning model as input data, wherein the machine learning model is trained to detect a hands-off state based on at least the detected steering variable and to output it as output data, wherein the trained machine learning model comprises at least one core model and a plurality of sub-models, and wherein a current context is detected and / or recognized and the sub-models are activated or deactivated taking into account the detected and / or recognized current context.

[0009] Furthermore, in particular, a device for detecting a hands-off state on a steering wheel is created, comprising a steering variable sensor which is configured to detect a steering variable on the steering wheel, a control device, wherein the control device is configured to receive the detected steering variable and to provide a trained machine learning model, wherein the machine learning model is trained to detect a hands-off state based on at least the detected steering variable and to output it as output data, and wherein the trained machine learning model comprises at least one core model and a plurality of sub-models, further to feed the detected steering variable to the trained machine learning model as input data, to receive a detected and / or detected current context and to activate or deactivate the sub-models taking into account the detected and / or detected current context.

[0010] The method and the device enable improved detection of a hands-off state. In particular, the method and the device enable improved detection of the hands-off state in different contexts without having to train and maintain a separate machine learning model for each of the contexts. For this purpose, it is provided that the trained machine learning model comprises at least one core model and several sub-models. The machine learning model is in particular composed of the at least one core model and the sub-models. It is in particular provided that each of the sub-models is specialized for a given context and is trained for precisely this context or in precisely this context. The sub-models each fulfill a specialized sub-function within the machine learning model.The submodels therefore form submodels with specific functionality within the machine learning model. According to the method, a current context is captured and / or recognized during application, and the submodels are activated or deactivated taking into account the captured and / or recognized current context. However, at least one core model remains the same in every context. This allows, in particular, the complexity of the machine learning model to be reduced or at least kept constant while increasing functional quality. In particular, the recognition of the hands-off state can be improved without having to maintain a fully trained machine learning model for each context. This can save effort, resources (computing power, storage space), and costs.

[0011] A particular advantage of the method and device is that redundant structures in the trained machine learning model are, or can be, reduced. Context-dependent specialization of the trained machine learning model can be achieved without having to completely train and replace the machine learning model for each context. In particular, the at least one core model remains the same for all contexts.

[0012] A steering variable is, in particular, a variable that represents and / or describes a current state of the steering wheel. A steering variable is, in particular, a torque, which is detected in particular by means of a torque sensor on the steering wheel. In principle, however, a steering variable can also be another variable detected directly or indirectly on the steering wheel. For example, it can be provided to detect a current at an electrical machine on the steering wheel and use it as a steering variable. The hands-off state can be detected exclusively on the basis of the steering variable detected on the steering wheel, in particular a detected torque. However, it is also possible, in particular, for the machine learning model to be provided with further (steering) variables that are detected on the steering wheel (e.g., a steering wheel angle and / or a steering wheel angular velocity, etc.).) and the trained machine learning model recognizes the hands-off state taking this additional variable into account. Furthermore, variables that are not detected at the steering wheel can also be taken into account, such as vehicle speed, lateral acceleration, yaw rate, wheel ticks, damper information, and / or other driving dynamics variables, etc. In particular, however, no capacitive sensor is provided on the steering wheel.

[0013] A hands-off state is, in particular, a state in which the driver is not touching the steering wheel. In particular, none of the driver's fingers are in contact with the steering wheel. Detecting the hands-off state can, in particular, include providing a hands-off state signal. This includes, for example, a hands-off probability or coded signals for the states "hands-off detected" and "hands-off not detected."

[0014] A context refers to or includes, in particular, properties of a situation in which the hands-off state is to be recognized. Examples of properties that can determine a context are: an outside temperature, an inside temperature, a steering wheel vibration, a load and / or weight of the vehicle, the presence of a trailer, characteristics (e.g., identity, gender, age, weight) of the driver, etc. The current context is recognized and / or determined, in particular, based on recorded sensor data. For example, it may be provided to query such sensor data via a vehicle's CAN bus and / or to receive it from sensors and / or a vehicle control system of the vehicle.

[0015] In particular, each context is assigned a configuration of activated submodels and deactivated submodels. It can be provided that there is at least one submodel that is used as a general submodel if no specialized submodel is available for a current context. A subdivision of the contexts and the associated activation or deactivation of the submodels can, for example, be based on expert knowledge. The need for submodels for different contexts or aspects of a context can also be determined based on expert knowledge.

[0016] In a neural network, a submodel comprises at least one layer. It can be provided that multiple levels or layers of sequentially connected submodels are provided, with at least two different submodels being provided for each of the levels or layers. This allows the complexity of different contexts, which include overlapping properties or describe overlapping domains, to be better represented in the machine learning model.

[0017] The machine learning model is designed, in particular, as a neural network and comprises, in particular, several inner layers. The core model comprises, in particular, several inner layers. The submodels also each comprise, in particular, several inner layers. The machine learning model is, in particular, an artificial recurrent neural network that processes the input data X at each time point t. t processed and a hands-off probability y tin [0,1] outputs: y t = p(x t | xo:ti). In particular, the neural network has a so-called memory h, in which information from previous journals is stored and which can be used for the output of the current journal. The output is further processed, for example, by filtering before the decreasing functions (e.g., a lateral guidance assistant) process it. In particular, it can be provided that, based on a comparison of the hands-off probability with a predetermined threshold, a binary hands-off signal is provided (with the two states "hands-off detected" and "hands-off not detected").

[0018] During a training phase, the machine learning model is or was trained, in particular in the different contexts, using training data that includes pairs in which data of the steering variable, in particular torque data, are each paired with a hands-off state. The data of the steering variable, in particular the torque data, are in particular time series of the steering variable recorded at the steering wheel, in particular time series of torque recorded at the steering wheel. The training data is obtained, in particular, with the aid of test drives and / or in simulators. In principle, the provision of training data can in particular be carried out in accordance with the

[0019] The training can be carried out using the methods described in DE 10 2019211 016 A1. Training is carried out in a conventional manner, particularly by means of supervised learning.

[0020] Parts of the device, in particular the control unit, can be implemented individually or collectively as a combination of hardware and software, for example as program code executed on a microcontroller or microprocessor. However, it can also be provided that parts are implemented individually or collectively as an application-specific integrated circuit (ASIC) and / or a field-programmable gate array (FPGA).

[0021] In one embodiment, the submodels are positioned upstream of the at least one core model. In particular, the outputs of the submodels are weighted and fed to an input layer of the at least one core model. The idea behind this is that a context change will primarily affect the first layers of the machine learning model, while the back layers, which perform the actual hands-off detection, are less affected by the context change. Therefore, depending on the context, the upstream submodels are activated or deactivated, and the respective outputs of the submodels are fed to the at least one core model.

[0022] In one embodiment, the current context is recognized using a trained second machine learning model. This allows a configuration of activated and deactivated submodels to be found for each context. The second machine learning model is or has been trained, in particular, using training data comprising pairs in which sensor data paired with a configuration of activated and / or deactivated submodels are linked. The training data can be generated, for example, empirically, for example during test drives, and / or by simulation, e.g., during simulation drives. Depending on the training, an output of the second machine learning model can designate a context or directly contain the configuration (e.g., as a bit sequence, where each bit of the bit sequence indicates an "activated" or "deactivated" state of one of the submodels).

[0023] In one embodiment, the weights of the deactivated submodels are set to zero during inference. This completely eliminates any influence of the deactivated submodels on the results of the trained machine learning model.

[0024] In one embodiment, the deactivated submodels are not calculated during inference. This allows for computational power to be saved during inference.

[0025] In one embodiment, during a training phase of the machine learning model, the weights of submodels that are deactivated in a context are set to zero. This ensures that deactivated submodels do not influence the training of the activated submodels and the core model. The quality of the trained machine learning model can thus be further improved.

[0026] In one embodiment, during a training phase of the machine learning model, parameters of submodels that are deactivated in a context remain fixed. This means that the training has no influence on submodels that are deactivated in the context in which training is currently taking place.

[0027] Further features of the device design will become apparent from the description of embodiments of the method. The advantages of the device are the same as those of the embodiments of the method.

[0028] Furthermore, a steering system is also provided, comprising a device according to one of the described embodiments.

[0029] Furthermore, a vehicle is also provided, comprising a steering system according to one of the described embodiments and / or a device according to one of the described embodiments. The invention is explained in more detail below using preferred embodiments with reference to the figures. Herein:

[0030] Fig. 1 is a schematic diagram illustrating an embodiment of the device for detecting a hands-off state on a steering wheel;

[0031] Fig. 2 is a schematic diagram illustrating a machine learning model of an embodiment of the device and method;

[0032] Fig. 3 is a schematic diagram illustrating the activation and deactivation of the submodels in a current context;

[0033] Fig. 4 is a schematic diagram illustrating training for a context;

[0034] Fig. 5 is a schematic diagram to illustrate the training for another

[0035] Context;

[0036] Fig. 6 is a schematic flow diagram to illustrate an embodiment of the method.

[0037] Fig. 1 shows a schematic representation of an embodiment of the device 1 for detecting a hands-off state on a steering wheel 51. The device 1 is arranged in particular in a vehicle 50. The method described in this disclosure is illustrated and explained in more detail below using the device 1.

[0038] The device 1 comprises a steering variable sensor 2 and a control device 3. The steering variable sensor 2 is configured to detect a steering variable 4 at the steering wheel 51 of the vehicle 50. The steering variable sensor 2 is, for example, a torque sensor, and the steering variable 4 is a torque.

[0039] The control device 3 comprises a computing device 3-1 and a memory 3-2. The computing device 3-1 is configured to perform the computing operations necessary for carrying out measures of the method and can access data stored in the memory 3-2 for this purpose. The control device 3 is configured to receive the detected steering variable 4 and to provide a trained machine learning model 5 (Fig. 2), wherein the machine learning model 5 is trained to recognize a hands-off state 6 based on at least the detected steering variable 4 and to output it as output data 20 (Fig. 2). The trained machine learning model 5 comprises at least one core model 8 and a plurality of sub-models 9-x. The control device 3 is further configured to supply the detected steering variable 4 to the trained machine learning model 5 as input data 10, to receive a detected and / or recognized current context 7, and to process the sub-models 9-x (Fig.2) to activate or deactivate taking into account the detected and / or recognized current context 7.

[0040] The hands-off state 6 is fed, for example, as a state signal to a control unit 52 of the vehicle 50 for further processing. The control unit 52 can be, for example, a lateral guidance assistant or another assistance system. The state signal can, for example, be a hands-off probability or a binary state value with the two states "hands-off detected" and "hands-off not detected."

[0041] For example, it may be provided that, based on the detected and / or recognized context 7, a configuration of activated and deactivated submodels 9-x is determined. This is done in particular by means of the control device 3, which, for example, determines such a configuration for each context 7 using a lookup table stored in the memory 3-2, which contains an assignment of contexts 7 to configurations.

[0042] Figure 2 shows a schematic representation illustrating a machine learning model 5 of an embodiment of the device and method. The machine learning model 5 comprises a core model 8 and several submodels 9-x located upstream of the core model 8. The submodels 9-x can be assigned to a first layer 11-1 and a second layer 11-x arranged between the first layer 11-1 and the core model 8.

[0043] The submodels 9-x can, for example, represent the following properties of a context:

[0044] - Submodel 9-1: a temperature lies within a certain range;

[0045] - Submodel 9-2: there is a steering wheel vibration;

[0046] - Submodel 9-3: a load condition / weight of the vehicle reaches a predefined threshold; - Submodel 9-4: a default model that is activated if no other model in this

[0047] Layer 11-2 is active;

[0048] - Submodel 9-5: a submodel that is active when a trailer is attached to the vehicle.

[0049] The core model 8 comprises, for example, a hands-off state detection logic 8-1 and an output layer 8-2. The output layer 8-2 outputs, in particular, a state signal. This can, for example, comprise a hands-off probability or a binary state value with the two states "hands-off detected" and "hands-off not detected."

[0050] Figure 3 shows a schematic representation illustrating the activation and deactivation of submodels 9-x in a current context. The configuration of machine learning model 5 corresponds to the configuration described with reference to Figure 2. The same reference numerals denote the same features and terms.

[0051] The current context 7 (Fig. 1) includes, for example, that a temperature is within a certain range, that a steering wheel vibration was not detected, and that a payload or a specified loading state / weight of the vehicle is below the specified threshold. Therefore, in this context 7, submodels 9-1 and 9-4 are activated, while submodels 9-2, 9-3, and 9-5 are deactivated. Accordingly, a signal flow through the machine learning model 5 also only occurs via submodels 9-1 and 9-4.

[0052] In particular, it is intended that during an inference, weights of the deactivated submodels 9-2, 9-3 and 9-5 are set to zero, as indicated in Fig. 3.

[0053] In particular, it is further provided that the deactivated submodels 9-2, 9-3, and 9-5 are not calculated during inference, thus saving computing power. In particular, all computational steps that must be performed in the activated state to propagate an input signal through submodel 9-x are not performed in the deactivated state.

[0054] Fig. 4 shows a schematic representation to illustrate training for a context. The context should correspond to the context shown in Fig. 3. The context includes that a temperature is within a certain range, that a steering wheel vibration was not detected, and that a payload or a specified loading state / weight of the vehicle is below the specified threshold. During training, exactly the same submodels 9-x are activated and deactivated as in the inference or application phase (following training), therefore Fig. 4 corresponds to Fig. 3. During the training phase, only the path that includes submodels 9-1 and 9-4 and core model 8 is used. Core model 8, however, remains the same as in other configurations. Only parameters in this path are changed during training.

[0055] In particular, it is provided that during a training phase of the machine learning model 5, the weights of submodels 9-x that are deactivated in a context are set to zero. In the example shown in Fig. 4, the weights of submodels 9-2, 9-3, and 9-5 are set to zero and do not affect submodel 9-4 and core model 8. In particular, this does not affect the training.

[0056] Furthermore, it is specifically provided that during the training phase of the machine learning model 5, parameters of submodels 9-x that are deactivated in a context remain fixed. In particular, the parameters of submodels 9-2, 9-3, and 9-5 are not changed by training in the context shown; therefore, submodels 9-2, 9-3, and 9-5 remain fixed.

[0057] Fig. 5 shows a schematic representation to illustrate training for a different context. The different context includes a temperature being within a certain range, a steering wheel vibration being detected, a payload or a specified loading state / weight of the vehicle being below the specified threshold, and a trailer being present. Accordingly, in this context, submodels 9-1, 9-2, and 9-5 are activated, while submodels 9-3 and 9-4 are not. The core model 8, however, remains the same as in other configurations. Training during the training phase takes place accordingly in this configuration. During a subsequent inference or application phase, the same submodels 9-x are activated or deactivated.

[0058] It can be provided that the current context 7 (Fig. 1) is recognized by means of a trained second machine learning model 12 (Fig. 1). For example, sensor data 13 from sensors (not shown) of the vehicle 50 are fed to the trained second machine learning model 12. The detected steering variable 4, for example a detected torque, can also be fed to the second machine learning model 12. The trained second machine learning model 12 estimates a context 7 based on the sensor data 13. Based on the estimated context 7, a configuration of activated and deactivated sub-models is then defined. This is done in particular by means of the control device 3, which, for example, defines such a configuration for each context 7 using a lookup table stored in the memory 3-2. Alternatively, the second machine learning model 12 can also estimate the configuration directly (e.g., as a bit sequence).

[0059] Fig. 6 shows a schematic flow diagram to illustrate an embodiment of the method for detecting a hands-off state on a steering wheel.

[0060] In a measure 100, a steering variable, for example a torque, is detected on a steering wheel by means of a steering variable sensor, for example a torque sensor.

[0061] In a measure 102, the detected torque is fed as input data to a trained machine learning model, wherein the machine learning model is trained to detect a hands-off state based on at least the detected steering variable and to output it as output data. The trained machine learning model comprises at least one core model and several submodels. In particular, this occurs continuously for several points in time. In particular, a hands-off state is detected and output continuously, i.e., for several points in time.

[0062] In an upstream measure 101, a current context is captured and / or recognized and the submodels are activated or deactivated taking into account the captured and / or recognized current context.

[0063] Subsequently, measures 100 to 102 are repeated in particular.

[0064] device

[0065] Steering size sensor

[0066] Control device -1 Computing device -2 Memory

[0067] Steering size

[0068] Machine learning model

[0069] Hands-off state

[0070] context

[0071] Core model -1 Hands-off condition detection logic -2 Output layer -x Submodel 0 Input data 1-x Layer 2 Second machine learning model 3 Sensor data 0 Output data 0 Vehicle 1 Steering wheel 0 Steering system 00-102 Measures of the procedure

Claims

Patent claims 1. Method for detecting a hands-off state (6) on a steering wheel (51), wherein a steering variable (4) on the steering wheel (51) is detected, wherein the detected steering variable (4) is fed to a trained machine learning model (5) as input data (10), wherein the machine learning model (5) is trained to detect a hands-off state (6) based on at least the detected steering variable (4) and to output it as output data (20), wherein the trained machine learning model (5) comprises at least one core model (8) and a plurality of sub-models (9-x), and wherein a current context (7) is detected and / or recognized and the sub-models (9-x) are activated or deactivated taking into account the detected and / or recognized current context (7).

2. Method according to claim 1, characterized in that the submodels (9-x) are connected upstream of the at least one core model (8).

3. Method according to claim 1 or 2, characterized in that the current context (7) is recognized by means of a trained second machine learning model (12).

4. Method according to one of the preceding claims, characterized in that during an inference, weights of the deactivated submodels (9-x) are set to zero.

5. Method according to one of the preceding claims, characterized in that the deactivated submodels (9-x) are not calculated during an inference.

6. Method according to one of the preceding claims, characterized in that during a training phase of the machine learning model (5), weights of submodels (9-x) that are deactivated in a context (7) are set to zero.

7. Method according to one of the preceding claims, characterized in that during a training phase of the machine learning model (5), parameters of submodels (9-x) that are deactivated in a context (7) remain fixed.

8. A device (1) for detecting a hands-off state (6) on a steering wheel (51), comprising: a steering variable sensor (2) configured to detect a steering variable (4) on the steering wheel (51), a control device (3), wherein the control device (3) is configured to receive the detected steering variable (4), provide a trained machine learning model (5), wherein the machine learning model (5) is trained to detect a hands-off state (6) based on at least the detected steering variable (4) and output it as output data (20), and wherein the trained machine learning model (5) comprises at least one core model (8) and a plurality of submodels (9-x), supply the detected steering variable (4) to the trained machine learning model (5) as input data (10), and obtain a detected and / or recognized current context (7),and to activate or deactivate the submodels (9-x) taking into account the detected and / or recognized current context (7).

9. Steering system (60) comprising a device (1) according to claim 8.

10. A vehicle (50) comprising a steering system (60) according to claim 9 and / or a device (60) according to claim 8.

Images