Method and device for detecting a hands-off state on a vehicle's steering wheel
A machine learning model with a main model and adapter model adapts to different vehicle conditions, addressing noise and variability in steering torque detection, reducing training effort and cost for hands-off state recognition.
Patent Information
- Application Number
- DE102024201064
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-02-06
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2044-02-06
AI Technical Summary
Existing steering torque-based detection methods for hands-off states on vehicle steering wheels are prone to noise and variability due to factors like sensor position, friction, road feedback, and vehicle characteristics, requiring extensive training and costly data generation for each vehicle class.
A method using a trained machine learning model with a main model and an adapter model, where the main model is trained on general input data and the adapter model adapts specific input data to the general domain, allowing the main model to process various operating conditions without retraining.
This approach reduces the effort and cost of training machine learning models for different vehicle conditions by using a fixed main model with an adaptable adapter model, enabling efficient hands-off state detection across varying vehicle platforms.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] The invention relates to a method and a device for detecting a hands-off state on the steering wheel of a vehicle. Furthermore, the invention relates to a steering system with such a device, as well as a vehicle with such a device and / or with such a steering system.
[0002] Vehicles use sensors, such as a capacitive steering wheel, to monitor driver activity. Such a steering wheel detects whether the driver is touching or not touching the steering wheel ("hands-off") using a capacitive sensor. This information is then transmitted to relevant functions, such as longitudinal and / or lateral guidance assistance systems. The presence of hands on the steering wheel indicates driver activity and attentiveness. For example, the system might prompt the driver to place their hands on the steering wheel if it detects that their hands have been off the wheel for a predetermined period during lateral guidance.
[0003] To save on the additional costs of a capacitive sensor in the steering wheel, it is known to monitor driver activity using machine learning models, in particular artificial neural networks, based on torque (hand torque) detected at the steering wheel. Such a method is known, for example, from DE 10 2019 211 016 A1.
[0004] A major challenge of steering torque-based detection systems is identifying the driver-induced steering torque within the measured (noisy) steering torque. Many factors can contribute to noisy steering torque, particularly the sensor's position (it typically forms part of the steering gear or power steering system, creating a torsional vibration system via the steering column's elasticity in conjunction with the steering wheel, whose inherent dynamics complicate the precise measurement of driver-induced torque); the degree of friction within the steering system; feedback from road irregularities; the weight of the steering wheel / steering system; and steering wheel vibration caused by an assistance function (e.g., haptic feedback when leaving the lane).
[0005] Furthermore, the characteristics (used for hands-off detection) of the measured steering torque can change due to external influences, e.g., temperature, vehicle load, the presence of a trailer, tire type and / or tire condition, steering system changes over its lifespan, road gradient / slope / incline, etc.
[0006] Furthermore, vehicle characteristics, in particular, influence hands-off detection. This means that, for example, different vehicle platforms, but also vehicle types (e.g., all-wheel drive / front-wheel drive), exhibit different characteristics, i.e., different patterns must be learned by the machine learning model. A separate machine learning model is trained for each vehicle class or vehicle type. This requires the generation and use of extensive training data, resulting in significant effort and costs.
[0007] The invention is based on the objective of improving a method and a device for detecting a hands-off state on a steering wheel.
[0008] The problem is solved according to the invention by a method with the features of claim 1, by a device with the features of claim 8, by a steering system with the features of claim 9, and by a vehicle with the features of claim 10. Advantageous embodiments of the invention are set forth in the dependent claims.
[0009] In particular, a method for detecting a hands-off state on a vehicle's steering wheel is provided, wherein at least one steering parameter is detected on the steering wheel, the detected at least one steering parameter is fed to a trained machine learning model as input data, the machine learning model being trained to detect a hands-off state based on at least the detected at least one steering parameter and to output a corresponding state parameter as output data, the trained machine learning model comprising a trained main model and a trained adapter model upstream of the main model, the main model being trained to detect the hands-off state based on general input data, and the adapter model being trained to determine the general input data based on specific input data.
[0010] Furthermore, a device for detecting a hands-off state on a steering wheel is provided, comprising at least one steering parameter sensor configured to detect at least one steering parameter on the steering wheel, and a data processing device, wherein the data processing device is configured to receive the detected at least one steering parameter, provide a trained machine learning model, and supply the detected at least one steering parameter to the trained machine learning model as input data, wherein the machine learning model is trained to detect a hands-off state based on at least the detected at least one steering parameter and to output associated state information as output data, wherein the trained machine learning model comprises a trained main model and a trained adapter model upstream of the main model, wherein the main model is trained toto detect the hands-off state based on general input data, and wherein the adapter model is trained to determine the general input data based on specific input data.
[0011] The method and the device enable the trained machine learning model to be adapted to different operating conditions with reduced effort. One of the fundamental concepts is that the trained machine learning model comprises a trained main model and a trained adapter model. The main model is trained to recognize the hands-off state based on general input data. The adapter model is trained to determine the general input data based on specific input data. The adapter model can therefore be used to adapt and / or transform specific input data so that it can be processed by the trained main model. In other words, the adapter model can transform specific input data residing in a (specific) data domain into a (general) data domain of the main model.This allows the main model, trained on general input data originating primarily from the general data domain, to also process, or better process, specific input data originating primarily from the specific data domain. One advantage is that the main model only needs to be trained once using a larger training dataset and does not require further modification after training. To enable the main model to handle specific input data from the specific data domain, a suitable trained adapter model is used. This adapter model is placed upstream of the trained main model and transforms the specific input data from the specific data domain into the general input data, which the trained main model can then process with higher efficiency.
[0012] The method and the device reduce the effort required to train and deploy the machine learning model for different operating conditions. In particular, the main model only needs to be trained once. Adaptation to other operating conditions, which involve input data from a changed data domain, is then performed, in particular exclusively, via the adapter model that precedes the main model.
[0013] A steering parameter is, in particular, a parameter that represents and / or describes the current state of the steering wheel. A steering parameter is, in particular, a torque, which is detected, especially by means of a torque sensor on the steering wheel. However, a steering parameter can also be any other parameter detected directly or indirectly at the steering wheel. For example, it might be possible to detect the current in an electric motor at the steering wheel and use it as a steering parameter. Detecting the hands-off state can be based solely on the steering parameter detected at the steering wheel, in particular a detected torque. However, it is also possible for the machine learning model to be provided with other (steering) parameters detected at the steering wheel (e.g., steering wheel angle and / or steering wheel angular velocity, etc.).The trained machine learning model recognizes the hands-off state, taking this additional parameter(s) into account. Parameters not measured at the steering wheel, such as vehicle speed, lateral acceleration, yaw rate, wheel ticks, damper information, and / or other vehicle dynamics parameters, can be considered as contextual information. However, no capacitive sensor is planned for the steering wheel.
[0014] It may be provided that, as part of the detection of the hands-off state, a hands-on state is also detected.
[0015] A hands-off state is, in particular, a state in which the driver does not touch the steering wheel. Specifically, none of the driver's fingers are in contact with the steering wheel. Detecting the hands-off state can include, in particular, providing a hands-off state signal. This could include, for example, a hands-off probability or coded signals for the states "hands-off detected" and "hands-off not detected." A hands-on state is, in particular, a state in which the driver touches the steering wheel. A hands-on / hands-off state can also be provided, for example, as a hands-on / hands-off state signal with, in particular, at least two signal states (e.g., "hands-on detected" or "hands-off detected"). In principle, more than two categories or classes ("hands-off" and "hands-on") can also be distinguished, for example, by differentiating intermediate states.a touch with just a few fingers as opposed to gripping the steering wheel with the whole hand, a grip with one hand, a grip with both hands, etc.
[0016] The machine learning model can, in particular, comprise one or more neural networks. The neural network(s) can, in particular, comprise several internal layers. The machine learning model specifically comprises an artificial recurrent neural network that processes the input data Xt at each time t and has a hands-off probability y. t outputs in [0,1]: yt= p(x t | x 0:t-1 In this case, the recurrent neural network possesses in particular a so-called memory h, in which information from previous time steps is stored and which can be used for output at the current time step.
[0017] The output data of the trained machine learning model is further processed, for example by filtering, before subsequent functions (e.g., a lateral guidance assistant) process it. In particular, it can be provided that, based on a comparison of the hands-off probability with a predefined threshold, a binary hands-off signal is generated (with the two states "hands-off detected" and "hands-off not detected").
[0018] During a training phase, the main model of the machine learning model is trained, particularly in a general data domain, using a large number of training data from this general data domain. The training data comprises pairs in which data from at least one steering variable, in particular torque data, is paired with a hands-off state (as the baseline truth). This training is performed in isolation, without the upstream adapter model. The data from the at least one steering variable, in particular the torque data, are time series of the at least one steering variable measured at the steering wheel, specifically time series of torques measured at the steering wheel. The training data is obtained, in particular, through test drives and / or in simulators for the general data domain.In principle, the provision of training data can be carried out in particular according to the procedure described in DE 10 2019 211 016 A1. Training is carried out in a manner known per se, in particular by means of supervised learning.
[0019] During a training phase of the adapter model, the adapter model to be trained is placed in front of the fully trained main model. The main model is fixed, meaning its parameters and / or weights are no longer changed during the training of the adapter model. In other words, the main model is used only for inference, without being modified during training. The adapter model is trained using training data in a specific data domain. This training data comprises pairs in which data from at least one steering variable, in particular torque data, is paired with a hands-off state (as the baseline). The training data in this specific data domain can be significantly smaller than the training data used to train the main model.The data for at least one steering parameter, in particular the torque data, are specifically time series of the at least one steering parameter measured at the steering wheel, specifically time series of torques measured at the steering wheel. The training data is obtained, in particular, by means of test drives and / or in simulators for the specific data domain. In principle, the provision of training data can be carried out, in particular, according to the procedure described in DE 10 2019 211 016 A1. Training is carried out in a manner known per se, in particular by means of supervised learning. During training, the adapter model estimates the general input data for the trained main model based on the input data of a training date. Based on this, the trained main model provides the hands-off state as output data, which is compared with the baseline value of the training date.Based on a resulting deviation, parameters and / or weights (only) of the adapter model are adjusted.
[0020] An output of the adapter model corresponds, in particular, to an input of the main model. In other words, the output data of the adapter model, with respect to a structure and a dimension, are in particular equal to the input data of the main model; that is, in particular, |y Adapter | = |X|, where y Adapter denote the output data of the adapter model and X the input data of the main model.
[0021] Parts of the device, in particular the data processing unit, can be configured individually or collectively as a combination of hardware and software, for example, as program code executed on a microcontroller or microprocessor. However, it is also possible for parts to be configured individually or collectively as an application-specific integrated circuit (ASIC) and / or a field-programmable gate array (FPGA). The data processing unit specifically comprises at least one computing unit and at least one memory.
[0022] In one embodiment, the trained adapter model is selected based on a vehicle model, a steering model, a vehicle class, and / or at least one vehicle characteristic. This allows the machine learning model to be adapted to the vehicle and its associated operating conditions. Since only a suitable adapter model needs to be selected for adaptation, the process is quick and easy. Alternatively, the adapter model can be trained for the vehicle model, the vehicle class, and / or at least one vehicle characteristic. The training of the adapter model is performed as described above.Once the machine learning model, that is, a main model and an adapter model, has been trained and deployed for a specific vehicle model, it can be easily adapted for a different vehicle model by replacing the adapter model with a more suitable trained adapter model. The at least one attribute can, for example, include a vehicle platform and / or a vehicle body (e.g., all-wheel drive / front-wheel drive).
[0023] In one embodiment, the trained main model is provided in hard-coded form. This enables a solution where the main model can be executed faster and more efficiently. For example, the trained main model can be provided using an ASIC. The parameters and weights of the trained main model are also hard-coded and cannot be changed. Adaptation to the operating conditions can then be achieved using a suitably trained adapter model.
[0024] In one embodiment, the trained adapter model is stored in a dedicated writable non-volatile memory area. This allows the device (e.g., as part of a control unit) to be used for different operating conditions, particularly different vehicle models, vehicle classes, and / or vehicle characteristics. For example, the device may have a hard-coded, trained main model and a writable non-volatile memory area reserved for the respective adapter model. The device can then be configured for the specific operating conditions by storing a suitably trained adapter model in this memory area.
[0025] In one embodiment, the trained main model is designed and / or provided as a recurrent neural network. The recurrent neural network can, for example, be designed as a long short-term memory (LSTM).
[0026] In one embodiment, the trained adapter model is designed and / or provided as a recurrent neural network. This improves the adaptation of the domain from the specific data domain to the general data domain. The recurrent neural network can, for example, be designed as a long short-term memory (LSTM). However, the trained adapter model can also be a fully connected network or a convolutional neural network (CNN).
[0027] In one embodiment, at least one piece of context information is captured and / or received, and this context information is fed to the trained adapter model as input data. The trained adapter model then considers this context information when determining the general input data. This allows for the consideration of an additional context or context information. In particular, this embodiment enables the number of inputs to be increased to accommodate additional context information. This is especially possible even if the trained main model does not consider this additional context information at all, i.e., if it does not have an input for the additional context information. A context or context information refers to...This includes, in particular, characteristics of a situation in which the detection of the hands-off state is to take place and / or in which one or more values of at least one steering parameter have been recorded. Examples of characteristics that can determine a context are: outside temperature, inside temperature, steering wheel vibration, vehicle load and / or weight, the presence of a trailer, the presence of snow chains, cobblestones, potholes, maximum steering interventions (e.g., steering wheel vibration), speed bumps, driver characteristics (e.g., identity, gender, age, weight, hand size, etc.), etc. The current context or at least one piece of contextual information is recognized and / or determined, in particular, based on recorded sensor data. At least one sensor may be provided for this purpose, which is configured to record sensor data related to the context.Alternatively or additionally, it may also be possible to query such sensor data via a vehicle's CAN bus and / or to obtain it from sensors and / or a vehicle control system.
[0028] Further features regarding the design of the device are derived from the description of embodiments of the method. The advantages of the device are the same in each case as in the embodiments of the method.
[0029] Furthermore, a steering system is also created, comprising a device according to one of the described embodiments.
[0030] Furthermore, a vehicle is also created, comprising a steering system according to one of the described embodiments and / or a device according to one of the described embodiments.
[0031] The invention is explained in more detail below with reference to preferred embodiments and the figures. These show: Fig. 1 a schematic representation to illustrate embodiments of the device for detecting a hands-off state on a steering wheel; Fig. 2 a schematic representation to illustrate the machine learning model; Fig. 3 A schematic representation to illustrate a training of the adapter model.
[0032] The Fig. Figure 1 shows a schematic representation of an embodiment of the device 1 for detecting a hands-off state 6 on a steering wheel 51. The device 1 is, in particular, arranged in a vehicle 50 and is part of a steering system 60 therein. The method described in this disclosure is illustrated and explained in more detail below with reference to the device 1.
[0033] The device 1 comprises a steering parameter sensor 2 and a data processing unit 3. The steering parameter sensor 2 is configured to detect a steering parameter 4 at the steering wheel 51 of the vehicle 50. The steering parameter sensor 2 is, for example, a torque sensor, and the steering parameter 4 is torque. Alternatively or additionally, further steering parameter sensors can be provided to detect other steering parameters.
[0034] The data processing unit 3 comprises a computing unit 3-1 and a storage unit 3-2. The computing unit 3-1 is configured to perform the necessary calculations for carrying out the procedures and can access data stored in the storage unit 3-2 for this purpose.
[0035] The data processing unit 3 is designed to obtain the recorded at least one control variable 4, a trained machine learning model 5 (see Fig. 2) to provide and to supply the recorded at least one control variable 4 to the trained machine learning model 5 as input data.
[0036] The machine learning model 5 is trained to recognize a hands-off state 6 based on at least one detected steering variable 4 and to output a corresponding state information as output data 20 ( Fig. 2) to spend.
[0037] A schematic structure of the trained machine learning model 5 is shown in the Fig. Figure 2 illustrates this. The trained machine learning model 5 comprises a trained main model 5-1 and a trained adapter model 5-2 upstream of the main model 5-1. The main model 5-1 is trained to recognize the hands-off state 6 based on general input data 10. The adapter model 5-2 is trained to determine, and in particular estimate, the general input data 10 based on specific input data 11. In other words, time-resolved values (time series) of at least one control variable 4 are supplied to the adapter model 5-2 as specific input data 11. Based on this, the adapter model 5-2 determines the general input data 10, which is then supplied to the trained main model 5-1. An output layer 12 of the trained adapter model 5-2 has the same dimension, i.e., the same number of nodes, as an input layer 13 of the trained main model 5-1.The trained main model 5-1 recognizes the hands-off state 6 in the general input data 10 and outputs corresponding state information as output data 20.
[0038] The hands-off state 6, for example, is sent as a status signal or status information to a control unit 52 ( Fig. 1) of the vehicle 50 for further processing. The control unit 52 can, for example, be a lateral guidance assistant or another assistance system. The status signal or status information can, for example, include a hands-off probability or a binary status value with the two states "hands-off detected" and "hands-off not detected".
[0039] It may be provided that the trained adapter model 5-2, depending on a vehicle model of vehicle 50 ( Fig. 1) and / or a steering model of the vehicle 50 and / or a vehicle class and / or at least one characteristic of the vehicle 50 is selected or is selected. This allows a suitable adapter model 5-2 to be selected for specific operating conditions.
[0040] It may be provided that the trained main model 5-1 is made available in hard-coded form. For example, the main model 5-1 can be made available as an ASIC after training. The device 1 has a corresponding memory 3-2 for this purpose.
[0041] It may be provided that the trained adapter model 5-2 is stored or already stored in a writable non-volatile memory area reserved for this purpose (e.g., in memory 3-2). This allows the device 1 ( Fig. 1) can also be configured for different operating conditions after manufacture. For example, the device 1 can be part of a control unit or form a control unit which is configured before installation in the vehicle 50 by storing an adapter model 5-2 tailored to the vehicle 50 in the memory area.
[0042] It is specifically intended that the trained main model 5-1 is designed and / or provided as a recurrent neural network. The recurrent neural network can, for example, be designed as a long short-term memory (LSTM).
[0043] The trained adapter model 5-2 may be designed and / or provided as a recurrent neural network. The recurrent neural network could, for example, be designed as a long short-term memory (LSTM). However, the trained adapter model could also be a fully connected network or a convolutional neural network (CNN).
[0044] It may be provided that at least one context information 7 is captured and / or received, wherein the at least one context information 7 is supplied to the trained adapter model 5-1 as input data 11, and wherein the trained adapter model 5-2 takes the at least one context information 7 into account when determining the general input data 10.
[0045] The Fig. Figure 3 shows a schematic representation illustrating the training of the adapter model 5-2. The adapter model 5-2 is trained using the already fully trained main model 5-1; that is, the main model 5-1 is applied without adjusting its parameters and weights. The main model 5-1 is fixed with respect to its parameters and weights. The training data includes specific input data, specifically at least one control variable 4, which originates from a specific data domain on which the adapter model 5-2 is to be trained. For each training data point, paired with the at least one control variable 4, there exists a basic truth 30 that contains the actual hands-off state 6. The at least one control variable 4 is fed to the (still untrained) adapter model 5-2.Based on this, the adapter model 5-2 determines (estimates) general input data 10, which are fed to the trained main model 5-1. Starting from the general input data 10, the trained main model 5-1 recognizes (estimates) the hands-off state 6 and outputs it. The hands-off state 6 is compared with the baseline truth 30, and based on a resulting deviation Δ, parameters and weights of the adapter model 5-2 are adjusted by backpropagation. This is repeated with further training data until the deviation Δ falls below a predefined (quality) threshold. Subsequently, the trained machine learning model 5 can be used to operate a vehicle 50 (. Fig.1) to detect the hands-off state 6. In particular, for this purpose, in a device 1 that has already been prepared (e.g. as part of a control unit), only the adapter model 5-2 can be loaded into a memory 3-2 reserved for this purpose, whereby the trained main model 5-1 is already stored in the memory 3-2 or in a hard-coded memory, as described above. Reference symbol list 1 Device 2 Steering parameter sensor 3 Data processing equipment 3-1 Computing Equipment 3-2 storage 4 Steering size 5 trained machine learning models 5-1 trained main model 5-2 trained adapter model 6 Hands-Off State 7 Contextual information 10 general input data 11 special input data 12 Initial layer 13 Entrance shift 20 Source data 30 Fundamental Truths 50 vehicles 51 Steering wheel 52 Control unit 60 Steering system Δ Deviation
Claims
[1] Method for detecting a hands-off state (6) on a steering wheel (51) of a vehicle (50), wherein at least one steering variable (4) is detected on the steering wheel (51), wherein the detected at least one steering variable (4) is fed to a trained machine learning model (5) as input data, wherein the machine learning model (5) is trained to recognize a hands-off state (6) based on at least the detected at least one steering variable (4) and to output an associated state variable as output data (20), wherein the trained machine learning model (5) comprises a trained main model (5-1) and a trained adapter model (5-2) preceding the main model (5-1), where the main model (5-1) is trained to recognize the hands-off state (6) based on general input data (10), and wherein the adapter model (5-2) is trained to determine the general input data (10) based on specific input data (11). [2] Method according to claim 1, characterized by that the trained adapter model (5-2) is or will be selected depending on a vehicle model of the vehicle (50) and / or a steering model of the vehicle (50) and / or a vehicle class and / or at least one property of the vehicle (50). [3] Method according to claim 1 or 2, characterized by that the trained main model (5-1) is provided in hard-coded form. [4] Method according to one of the preceding claims, characterized by that the trained adapter model (5-2) is or is stored in a writable non-volatile memory area reserved for this purpose. [5] Method according to one of the preceding claims, characterized bythat the trained main model (5-1) is designed and / or provided as a recurrent neural network. [6] Method according to one of the preceding claims, characterized by that the trained adapter model (5-2) is designed and / or provided as a recurrent neural network. [7] Method according to one of the preceding claims, characterized by that at least one piece of context information (7) is detected and / or obtained, wherein the at least one piece of context information (7) is supplied to the trained adapter model (5-2) as input data (11), and wherein the trained adapter model (5-2) takes the at least one piece of context information (7) into account when determining the general input data (10). [8] Device (1) for detecting a hands-off state (6) on a steering wheel (51), comprising: at least one steering variable sensor (2) which is designed to detect at least one steering variable (4) on the steering wheel (51), and a data processing device (3), wherein the data processing device (3) is configured to receive the detected at least one steering variable (4), to provide a trained machine learning model (5), and to supply the detected at least one steering variable (4) to the trained machine learning model (5) as input data, wherein the machine learning model (5) is trained to recognize a hands-off state (6) based on at least the detected at least one steering variable (4) and to output associated state information as output data (20), wherein the trained machine learning model (5) comprises a trained main model (5-1) and a trained adapter model (5-2) upstream of the main model (5-1), where the main model (5-1) is trained to recognize the hands-off state (6) based on general input data (10), and wherein the adapter model (5-2) is trained to determine the general input data (10) based on specific input data (11). [9] Steering system (60) comprising a device (1) according to claim 8. [10] Vehicle (50) comprising a steering system (60) according to claim 9 and / or a device (1) according to claim 8.
Citation Information
Patent Citations
Detection of hands-off situations through machine learning
DE102019211016A1
Cited By
Method and device for detecting a hands-off state on a vehicle's steering wheel
DE102024205848A1
