Method and device for detecting hand release state at steering wheel of vehicle

Abstract

The invention relates to a method for detecting a hand-out state at a steering wheel of a vehicle, in which at least one steering variable at the steering wheel is detected, in which the detected at least one steering variable is supplied as input data to a trained machine learning model, in which the machine learning model is trained for detecting a hand-out state at the steering wheel of the vehicle. A hand-out state is identified starting from at least one steering variable obtained and a state variable dependent on the hand-out state is output as output data, and wherein the trained machine learning model comprises a trained master model and a trained adaptive model upstream of the master model, the master model being trained for the following purposes: the adaptive model is adapted to the master model, and the adaptive model is adapted to the master model. The hand-out state is identified starting from generic input data, and wherein the adaptation model is trained for determining generic input data starting from special input data. The invention further relates to a device for detecting a hand release state on a steering wheel.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for identifying a hands-off state at a steering wheel of a vehicle. Background Art

[0002] Sensors, such as capacitive steering wheels, are used in vehicles to monitor driver activity. These steering wheels use capacitive sensors to detect whether the driver is touching or not touching the steering wheel ("hands off"). The results are transmitted to the functions used, such as longitudinal and / or lateral steering assistance systems. Inferences about the driver's activity and attentiveness can be drawn from the hand contact on the steering wheel. For example, if it is detected that the hands are not on the steering wheel for a predetermined period of time during lateral control, the driver is prompted to place their hands on the steering wheel.

[0003] In order to save the additional costs for capacitive sensors in the steering wheel, it is known to monitor the driver's activity based on the torque (hand torque) detected at the steering wheel using a machine learning model, in particular an artificial neural network. Such a method is known, for example, from DE 10 2019 211 016 A1.

[0004] A major challenge in steering torque recognition is distinguishing the driver-induced steering torque from the measured (noisy) steering torque. Many factors can contribute to noisy steering torque, particularly the sensor's position (which typically forms part of the steering gear or power steering system, thereby forming a rotationally vibrating system in conjunction with the steering wheel via the steering column's elasticity, whose inherent dynamics make accurate measurement of the driver-induced torque difficult); the amount of friction in the steering system; counterexcitation from road unevenness; the weight of the steering wheel / steering system; and steering wheel vibrations due to assistance functions (e.g., haptic feedback when leaving the lane).

[0005] Furthermore, the characteristics of the measured steering torque (which are used for hands-off detection) may be modified by external influences, such as temperature, the load state of the vehicle, the presence of a trailer, tire type and / or tire condition, changes in the steering system over its service life, lane gradient / slope / inclined position, etc.

[0006] Furthermore, vehicle characteristics also have an impact on hands-off detection. This dictates that, for example, different vehicle platforms and drive configurations (e.g., all-wheel drive / front-wheel drive) have different characteristics, meaning that different patterns can be learned by the machine learning model. A separate machine learning model is trained for each vehicle type or each drive configuration. This requires the generation and use of extensive training data, which results in significant effort and costs. Summary of the Invention

[0007] The present invention is based on the object of improving a method and a device for detecting a hands-off state at the steering wheel.

[0008] According to the invention, this object is achieved by a method according to the invention and a device according to the invention. Advantageous embodiments of the invention are disclosed in the dependent claims.

[0009] In particular, a method for identifying a hands-off state at a steering wheel of a vehicle is made available, wherein at least one steering parameter at the steering wheel is acquired, wherein the at least one acquired steering parameter is supplied as input data to a trained machine learning model, wherein the machine learning model is trained to identify the hands-off state based on at least the at least one acquired steering parameter and output subordinate state parameters as output data, wherein the trained machine learning model includes a trained main model and a trained adaptation model upstream of the main model, wherein the main model is trained to identify the hands-off state based on general input data, and wherein the adaptation model is trained to determine general input data based on specific input data.

[0010] In addition, a device for identifying a hands-off state at a steering wheel is particularly created, comprising at least one steering parameter sensor, which is configured to acquire at least one steering parameter at the steering wheel, and a data processing device, wherein the data processing device is configured to obtain at least one acquired steering parameter, provide a trained machine learning model and supply at least one acquired steering parameter as input data to the trained machine learning model, wherein the machine learning model is trained to identify the hands-off state based on at least at least one acquired steering parameter and output subordinate state information as output data, wherein the trained machine learning model comprises a trained main model and a trained adaptation model upstream of the main model, wherein the main model is trained to identify the hands-off state based on general input data, and wherein the adaptation model is trained to determine general input data based on specific input data.

[0011] The method and device make it possible to adapt a trained machine learning model to different conditions of use with reduced effort. One of the basic concepts is that the trained machine learning model includes a trained main model and a trained adaptation model. The main model is trained to detect a hands-off state starting from universal input data. The adaptation model is trained to determine universal input data starting from specific input data. The adaptation model can therefore be used to adapt and / or transform the specific input data so that it can be processed by the trained main model. In other words, the adaptation model can transform the specific input data in a (specific) data domain from this (specific) data domain into the (general) data domain of the main model. As a result, the main model trained for universal input data (which in particular originate from the universal data domain) can also process or better process specific input data (which in particular originate from the specific data domain). An advantage of this is that the main model can be trained once with the aid of a larger training data set and does not need to be modified after this training. In order to be able to use the main model also for special input data of a special data domain, a suitable, trained adaptation model is used upstream of the trained main model and which transfers the special input data of the special data domain into universal input data that can be processed with higher quality by the trained main model.

[0012] This method and apparatus reduces the effort involved in training and adapting the machine learning model to different usage conditions. In particular, the master model only needs to be trained once. Adapting the input data to different usage conditions with varying data domains is then accomplished solely via an adaptation model upstream of the master model.

[0013] The steering variable is in particular a variable that represents and / or describes the current state of the steering wheel. The steering variable is in particular a torque, which is in particular acquired by means of a torque sensor at the steering wheel. In principle, the steering variable can also be other variables acquired directly or indirectly at the steering wheel. For example, it can be provided that the current at the motor at the steering wheel is acquired and used as the steering variable. The recognition of the hands-off state can be achieved solely based on the steering variable acquired at the steering wheel, in particular the acquired torque. However, it is also possible that the machine learning model is supplied with other (steering) variables acquired at the steering wheel (such as the steering wheel angle and / or steering wheel angular velocity, etc.), and the trained machine learning model recognizes the hands-off state while taking into account these other variables. Variables that are not acquired at the steering wheel (such as vehicle speed, lateral acceleration, yaw rate, wheel speed, shock absorber information and / or other driving dynamics variables, etc.) can be considered as background information. In particular, however, no capacitive sensor is provided at the steering wheel.

[0014] Provision can be made such that within the scope of the detection of the hands-off state (Hands-On-Zustand), the hands-on state (Hands-On-Zustand) is also detected.

[0015] The hands-off state is, in particular, a state in which the steering wheel is not touched by the driver. In particular, the driver's fingers do not come into contact with the steering wheel. The recognition of the hands-off state can, in particular, comprise the provision of a hands-off state signal. This comprises, for example, a hands-off probability or a coded signal for the states "hands-off detected" and "hands-off not detected". The gripping state is, in particular, a state in which the steering wheel is touched by the driver. The gripping / hands-off state can also be provided, for example as a gripping / hands-off state signal with, in particular, at least two signal states (for example, "grip detection" or "hands-off detection"). In principle, more than two categories or levels ("hands-off" or "grip") can also be distinguished, for example by distinguishing between intermediate levels, such as touching with only a few fingers, compared to gripping the steering wheel with the whole hand, gripping with one hand, gripping with both hands, etc.

[0016] The machine learning model may in particular comprise one or more neural networks. The one or more neural networks may in particular comprise a plurality of inner layers. The machine learning model may in particular comprise an artificial recurrent neural network which processes the input data X at any time t t And the output is the probability of selling y in [0,1] t :y t =p(x t |x 0:t-1 ). In this case, the recurrent neural network has in particular a so-called memory h, in which information from previous time steps is stored and can be used for the output in the current time step.

[0017] The output data of the trained machine learning model is further processed, for example by filtering, before being processed by a reduction function (e.g., lateral control assistance). In particular, it can be provided that a binary hands-off signal (with the two states "hands-off detected" and "hands-off not detected") is provided based on a comparison of the hands-off probability with a predefined threshold value.

[0018] During the training phase, the main model of the machine learning model is trained or has been trained, in particular in a common data domain with the aid of a large amount of training data from this common data domain, wherein the training data accordingly include data pairs in which data of at least one steering variable (in particular torque data) are paired with each hands-off state (as a true value). In this case, the training is carried out in particular in isolation without an upstream adaptation model. The data of at least one steering variable, in particular torque data, are in particular a time series of at least one steering variable acquired at the steering wheel, in particular a time series of torque acquired at the steering wheel. The training data are obtained in particular by means of test drives and / or in a simulator for the common data domain. In principle, the provision of the training data can be carried out in particular according to the method described in DE 10 2019 211016A1. The training is carried out in a manner known per se, in particular by means of supervised learning.

[0019] During the training phase of the adaptive model, the adaptive model to be trained is trained upstream of the fully trained main model. The main model is fixed, meaning its parameters and / or weights do not change during the training of the adaptive model. In other words, only inference is performed using the main model, which is then adapted during the training phase. The adaptive model is trained based on training data from a special data domain. The training data accordingly includes data pairs in which data of at least one steering variable, particularly torque data, are paired with a respective hands-off state (as a true value). The training data for the special data domain can be significantly smaller in scope than the training data used to train the main model. The data of at least one steering variable, particularly torque data, is particularly a time series of at least one steering variable acquired at the steering wheel, particularly a time series of torque acquired at the steering wheel. The training data is acquired, in particular, through test drives and / or in a simulator specific to the special data domain. In principle, the provision of the training data can be implemented, in particular, according to the method described in DE 10 2019 211 016 A1. This training is carried out in a known manner, in particular using supervised learning. During training, the adaptive model estimates the general input data for the trained master model based on the input data of the training data. The trained master model then provides the hands-off state as output data, which is compared with the true values of the training data. Based on the resulting deviations, (only) the parameters and / or weights of the adaptive model are adapted.

[0020] The output of the adaptation model corresponds in particular to the input of the main model. In other words, the output data of the adaptation model are in particular identical to the input data of the main model in terms of structure and dimensions, ie the following applies in particular: Adapter |=|X|, where y Adapter represents the output data of the adapted model and X represents the input data of the main model.

[0021] The components of the apparatus, in particular the data processing device, can be designed individually or in combination as a combination of hardware and software, for example as program code executed on a microcontroller or microprocessor. However, it is also possible for these components, individually or in combination, to be designed as an application-specific integrated circuit (ASIC) and / or a field-programmable gate array (FPGA). The data processing device particularly includes at least one computing device and at least one memory.

[0022] In one embodiment, the trained adaptation model is selected or selected as a function of the vehicle model of the vehicle and / or the steering model of the vehicle and / or the vehicle type and / or at least one characteristic of the vehicle. The machine learning model can thus be adapted to the vehicle and the associated conditions of use. Adaptation can be achieved with relatively little effort, since only one suitable adaptation model must be selected for the adaptation. It can also be provided that the adaptation model is trained for the vehicle model and / or the vehicle type and / or at least one characteristic of the vehicle. The training of the adaptation model is achieved as previously described. If the machine learning model (i.e., the main model and the adaptation model) has already been trained and used, for example, for a vehicle model, this machine learning model can be easily adapted for another vehicle model by being replaced by a more suitable, trained adaptation model. The at least one characteristic can, for example, include the vehicle platform and / or the vehicle drive configuration (e.g., all-wheel drive / front-wheel drive).

[0023] In one embodiment, the trained master model is provided in hard-coded form. This enables a solution in which the master model can be implemented more quickly and resource-efficiently. For example, the trained master model can be provided via an ASIC. Likewise, the parameters and weights of the trained master model are hard-coded and cannot be modified. Adaptation to the usage conditions can then be achieved using a correspondingly trained adaptation model.

[0024] In one embodiment, the trained adaptation model is or has been stored in a writable non-volatile memory area reserved for this purpose. This allows the device (e.g., as part of a controller) to be used for different operating conditions, in particular different vehicle models, vehicle types, and / or vehicle characteristics. For example, the device can be configured to have a hard-coded, trained master model and a writable non-volatile memory area reserved for the corresponding adaptation model. The device can then be configured for the corresponding operating conditions by storing the appropriate trained adaptation model in this memory area.

[0025] In one embodiment, it is provided that the trained main model is designed and / or provided as a recurrent neural network. The recurrent neural network can be designed, for example, as a long short-term memory network (LSTM).

[0026] In one embodiment, the trained adaptation model is designed and / or provided as a recurrent neural network. This allows for improved adaptation from a specific data domain to a universal data domain. The recurrent neural network can be designed, for example, as a long short-term memory network (LSTM). In principle, the trained adaptation model can also be a fully connected network or a convolutional neural network (CNN).

[0027] In one embodiment, it is provided that at least one piece of background information is acquired and / or obtained, wherein the at least one piece of background information is supplied as input data to a trained adaptation model, and wherein the trained adaptation model takes the at least one piece of background information into account when determining the general input data. Thus, additional background or background information can also be taken into account. In particular, this embodiment makes it possible to increase the number of inputs in order to take into account the additional background information. In particular, this is possible even when the trained main model does not take such additional background information into account at all, that is, does not have an input for this additional background information. The background or background information in particular represents or includes characteristics of a situation in which a hands-off state is to be detected and / or in which one or more values of at least one steering variable are acquired. Examples of characteristics that can determine the background include: external temperature, internal temperature, steering wheel vibrations, the load and / or weight of the vehicle, the presence of a trailer, the presence of snow chains, cobblestones, potholes, maximum steering maneuvers (e.g., steering vibrations), speed bumps on the road for speed limits (speed bumps in English), characteristics of the driver (e.g., identity, gender, age, weight, hand size, etc.). The current background or at least one piece of background information is identified and / or determined, in particular, based on acquired sensor data. To this end, at least one sensor can be provided that is configured to acquire sensor data related to the background. Furthermore, it can be alternatively or additionally provided that, during application, such sensor data is queried via the vehicle's CAN bus and / or obtained from sensors of the vehicle and / or a vehicle control unit.

[0028] The other features of the embodiment of the device are derived from the description of the embodiment of the method. Here, the advantages of the device are correspondingly the same advantages as in the case of the embodiment of the method.

[0029] Furthermore, a steering system is also provided, which comprises a device according to one of the described embodiments.

[0030] Furthermore, a vehicle is also provided, which comprises a steering system according to one of the described embodiments and / or a device according to one of the described embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will now be explained in more detail with reference to the accompanying drawings using preferred exemplary embodiments.

[0032] Figure 1 A schematic diagram for explaining an embodiment of a device for detecting a hands-off state at a steering wheel is shown;

[0033] Figure 2 A schematic diagram illustrating a machine learning model is shown;

[0034] Figure 3 A schematic diagram illustrating the training of the adapted model is shown. DETAILED DESCRIPTION

[0035] Figure 1 A schematic illustration of an embodiment of a device 1 for detecting a hands-off state 6 at a steering wheel 51 is shown. The device 1 is arranged in particular in a vehicle 50 and is part of a steering system 60. The method described in this disclosure is subsequently described and explained in more detail using the device 1.

[0036] Device 1 includes a steering variable sensor 2 and a data processing device 3. Steering variable sensor 2 is configured to detect a steering variable 4 at a steering wheel 51 of a vehicle 50. Steering variable sensor 2 is, for example, a torque sensor, and steering variable 4 is torque. In principle, further steering variable sensors may alternatively or additionally be provided to detect further steering variables.

[0037] The data processing device 3 comprises a computing device 3 - 1 and a memory 3 - 2 . The computing device 3 - 1 is set up to perform the computing operations necessary for carrying out the steps of the method and, for this purpose, can access data stored in the memory 3 - 2 .

[0038] The data processing device 3 is configured to obtain at least one steering variable 4, provide a trained machine learning model 5 (see Figure 2 ) and supplies the acquired at least one steering parameter 4 as input data to the trained machine learning model 5.

[0039] The machine learning model 5 is trained to recognize the hands-off state 6 based on at least one acquired steering variable 4 and to output data 20 ( Figure 2 ) outputs slave status information.

[0040] The structure of the trained machine learning model 5 is Figure 2 . The trained machine learning model 5 includes a trained main model 5-1 and a trained adaptation model 5-2 upstream of the main model 5-1. The main model 5-1 is trained to identify the hands-off state 6 starting from universal input data 10. The adaptation model 5-2 is trained to determine, in particular estimate, the universal input data 10 from special input data 11. In other words, the time-resolved value (time series) of at least one steering variable 4 is supplied to the adaptation model 5-2 as special input data 11. Starting from this, the adaptation model 5-2 determines the universal input data 10, which is supplied to the trained main model 5-1. To this end, the output layer 12 of the trained adaptation model 5-2 has the same dimension or the same number of nodes as the input layer 13 of the trained main model 5-1. The trained main model 5-1 identifies the hands-off state 6 in the universal input data 10 and outputs the corresponding state information as output data 20.

[0041] The hands-off state 6 is supplied to the controller 52 ( Figure 1 ) for further processing. Controller 52 may, for example, be a lateral control assistance device or another assistance system. The status signal or status information may, for example, include a hands-off probability or a binary status value with two states: "hands-off recognized" and "hands-off not recognized."

[0042] It can be set as follows, that is, the trained adaptation model 5-2 depends on the vehicle 50 ( Figure 1 ) and / or the steering model of the vehicle 50 and / or the vehicle type and / or at least one characteristic of the vehicle 50. Thus, a suitable adaptation model 5-2 can be selected for specific usage conditions.

[0043] It can be provided that the trained master model 5-1 is provided in hard-coded form. For example, the master model 5-1 can be provided as an ASIC after training. For this purpose, the device 1 has a corresponding memory 3-2.

[0044] It can be provided that the trained adaptation model 5-2 is stored or has been stored in a writable non-volatile storage area reserved for this purpose (for example in the memory 3-2). Figure 1) can also be set up for different usage conditions after manufacturing. For example, the device 1 can be part of a controller or a component controller that is set up before being installed in the vehicle 50 by storing the adaptation model 5-2 customized for the vehicle 50 in the storage area.

[0045] In particular, it is provided that the trained main model 5 - 1 is designed and / or provided as a recurrent neural network. This recurrent neural network can be designed, for example, as a long short-term memory network (LSTM).

[0046] It can be provided that the trained adaptation model 5-2 is designed and / or provided as a recurrent neural network. This recurrent neural network can be designed, for example, as a long short-term memory network (LSTM). In principle, the trained adaptation model can also be a fully connected network or a convolutional neural network (CNN).

[0047] It can be provided that at least one background information 7 is acquired and / or obtained, wherein the at least one background information 7 is supplied as input data 11 to the trained adaptation model 5-1, wherein the trained adaptation model 5-2 takes the at least one background information 7 into account when determining the general input data 10.

[0048] Figure 3 A schematic diagram is shown for illustrating the training of the adaptation model 5-2. The training of the adaptation model 5-2 is carried out using the fully trained main model 5-1, that is, the main model 5-1 is only used, but its parameters and weights are not adapted. The main model 5-1 is fixed in particular with respect to the parameters and weights. The training data includes specific input data, that is, in particular at least one steering variable 4 from a special data domain for which the adaptation model 5-2 is to be trained. For each training data, there is a corresponding true value 30 paired with at least one steering variable 4, which contains the actually existing hands-off state 6. The at least one steering variable 4 is supplied to the (untrained) adaptation model 5-2. Proceeding from this, the adaptation model 5-2 determines (estimates) universal input data 10, which are supplied to the trained main model 5-1. Starting from the universal input data 10, the trained main model 5-1 recognizes (estimates) the hands-off state 6 and outputs it. The hands-off state 6 is compared with the true value 30 and, starting from the resulting deviation Δ, the parameters and weights of the adaptation model 5-2 are adapted by backpropagation. This is repeated with further training data until the deviation Δ falls below a predefined (quality) threshold. The trained machine learning model 5 can then be used to control the vehicle 50 ( Figure 1 ) in the hands-off state 6. In particular, for this purpose, in a prepared device 1 (e.g. as part of a controller), only the adaptation model 5-2 can be loaded into the memory 3-2 reserved for this purpose, wherein the trained main model 5-1 is already stored in the memory 3-2 or hard-coded memory, as described above.

[0049] Reference Signs List

[0050] 1 device

[0051] 2 Steering parameter sensor

[0052] 3 Data processing equipment

[0053] 3-1 Computing Equipment

[0054] 3-2 Memory

[0055] 4 Steering parameters

[0056] 5 Trained Machine Learning Models

[0057] 5-1 Trained Main Model

[0058] 5-2 Trained Adaptation Model

[0059] 6 Hands-off state

[0060] 7 Background Information

[0061] 10 General input data

[0062] 11 Special input data

[0063] 12 Output Layer

[0064] 13 Input Layer

[0065] 20 Output data

[0066] 30 truth value

[0067] 50 vehicles

[0068] 51 Steering Wheel

[0069] 52 Controller

[0070] 60 Steering system

[0071] Δ Deviation

Claims

1. A method for identifying a hands-off state (6) at a steering wheel (51) of a vehicle (50), in, At least one steering parameter (4) is acquired at the steering wheel (51), wherein the acquired at least one steering parameter (4) is supplied as input data to a trained machine learning model (5), The machine learning model (5) is trained to identify a hands-off state (6) based on at least one acquired steering variable (4) and to output a corresponding state variable as output data (20). The trained machine learning model (5) includes a trained main model (5-1) and a trained adaptation model (5-2) upstream of the main model (5-1). The main model (5-1) is trained to identify the hands-off state (6) based on general input data (10), and The adaptation model (5-2) is trained to determine universal input data (10) starting from specific input data (11).

2. The method according to claim 1, characterized in that The trained adaptation model (5-2) is selected or is selected depending on a vehicle model of the vehicle (50) and / or a steering model of the vehicle (50) and / or a vehicle class and / or at least one characteristic of the vehicle (50).

3. The method according to claim 1 or 2, characterized in that The trained main model (5-1) is provided in hard-coded form.

4. The method according to any one of the preceding claims, characterized in that The trained adaptation model (5-2) is stored or has been stored in a writable non-volatile memory area reserved for this purpose.

5. The method according to any one of the preceding claims, characterized in that The trained main model (5-1) is designed and / or provided as a recurrent neural network.

6. The method according to any one of the preceding claims, characterized in that The trained adaptation model (5-2) is designed and / or provided as a recurrent neural network.

7. The method according to any one of the preceding claims, characterized in that At least one piece of background information (7) is acquired and / or obtained, wherein the at least one piece of background information (7) is supplied as input data (11) to a trained adaptation model (5-2), and wherein the trained adaptation model (5-2) takes the at least one piece of background information (7) into account when determining generic input data (10).

8. A device (1) for identifying a hands-off state (6) at a steering wheel (51), comprising: at least one steering variable sensor (2) configured to acquire at least one steering variable (4) at the steering wheel (51); and A data processing device (3), wherein the data processing device (3) is configured to obtain at least one acquired steering variable (4), to provide a trained machine learning model (5) and to supply the at least one acquired steering variable (4) as input data to the trained machine learning model (5), wherein the machine learning model (5) is trained to recognize a hands-off state (6) based on at least the at least one acquired steering variable (4) and to output associated state information as output data (20), wherein the trained machine learning model (5) comprises a trained main model (5-1) and a trained adaptation model (5-2) upstream of the main model (5-1), The main model (5-1) is trained to identify the hands-off state (6) based on general input data (10), and The adaptation model (5-2) is trained to determine universal input data (10) starting from specific input data (11).

9. A steering system (60) comprising the device (1) according to claim 8.

10. A vehicle (50) comprising a steering system (60) according to claim 9 and / or a device (1) according to claim 8.

Images