Detection of hands-off situations through machine learning

Abstract

Method for operating a driver assistance system (48) in which a hands-off situation (54) in a final product vehicle (46) is detected on the basis of a learning algorithm (44), comprising the following steps: - driving the final product vehicle (46) equipped with the driver assistance system (48) which sensorily detects steering torque data (16) during driving; - Detection of hands-off situations (54) from the measured steering torque data (16) using a trained learning algorithm (52), characterized in that the learning algorithm (44) has been trained in the end product vehicle (46) to the trained learning algorithm (52) using training data (40) that have been created by means of a development vehicle (10).

Description

[0001] The invention relates to a method for operating a driver assistance system according to the preamble of patent claim 1, a method for creating training data according to the preamble of patent claim 2, a data carrier according to the preamble of patent claim 4, a method for training an adaptive algorithm according to the preamble of patent claim 5, a data carrier according to the preamble of patent claim 6, a driver assistance system according to the preamble of patent claim 8 and a driver assistance system according to the preamble of patent claim 9 and a vehicle according to the preamble of patent claim 10.

[0002] In the technical field of vehicles, especially motor vehicles, monitoring a driver's steering activity is a well-known task. A wide variety of safety-related conclusions can be drawn from steering activity. A particularly important aspect is the detection of hands-off situations, i.e., moments in which the driver does not have their hands on the steering wheel.

[0003] It is known, for example, to evaluate steering torque curves and analytically examine them for indications that a hands-off situation is present. However, such approaches are only partially reliable and accurate.

[0004] It is also known to use capacitive distance sensors, for example, to directly measure the hands-off situation. Such solutions are highly accurate, but also involve considerable technical effort.

[0005] US Pat. No. 9,604,649 B1, for example, describes a method for checking whether a driver has his or her hands on the steering wheel of a vehicle. To do this, a disturbance signal, such as a vibration, is generated at the steering wheel and a steering torque is measured.

[0006] DE 10 2008 042 277 A1 describes a method in which steering torque curves are examined using a machine learning approach, for example a neural network, for special properties that indicate a hand-on-the-steering-wheel or hand-not-on-the-steering-wheel situation.

[0007] From US 2004 / 0 039 509 A1 a system for controlling an occupant restraint system is known that works with a machine learning approach.

[0008] Furthermore, DE 10 2017 216 887 A1 discloses a method for detecting the contact of hands with the steering wheel of a vehicle, wherein the steering wheel is assigned exactly one touch sensor for detecting the contact of hands with the steering wheel.

[0009] However, such machine-learning-based solutions suffer from the weakness of the production and validity of the training data. The production of the training data is extremely complex, as an extremely large number of possible steering torque curves must be linked with the information as to whether or not they are based on a hands-off situation. This is also referred to as "labeling" the steering torque curves. Furthermore, with manual labeling, there is no uniform evaluation criterion for where the boundary between a hands-off and a hands-on situation should be drawn.

[0010] The invention is based on the object of finding an improved machine learning approach to solve the problems mentioned.

[0011] The object is achieved by the subject matter of independent patent claims 1, 2, 4, 5, 6, 8, 9, and 10. Further preferred embodiments of the invention emerge from the remaining features mentioned in the subclaims.

[0012] A first aspect of the invention relates to a method for operating a driver assistance system in which a hands-off situation in a final product vehicle is detected on the basis of a learning algorithm, comprising the following steps: - Driving the final product vehicle equipped with a driver assistance system that uses sensors to collect steering torque data while driving; - Detection of hands-off situations from the measured steering torque data using a trained learning algorithm.

[0013] According to the invention, the adaptive algorithm in the final product vehicle is trained to the trained adaptive algorithm using training data created by a development vehicle.

[0014] A second aspect of the invention relates to a method for creating training data suitable for training a driver assistance system based on a learning algorithm for detecting a hands-off situation in a final product vehicle, comprising the following steps: - Driving a development vehicle equipped with a driver assistance system that uses sensors to record steering torque data while driving and additionally records at least distance data from a driver's hands to the steering wheel; - Detection of hands-off situations at least from the measured distance data; - Linking the detected hands-off situations with the steering torque data during the period of the detected hands-off situations to create labeled steering torque data as training data.

[0015] The highly accurate sensor-based distance data of the hands to the steering wheel can be used as a clear basis of information regarding whether a hands-off situation exists. For this purpose, the expert can set a threshold value for the signal level of the sensor used. Each time the threshold value is exceeded, for example, an automatic assessment can be made that a hands-off situation exists. This assessment can be synchronized with the measured steering torque data using time stamps, for example. Thus, the period during which the hands-off situation existed can be clearly assigned to the same period in the steering torque signal curve.

[0016] By driving the development vehicle, any amount of training data can be generated automatically under real-world conditions. The evaluation of the hands-off situation is consistent and reproducible.

[0017] In a preferred embodiment of the method of the invention, a capacitive distance sensor is used to acquire the distance data. Such sensors are highly accurate and can also be easily integrated into the steering wheel.

[0018] In a further preferred embodiment of the method according to the second aspect of the invention, additional data from the group consisting of steering wheel movement data, lane progression data, lane detection data, and vehicle movement data are used to detect hands-off situations. Thus, the distance data from the distance sensor can be examined for hands-off situations using supplementary information sources in addition to the specified threshold value, for example, to increase the validity of the detection in the area around the threshold value.

[0019] In other words, the invention relates to a method for the automated production of labeled steering torque data with which an AI unit can be trained to then recognize hands-off situations during driving.

[0020] A further aspect of the invention relates to a data carrier on which training data produced using a method according to the invention for creating training data according to the preceding description is stored. The data carrier is also referred to below as a training data carrier. The data carrier can be any data carrier, preferably a digital data carrier, such as a flash storage media.

[0021] A further aspect of the invention relates to a method for training a learning algorithm, in which training data generated in a method according to the invention for generating training data according to the preceding description or stored on a training data carrier according to the invention according to the preceding description are used. The learning algorithm can, for example, comprise a neural network that processes the labeled steering torque curves for learning purposes.

[0022] A further aspect of the invention relates to a data carrier on which an adaptive algorithm is stored, which has been trained using a method according to the invention as described above. The data carrier is also referred to below as an algorithm data carrier.

[0023] In the method according to the first aspect, it is alternatively or additionally provided that the trained learning algorithm - has been trained using training data produced in a method according to the invention for producing training data as described above; or - has been trained using a training data carrier according to the invention in accordance with the preceding description; or - has been trained in a method according to the invention for training a learning algorithm according to the preceding description; or - is stored on an algorithm data carrier according to the invention as described above.

[0024] The driver assistance system thus provides the trained adaptive algorithm with the measured, unlabeled steering torque data as input. The adaptive algorithm, trained on the basis of the labeled steering torque data, then assesses whether a hands-off situation exists. The detection quality of the method according to the invention is significantly improved compared to conventional methods, without requiring significant technical effort.

[0025] In particular, a hand distance sensor can be omitted in the final product vehicle, which is intended to detect hands-off situations during use by a consumer.

[0026] A further aspect of the invention relates to a driver assistance system designed to carry out a method according to the invention for creating training data according to the preceding description or for creating a training data carrier according to the invention according to the preceding description or for carrying out a method according to the invention for training a learning algorithm according to the preceding description or for creating an algorithm data carrier according to the invention according to the preceding description.

[0027] A further aspect of the invention relates to a driver assistance system designed to carry out a method according to the invention for operating a driver assistance system in which a hands-off situation in a vehicle is detected on the basis of a trained, adaptive algorithm according to the preceding description.

[0028] A further aspect of the invention relates to a vehicle comprising a driver assistance system according to the invention as described above. The vehicle can preferably be a motor vehicle, particularly preferably an automobile or truck.

[0029] The various embodiments of the invention mentioned in this application can be advantageously combined with one another, unless otherwise stated in the individual case.

[0030] The invention is explained below in exemplary embodiments with reference to the accompanying drawings. They show: Fig. 1 a development vehicle according to the invention in a method according to the invention for creating training data; and Fig. 2 an end product vehicle according to the invention in a method according to the invention for detecting a hands-off situation.

[0031] Fig. 1 shows a development vehicle 10 according to the invention with a first driver assistance system 12 according to the invention.

[0032] The first driver assistance system 12 comprises various components distributed throughout the development vehicle 10. The components include a steering torque sensor 14 capable of measuring steering torque data 16. Furthermore, the first driver assistance system 12 includes a capacitive distance sensor 18 capable of measuring distance data 20 between the hands 22 of a driver 24 and a steering wheel 26 of the development vehicle 10.

[0033] The first driver assistance system 12 can be operated in a method according to the invention as follows.

[0034] While the development vehicle 10 is moving, the steering torque data 16 is recorded by the steering torque sensor 14, and the distance data 20 is also recorded by the distance sensor 18. This initial state is shown as a block diagram below the development vehicle 10.

[0035] Hands-off situations are determined in a known manner from the measured distance data 20. For example, a hands-off situation can be detected and coded with a logical one if the hands 22 of the driver 24 are farther away from the distance sensor 18 than the distance from the distance sensor 18 to the surface 28 of the steering wheel 26. Otherwise, a logical zero can be used to code that no hands-off situation exists.

[0036] The block diagram indicates how these coded states are provided with time stamps 30 in time with a sampling rate of the distance sensor 18 and are temporarily stored as intermediate results 32 by the driver assistance system 12.

[0037] At the same time, the steering torque data 16 are provided with the same time stamps 30 and temporarily stored as intermediate results 34 by the first driver assistance system 12.

[0038] In a further step, the intermediate results 34 are then linked to the intermediate results 32. For each period formed between two consecutive time stamps 30, the steering torque data 16 is assigned the logical coding of the distance data 20 that is present for the same period. The time stamps 30 are thus eliminated in the result 36. This leaves steering torque data 16 corresponding to the original periods, each of which is assigned a logical one if the steering torque data 16 corresponds to a hands-off situation, or a logical zero if the steering torque data 16 does not correspond to a hands-off situation.

[0039] These results 36 are stored by the first driver assistance system 12 as labeled steering torque data 38. The labeled steering torque data 38 constitute training data 40, which can be used as described in more detail below.

[0040] The training data 40 can be stored by the first driver assistance system 12 on a data carrier 42.

[0041] The training data 40 are used to train a further (second) driver assistance system 48 (see Fig. 2), which can be designed, for example, like the first driver assistance system (12) or can be designed without a distance sensor 18, on the basis of a learning algorithm 44 (cf. Fig. 2) to train.

[0042] Fig. 2 shows a final product vehicle 46 according to the invention with the further or second driver assistance system 48 according to the invention. The final product vehicle 46 differs from the development vehicle 10 in that the second driver assistance system 48 does not have a distance sensor.

[0043] However, the second driver assistance system 48 includes the adaptive algorithm 44, which in other embodiments may also be provided in the first driver assistance system 12. In the examples described here, the first driver assistance system 12 serves to create the training data 40, and the second driver assistance system 48 uses this data as described below. If the first driver assistance system 12 also includes the adaptive algorithm 44, it can, for example, immediately test the created training data 40 with the adaptive algorithm 44. The procedure is analogous to the method described below.

[0044] The second driver assistance system 48 in the present case has a control unit 50 on which the learning algorithm 44 is stored or installed.

[0045] The second driver assistance system 48 reads the training data 40, for example, from the data storage device 42, and trains the adaptive algorithm 44 with the training data 40, resulting in a trained adaptive algorithm 52. This is indicated in the block diagram below the final product vehicle 46.

[0046] With the second driver assistance system 48, a method according to the invention can then be carried out in which a hands-off situation 54 in the end product vehicle 46 is detected on the basis of the trained learning algorithm 52.

[0047] For this purpose, the end product vehicle 46 drives and the second driver assistance system 48 again records the steering torque data 16 using sensors while driving.

[0048] The steering torque data 16 are made available to the trained learning algorithm 52 as input data 56.

[0049] Since the trained adaptive algorithm 52, as previously described, has been trained to recognize characteristics in the steering torque data 16 that correspond to a hands-off situation, the trained adaptive algorithm 52 can output, for example, a logical one as output data 58 if, as here, the hands-off situation 54 is present. Otherwise, a logical zero can be output.

[0050] The statement about the presence of a hands-off situation 54 can then be further processed in a variety of ways. For example, a warning signal can be issued or automatic steering can be activated.

[0051] In the examples described here, vehicle 10 is particularly suitable for the automated generation of training data 40, in other words, as a development vehicle. Vehicle 46 is particularly well-suited as a final product, in which distance sensor 18 can be omitted and only the functionality for detecting hands-off situations during driving is used. List of reference symbols 10 development vehicle 12 first driver assistance system 14 Steering torque sensor 16 Steering torque data 18 Distance sensor 20 distance data 22 hands 24 drivers 26 Steering wheel 28 Surface 30 timestamps 32 Interim result 34 Interim result 36 results 38 labeled steering torque data 40 training data 42 data carriers 44 learning algorithm 46 End product vehicle 48 second driver assistance system 50 control unit 52 trained learning algorithms 54 Hands-off situation 56 input data 58 Output data

Claims

[1] Method for operating a driver assistance system (48) in which a hands-off situation (54) in a final product vehicle (46) is detected on the basis of a learning algorithm (44), comprising the following steps: - driving the final product vehicle (46) equipped with the driver assistance system (48) which sensorily detects steering torque data (16) during driving; - Detection of hands-off situations (54) from the measured steering torque data (16) using a trained learning algorithm (52), characterized by that the learning algorithm (44) in the final product vehicle (46) has been trained to the trained learning algorithm (52) using training data (40) created by means of a development vehicle (10). [2] Method for creating training data (40) suitable for training a driver assistance system (48) based on a learning algorithm (44) for detecting a hands-off situation (54) in a final product vehicle (46), comprising the following steps: - driving a development vehicle (10) equipped with a driver assistance system (12) which, during driving, detects steering torque data (16) using sensors and additionally detects at least distance data (20) from the hands (22) of a driver (24) to the steering wheel (26); - detection of hands-off situations (54) at least from the measured distance data (20); - Linking the detected hands-off situations (54) with the steering torque data (16) in the period of the detected hands-off situations (54) in order to create labeled steering torque data (38) as training data (40). [3] Method according to claim 2, characterized bythat additional data are used to detect hands-off situations (54), from the group: Steering wheel movement data; road trajectory data; lane detection data; vehicle movement data. [4] Data carrier (42) on which training data (40) produced in a method according to one of claims 2 or 3 are stored. [5] Method for training a learning algorithm (44) in which training data (40) produced in a method according to one of claims 2 or 3 or stored on a data carrier (42) according to claim 4 are used. [6] Data carrier on which a learning algorithm (52) is stored which has been trained in a method according to claim 5. [7] Method according to claim 1, characterized by that the trained learning algorithm (52) - has been trained using training data (40) produced in a method according to one of claims 2 or 3; or - has been trained using a data carrier (42) according to claim 4; or - has been trained in a method according to claim 5; or - is stored on a data carrier according to claim 6. [8] Driver assistance system (12), trained - for carrying out a method according to one of claims 2 or 3 or - for creating a data carrier (42) according to claim 4 or - for carrying out a method according to claim 5 or - for creating a data carrier according to claim 6. [9] Driver assistance system (48) designed to carry out a method according to claim 1. [10] Vehicle (10; 46) comprising a driver assistance system (12; 48) according to claim 8 or 9.

Images