COMBINED SENSORS FOR CONTROLLING TRANSFERS IN AUTONOMOUS DRIVING

Abstract

System for integrating sensors to control transitions in autonomous driving, wherein the system comprises: a sensor merging module (102) that merges a plurality of steering sensor data (110) from one or more sensors of a steering system (12) with a plurality of driver status data (112) from a plurality of driver status sensors to form a plurality of merged status data (114) that are aligned in time; a decision module (104) that determines, based on the aggregated status data (114), whether to switch from an autonomous driving mode to a manual driving mode; and a current data store (106) of the merged status data and a historical data store (108) of the merged status data (114), wherein the historical data store (106) can be operated to record older values ​​of the merged status data (114) than those recorded in the current data store (106), wherein the decision module (104) can be operated to average several samples of the merged status data (114) from the historical data storage (108) as time-averaged merged status data and to determine, on the basis of the merged status data (114) from the current data storage (106) in combination with the time-averaged merged status data, whether to switch from autonomous driving mode to manual driving mode, and to continue monitoring the merged status data (114) after the switch from autonomous driving mode to manual driving mode has been initiated, and to confirm, based on switch validity criteria, whether to return to autonomous driving mode.

Description

BACKGROUND OF THE INVENTION

[0001] Advanced driver assistance systems (ADAS) and automated driving systems are beginning to use a driver sensing system to monitor driver status when a vehicle is being driven autonomously or in near-autonomous (semi-autonomous) mode. These systems need to monitor the driver to ensure their status is appropriate for the driving mode. Examples include built-in steering and torque sensors to estimate driver input, steering wheel touch sensors to verify that the driver's hands are on the wheel for lane keeping assist and similar functions, and camera monitoring to ensure the driver is sufficiently alert for hands-free driving—that is, the driver is not asleep or taking their eyes off the road for longer than a specified time interval.

[0002] However, each of these detection systems has limitations when switching from automated to manual driving. If the switch was not the driver's intention, a safety hazard is created because the system would transfer control to a human who is unwilling to take over. Consider an application where the steering system's built-in torque and position sensors are used to detect driver input as a signal to override autonomous control and switch to manual control. If, in this situation, something other than the driver's hands was the source of the steering input (e.g., the driver's knee contacting the steering wheel), the system would ultimately switch to manual driving, contrary to the driver's intention, thus creating a hazardous situation.

[0003] DE 10 2013 224 118 A1 discloses a method for controlling a vehicle in which a control status is changed based on driver behavior information. DE 10 2015 101 563 A1 teaches a method for controlling a driving characteristic for an automated driving system based on various operating and sub-operating states of the vehicle. DE 10 2014 218 526 A1 relates to a system for recognizing an intended operator input, which changes a driving mode based on a recognized intended operator input. SUMMARY OF THE INVENTION

[0004] One of the problems underlying the invention is to provide an improved system for combining sensors to control transitions in autonomous driving, as well as a corresponding method.

[0005] This task is solved by the subject matter of the independent claims.

[0006] A system for merging sensors to control transitions in autonomous driving comprises a sensor merging module and a decision module. The sensor merging module combines a variety of steering sensor data from one or more sensors of a steering system with a variety of driver status data from a variety of driver status sensors to create a variety of time-aligned merged status data. Based on the merged status data, the decision module determines whether to switch from an autonomous driving mode to a manual driving mode. The system further includes a current data store of the merged status data and a historical data store of the merged status data, with the historical data store being capable of storing older values ​​of the merged status data than those stored in the current data store.The decision module can further be operated to average multiple samples of the merged status data from the historical data store as time-averaged merged status data, and to determine, based on the merged status data from the current data store in combination with the time-averaged merged status data, whether to switch from autonomous driving mode to manual driving mode, and to continue monitoring the merged status data after the switch from autonomous driving mode to manual driving mode has been initiated, and to confirm, based on switch validity criteria, whether to return to autonomous driving mode.

[0007] A method for aggregating sensors to control transitions in autonomous driving involves a control module acquiring a variety of steering sensor data from one or more sensors of a steering system and acquiring a variety of driver status data from a variety of driver status sensors. The steering sensor data is combined with the driver status data to form a variety of time-aligned aggregated status data. Based on this aggregated status data, the control module determines whether to switch from an autonomous driving mode to a manual driving mode.The procedure further includes operating a historical data store to record older values ​​of the merged status data than those recorded in the current data store. Multiple samples of the merged status data from the historical data store are averaged to form time-averaged merged status data. Based on this time-averaged merged status data, a decision is made as to whether to switch from autonomous driving mode to manual driving mode. Furthermore, monitoring of the merged status data continues after the switch from autonomous driving mode to manual driving mode has been initiated, and based on switch validity criteria, a decision is made as to whether to return to autonomous driving mode.

[0008] These and other advantages and features will become clearer from the following description when read in conjunction with the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The subject matter considered to be the invention is specifically set forth and claimed separately in the claims at the end of the description. The foregoing and further features and advantages of the invention will become apparent from the following detailed description when read in conjunction with the accompanying drawings, which: Fig. 1. A functional block diagram illustrates a vehicle which incorporates a steering system in accordance with some embodiments; Fig. 2. A system for combining sensors to control transitions in autonomous driving is illustrated in accordance with some embodiments; Fig. 3 illustrates a process for merging sensors to control transitions in autonomous driving in accordance with some embodiments; and Fig. 4 illustrates a process for confirming transitions in autonomous driving in accordance with some embodiments. DETAILED DESCRIPTION

[0010] With reference to the figures, in which the invention is described with reference to specific embodiments without limiting them, an exemplary embodiment of a vehicle 10 is illustrated, which includes a steering system 12. In various embodiments, the steering system 12 includes a steering wheel 14 coupled to a steering shaft 16. In the exemplary embodiment shown, the steering system 12 is an electric power steering system (EPS system) which further includes a steering assist unit 18 coupled to the steering shaft 16 of the steering system 12 and to a left tie rod 20 and a right tie rod 22 of the vehicle 10. It should be noted that the steering system 12 can also be a rack-and-pinion EPS (REPS).The steering assistance unit 18, for example, includes a steering mechanism (not shown) with a rack and pinion, which can be coupled via the steering shaft 16 to a steering actuator motor 19 and a steering gear. When the steering wheel 14 is turned by a vehicle operator during operation, the steering actuator motor 19 provides assistance to move the left tie rod 20 and the right tie rod 22, which in turn move the respective left and right steering knuckles 24, 26. The left steering knuckle 24 is coupled to a left road wheel 28, and the right steering knuckle 26 is coupled to a right road wheel 30 of the vehicle 10.

[0011] As in Fig. As shown in Figure 1, the vehicle 10 further includes various sensors 31-36 that detect and measure signals from the steering system 12, the vehicle 10, and the driver's attention. The sensors 31-36 generate sensor signals based on the measured / observed signals. In one embodiment, a steering wheel torque sensor 31 is provided to detect torque applied to the steering wheel 14. In the exemplary embodiment shown, the steering wheel torque sensor 31 is located on the steering wheel 14; however, it is understood that the steering wheel torque sensor 31 may not always be located near or on the steering wheel 14. In another embodiment, an engine position / speed sensor 32 detects an engine position and / or an engine speed, and a steering wheel position / speed sensor 33 detects a steering wheel position and / or a steering wheel speed.Furthermore, the vehicle 10 may include a wheel speed sensor 34 to assist in measuring vehicle speed. In some embodiments, one or more steering wheel touch sensors 35 measure a grip force or grip pressure on the steering wheel 14 at various locations, which are detected as a field of contact with the steering wheel 14, a span of contact with the steering wheel 14, a force of contact with the steering wheel 14, and / or a position of contact with the steering wheel 14. Data from one or more steering wheel touch sensors 35 may include a magnitude in combination with an angular position. A camera 36 may detect one or more of the following: the driver's posture, the driver's head pose, the driver's eye gaze, and the driver's hand position.The camera 36 can be mounted at any suitable location to monitor the driver's body and / or face and / or eyes and / or hands. For example, the camera 36 can be mounted in a steering column, on a dashboard, on an A-pillar, or on a roof console. In some embodiments, multiple cameras 36 are used to capture image data from different angles / locations. The one or more steering wheel touch sensors 35 and cameras 36 are also referred to as driver status sensors, which record information about the driver's attention.

[0012] A control module 40 controls the operation of the steering system 12 based on one or more of the sensor signals and further on the basis of the steering control systems and methods of this disclosure. The control module 40 generates a command signal to control the steering actuator motor 19 of the steering system 12 based on one or more of the inputs and further on the basis of the steering control systems and methods of this disclosure. The steering control systems and methods of this disclosure combine time-aligned status data for two or more of the sensors 31-36 in order to determine, based on the combined status data, whether to switch from an autonomous driving mode to a manual driving mode. The control module 40 can be implemented in one or more controllers.It can be stated that such a system is capable of operating in both an autonomous state and a manual driving mode, whereby the overall vehicle system, which is capable of operating in both states, is a semi-autonomous vehicle.

[0013] Fig. Figure 2 illustrates a system 100 for handle-based steering wheel compensation in accordance with one embodiment. The system 100 includes the control module 40 and receives data from two or more of the sensors 31-36 of Fig. 1. In various embodiments, the control module 40 can contain one or more submodules and data storage devices, such as a sensor aggregation module 102, a decision module 104, a current data storage device 106, and a historical data storage device 108. The terms module and submodule, as used here, refer to an application-specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) with memory that executes one or more software or firmware programs, a combinational logic circuit, or other suitable components that provide the described functionality. As can be seen, the Fig. The control module 40 shown in Figure 2 can be further subdivided and contain additional control elements known in the field of steering control systems. For example, the control module 40 or another (not shown) module of the vehicle 10 can be divided into two parts. Fig. 1 known techniques for automated steering control of the steering system 12 of Fig. 1. Implement.

[0014] Inputs to the control module 40 can be received from sensors 31 - 36 ( Fig. 1) of vehicle 10 ( Fig. 1) as well as from other (not shown) sensors. In addition, inputs from other (not shown) control modules in the vehicle 10 ( Fig. 1) are received and can be modeled or predefined. Steering sensor data 110 from one or more of sensors 31, 32 and 33 ( Fig. 1) of the steering system 12 and driver status data 112 from one or more of the driver status sensors 35 and 36 ( Fig. 1) can be provided to the sensor aggregation module 102 to aggregate the steering sensor data 110 and the driver status data 112 into a variety of aggregated status data 114 that are aligned in time. For example, steering sensor data 110 can include acquired or inferred data from one or more of the steering wheel torque sensors 31 as EPS torque 116 and from the steering wheel position / speed sensor 33 as EPS position 118. The driver status data 112 can include acquired or inferred data from one or more steering wheel touch sensors 35 as touch / distance data 120 and from the camera 36 as image data 122. The touch / distance data 120 can include a variety of acquired or inferred data, such as a field of contact with the steering wheel 14 ( Fig. 1), a span of contact with the steering wheel 14, a force of contact with the steering wheel 14 and / or a position of contact with the steering wheel 14. The image data 122 can be used to determine various indicators of driver activity, such as the driver's posture (e.g., upright, to the left, to the right, bent down, obscured), the pose of the driver's head (e.g., to the left, to the right, up, down, obscured), the direction of the driver's eyes (e.g., the direction such as left / right / up / down or directed at specific objects such as the road, the instrument panel, the center console, the rearview mirror, the side mirror, determining a level of obscuration for one or both eyes), and / or the position of the driver's hands (e.g.,both hands visible and not on the steering wheel 14, one hand in an observed position on the steering wheel 14, both hands in observed positions on the steering wheel 14, both hands not visible or covered, etc.).

[0015] The current data store 106 can hold or buffer recently acquired values ​​of the merged status data 114, for example, a data merge period of the last 200 milliseconds. The historical data store 108 stores older values ​​of the merged status data 114 than those stored in the current data store 106, for example, a data merge period from 200 milliseconds to 5 seconds. In some embodiments, the current data store 106 can hold values ​​of the merged status data 114 for a most recent data merge period of 100 milliseconds, and the historical data store 108 stores values ​​of the merged status data 114 that are older than 100 milliseconds. Alternative partitioning of time-lapse data between the current data store 106 and the historical data store 108 is considered in other embodiments.In some embodiments, the current data memory 106 and the historical data memory 108 are combined into a single shared buffer, such as a ring buffer, where a first set of pointers identifies the positions of the current data memory 106 and a second set of pointers identifies the positions of the historical data memory 108.

[0016] The decision module 104 determines, by means of a mode command 124 and based on the aggregated status data 114, whether to switch from an autonomous driving mode to a manual driving mode. In embodiments, the decision module 104 can average several samples of the aggregated status data 114 from the historical data memory 108 as time-averaged aggregated status data and, based on the aggregated status data 114 from the current data memory 106 in combination with the time-averaged aggregated status data, determine whether to switch from the autonomous driving mode to the manual driving mode. For example, switching criteria 126 can define a truth table to cover use cases for the current data memory 106 in combination with the historical data memory 108 for determining an action to be taken, such as a change in the status of the mode command 124 (e.g.,Transition to manual driving mode). The time-averaged merged status data from historical data store 108 can be used as a baseline status to filter out noise, and relative changes in the merged status data 114 from current data store 106 can indicate a likely switchover request. For example, a combination of status values ​​of body position, face / head position, eye gaze, hand position, EPS input captured from EPS torque 116 and / or EPS position 118, from both current data store 106 and historical data store 108, can be determined to interpret a driver's intention to make the decision to release automatic control and switch to manual steering.A combination of an upright body position and / or an upright face / head position and / or eyes looking at the road and / or hands on the steering wheel, and a detected EPS input, can confirm that the driver is ready to switch from automatic to manual control. In another embodiment, a weighted average alarm index = w1 × Boolean value of body + w2 × Boolean value of face / head position + w3 × Boolean value of eyes looking at the road + w4 × Boolean value of hands on steering wheel, where the weighted average alarm index can be passed through a low-pass filter. If the automatic control is not released quickly enough, the driver may experience excessive effort from the steering system 12, resulting in increased difficulty in manually controlling the steering of the vehicle 10.To ensure that the control is not incorrectly released, a process for confirming the change during autonomous driving can also be carried out, as described further below.

[0017] Fig. Figure 3 illustrates a process 200 for integrating sensors to control transitions in autonomous driving. Process 200 is further described with reference to Fig. 1 and Fig. 2 described. Process 200 can be controlled by the control module 40 of Fig. 1 and Fig. 2. In block 202, the control module 40 acquires a variety of steering sensor data 110 from one or more sensors 31-33 of the steering system 12. In block 204, the control module 40 acquires a variety of driver status data 112 from a variety of driver status sensors 34, 35. In block 206, the sensor aggregation module 102 combines the steering sensor data 110 with the driver status data 112 to form a variety of aggregated status data 114, which are aligned in time. In block 208, the decision module 104 of the control module 40 determines, based on the aggregated status data 114, whether to switch from an autonomous driving mode to a manual driving mode. The decision module 104 can access the merged status data 114, which is stored in the current data store 106 and in the historical data store 108 (e.g.(as time-averaged aggregated status data), to compare them with the switching criteria 126 in order to determine the mode command 124.

[0018] Fig. Figure 4 illustrates a Process 300 for confirming transitions in autonomous driving. Process 300 is further described with reference to Fig. 1-3 described. Process 300 can be used in combination with process 200. Fig.3. At block 302, the merged sensor data 114 is monitored by the decision module 104 to determine whether the driver intends to change the mode command 124. If, at block 304, the decision module 104 determines, based on a comparison of the merged sensor data 114 from the current data memory 106 in combination with the historical data memory 108 and the switching criteria 126, that no switch to manual mode should be performed, process 300 returns to block 302 to continue monitoring the merged sensor data 114. If a switch to manual mode is determined at block 304, then at block 306 a switch to manual driving is initiated, for example, by reducing inputs from the automated steering control applied to the steering actuator motor 19.At block 308, the decision module 104 continues monitoring the merged status data 114 after the switch from autonomous driving mode to manual driving mode has been initiated. At block 310, the decision module 104 confirms, based on switch validity criteria in the switch criteria 126, whether to switch back to autonomous driving mode. If, for example, the merged status data 114 in the current data store 106 and / or in the historical data store 108 indicates that the driver is not in an attentive state or that inputs have been blocked, the decision module 104 can restore autonomous driving mode at block 312 and switch back to it. If the driver remains attentive for a predetermined period after the switch has occurred (e.g.,(a time period greater than or equal to the time storage capacity of the historical data storage 108), then the decision is confirmed and the switch to manual driving is completed at block 314.

[0019] Although the invention has been described in detail in connection with only a limited number of embodiments, it is readily understood that the invention is not limited to these disclosed embodiments. Instead, the invention can be modified to include any number of variations, alterations, substitutions, or equivalent arrangements not described above, but which are consistent with the spirit and scope of the invention. Furthermore, although various embodiments of the invention have been described, it is understood that aspects of the invention may include only some of the described embodiments. Consequently, the invention should not be interpreted as being limited to the foregoing description.

Claims

[1] System for integrating sensors to control transitions in autonomous driving, the system comprising: a sensor merging module (102) that merges a plurality of steering sensor data (110) from one or more sensors of a steering system (12) with a plurality of driver status data (112) from a plurality of driver status sensors to form a plurality of merged status data (114) that are aligned in time; a decision module (104) that determines, based on the aggregated status data (114), whether to switch from an autonomous driving mode to a manual driving mode; and a current data store (106) of the merged status data and a historical data store (108) of the merged status data (114), wherein the historical data store (106) can be operated to record older values ​​of the merged status data (114) than those recorded in the current data store (106), wherein the decision module (104) can be operated to average several samples of the merged status data (114) from the historical data storage (108) as time-averaged merged status data and to determine, on the basis of the merged status data (114) from the current data storage (106) in combination with the time-averaged merged status data, whether to switch from autonomous driving mode to manual driving mode, and to continue monitoring the merged status data (114) after the switch from autonomous driving mode to manual driving mode has been initiated, and to confirm, based on switch validity criteria, whether to return to autonomous driving mode. [2] System according to claim 1, wherein the steering sensor data (110) comprise data acquired or derived from a steering wheel torque sensor (31) and / or a steering wheel position / speed sensor (33). [3] System according to claim 1, wherein the driver status data (112) comprise data acquired or derived from a steering wheel touch sensor (35) and / or a camera (36). [4] System according to claim 3, wherein the driver status data (112) acquired by or derived from the steering wheel touch sensor (35) comprise a field of contact with a steering wheel (14), a span of contact with the steering wheel (14), a force of contact with the steering wheel (14) and / or a position of contact with the steering wheel (14), and wherein the driver status data acquired by or derived from the camera (36) comprise a body posture of the driver and / or a pose of the driver's head and / or a gaze of the driver's eyes and / or a position of the driver's hands. [5] Method for combining sensors for controlling transitions in autonomous driving, the method comprising: a control module (40) obtains a large number of steering sensor data (110) from one or more sensors of a steering system (12); a large number of driver status data (112) are obtained from a large number of driver status sensors; the steering sensor data (110) are combined with the driver status data (112) to form a multitude of merged status data (114) that are aligned in time; The control module (40) determines, based on the aggregated status data (114), whether to switch from an autonomous driving mode to a manual driving mode; and a historical data store (108) is operated to record older values ​​of the merged status data (114) than those recorded in the current data store (106), wherein several samples of the merged status data (114) from the historical data storage (108) are averaged as time-averaged merged status data and on the basis of the merged status data (114) from the current data storage (106) in combination with the time-averaged merged status data, it is determined whether to switch from autonomous driving mode to manual driving mode, where monitoring of the merged status data (114) continues after the switch from autonomous driving mode to manual driving mode has been initiated, and based on switch validity criteria it is confirmed whether to return to autonomous driving mode. [6] Method according to claim 5, wherein the steering sensor data (110) comprise data acquired from or derived from a steering wheel torque sensor (31) and / or a steering wheel position / speed sensor (33). [7] Method according to claim 5, wherein the driver status data (112) comprise data acquired or derived from a steering wheel touch sensor (35) and / or a camera (36), wherein the driver status data (112) acquired or derived from the steering wheel touch sensor (35) comprise an area of ​​contact with a steering wheel (14) and / or a span of contact with the steering wheel (14) and / or a force of contact with the steering wheel (14) and / or a position of contact with the steering wheel (14), and wherein the driver status data (112) acquired or derived from the camera (36) comprise a body posture of the driver, a pose of the driver's head and / or a gaze of the driver's eyes and / or a position of the driver's hands. [8] The method of claim 5, further comprising: the merged status data (114) are stored in a current data store (106); and the merged status data (114) are stored in the historical data store (108), the historical data store (108) containing older values ​​of the merged status data (114) than those contained in the current data store (106). [9] The method of claim 8, further comprising: Several samples of the merged status data (114) from the historical data storage (108) are averaged to form time-averaged merged status data; and Based on the merged status data (114) from the current data storage (106) in combination with the time-averaged merged status data, it is determined whether to switch from autonomous driving mode to manual driving mode. [10] The method of claim 5, further comprising: monitoring of the merged status data (114) continues after the switch from autonomous driving mode to manual driving mode has been initiated; and Based on switching validity criteria, it is confirmed whether to return to autonomous driving mode.

Images