Optical-based detection for human driver hand-clutch and gesture-based function selection
Through the appendage tracking module combined with optical sensors and cameras, the driver's hand position and posture are analyzed, and the problem of the steering system in the prior art is difficult to accurately explain the driver's intentions, improving the reliability and driving experience of the autonomous steering system.
Patent Information
- Application Number
- CN202011091393.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-10-09
- Filing Date
- 2020-10-13
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-10-13
AI Technical Summary
Existing steering systems have difficulty explaining the driver's intentions accurately, especially during the initialization and preparation phase of autonomous driving assistance systems, where purely touch-based torque input is not sufficient to provide reliable input.
The appendage tracking module combined with optical sensors and cameras is used to analyze the driver's hand position and posture characteristics, determine its intentions, and combine neural networks for image analysis and sensor data correlation, identify the driver's intentions and communicate them to the autonomous steering system.
It improves the accurate identification and interpretation of driver intentions, enhances the reliability of the autonomous steering system and the driver's intuitive driving experience, and reduces the startup delay of the autonomous driving mode.
Smart Images

Figure CN112722070B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of U.S. Provisional Application Serial No. 62 / 914,650, filed on October 14, 2019. Technical Field
[0003] The general field to which the present disclosure relates includes steering systems. Background Art
[0004] Vehicles often include a steering system. The prevalence of camera-based hand detection for interpreting the driver's intent along with other sensors in steering systems or other monitoring systems is known. Known capacitive strips that can be mounted on steering wheels can give some improved capabilities, but they can also be dependent on physical touch and it is difficult to interpret the driver's intent using the same device. Interpreting the driver's intent purely based on toucher torque does not provide the necessary input to initialize or prepare an autonomous driving assistance system in a reliable manner. Summary of the invention
[0005] A number of illustrative variations may include a method or product for sensing driver intervention in an autonomous steering system.
[0006] Other exemplary variations within the scope of the invention will become apparent from the detailed description provided hereinafter. It should be understood that the detailed description and specific examples, while disclosing variations of the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Selected examples of variations within the scope of the invention will become more fully understood from the detailed description and accompanying drawings, in which: Figure 1 An illustrative variation is depicted in which an image representing a camera image or optical sensor data depicting a human driver interacting with a steering interface of an autonomous steering system is being analyzed and labeled by a human appendage tracking module. In the illustrative variation, the human appendage tracking module utilizes data from at least one optical sensor or images from at least one camera to determine that the human driver desires to control the steering interface based on at least one of a hand position or a hand gesture characteristic of the human driver according to a plurality of variations.
[0008] Figure 2An illustrative variation is depicted in which an image representing a camera image or optical sensor data depicting a human driver interacting with a steering interface of an autonomous steering system is being analyzed and labeled by a human appendage tracking module. In the illustrative variation, the human appendage tracking module utilizes data from at least one optical sensor or an image from at least one camera to determine that the human driver desires to relinquish control of the steering interface based on at least one of a hand position or a hand gesture characteristic of the human driver according to a plurality of variations.
[0009] Figure 3A is an illustration of a driver's hand on a steering wheel with multiple fingers curled around and / or gripping the steering wheel according to a number of variations.
[0010] Figure 3B 200 according to a number of variants in the field of view of an optical sensor positioned to observe the front side 200 of the steering wheel, for example Figure 3A The driver's hand is shown in FIG.
[0011] Figure 3C 1 shows, for example, a view of a side edge of a steering wheel in the field of view of an optical sensor positioned to observe the side edge of a steering wheel according to a number of variations. Figure 3A The driver's hand is shown in FIG.
[0012] Figure 4A is an illustration of a driver's hand in the field of view of an optical sensor positioned to view the front of a steering wheel according to multiple variations, wherein the driver is making a gesture that indicates the driver's intent to hand over steering control to a vehicle autonomous driving system by flicking multiple fingers of the driver's hand, wherein the multiple fingers are extended and not curled around or gripping the steering wheel.
[0013] Figure 4B 1 shows, for example, a view of a front view of a steering wheel in the field of view of an optical sensor positioned to observe the front of the steering wheel according to a number of variations. Figure 4A The driver's hand is shown in FIG.
[0014] Figure 4C is an illustration of a driver's hand in a field of view of an optical sensor positioned to view a side edge of a steering wheel according to multiple variations, wherein the driver is making a gesture indicating the driver's intent to hand over steering control to a vehicle autonomous driving system by flicking the driver's fingers, wherein the fingers are extended and not curled around or gripping the steering wheel.
[0015] Figure 4DData is shown that an optical sensor according to multiple variations may generate for a driver's hands in the optical sensor's field of view, wherein the optical sensor is positioned to view a side edge of a steering wheel, wherein the driver is making a gesture that indicates the driver's intent to hand over steering control to a vehicle autonomous driving system by flicking multiple fingers of the driver, wherein the multiple fingers are extended and not curled around or gripping the steering wheel.
[0016] Figure 4E is an illustration of a driver indicating the driver's intent to hand over steering control to a vehicle autonomous driving system by completely removing the driver's hand from a steering wheel according to a number of variations.
[0017] Figure 5A A driver's hand is shown in the field of view of an optical sensor positioned to view the side edge of a steering wheel according to multiple variations, with multiple fingers of the driver's hand curled around the steering wheel, indicating the driver's intent to take over control of the vehicle's steering system.
[0018] Figure 5B Shown is data that an optical sensor may generate for a driver's hand in the optical sensor's field of view, wherein the optical sensor is positioned to view the side edge of a steering wheel, with multiple fingers of the driver's hand curled around the steering wheel, indicating the driver's intent to take over control of the vehicle's steering system.
[0019] Fig. 6A A driver's hands are shown in the field of view of an optical sensor positioned to view the side edge of a steering wheel according to multiple variations, with multiple fingers of each of the driver's hands curled around the steering wheel, indicating the driver's intent to take over control of the vehicle's steering system.
[0020] Figure 6B Data is shown that an optical sensor according to multiple variations can generate for a driver's hands in the optical sensor's field of view, wherein the optical sensor is positioned to observe the side edges of a steering wheel, with multiple fingers on each of the driver's hands curled around the steering wheel, thereby indicating the driver's intent to take over control of the vehicle's steering system.
[0021] Figure 7 The field of view of the optical sensor positioned to view the front face of the steering wheel is shown, and shows the detection area where the driver has placed his hand, indicating the driver's intent to take control of the vehicle's steering system.
[0022] Figure 8 The field of view of an optical sensor according to a number of variations is shown. DETAILED DESCRIPTION
[0023] The following description of the variations is merely illustrative in nature and is in no way intended to limit the scope of the invention, its application, or uses.
[0024] In a number of illustrative variations, a vehicle for cargo or passengers can be driven forward by motor power derived from a motor that converts a stored energy source into a driving force for the vehicle, such as, but not limited to, an internal combustion engine, a battery-powered engine, a fuel cell-powered engine, or any other known motor for providing motorized driving power for a passenger or cargo vehicle. The driving force generated by the conversion of the energy stored by the motor can be transmitted from the motor to a driving medium along which the vehicle will travel, such as, but not limited to, a large piece of land, a road, a waterway, a route, or any other medium in which a known vehicle is to travel through space. The transmission of the driving force from the motor to the driving medium can occur via any device that drives the motor vehicle to move, such as, but not limited to, wheels, treads, casters, rollers, propellers, gas thrusters, liquid thrusters, or ion drive thrusters, or any other known device that drives the motor vehicle to move.
[0025] As used herein, "wheel" or "wheel", even when modified by a descriptive adjective (such as but not limited to in the description of "steerable wheel", "steerable wheel", "wheel" or "drive wheel"), may refer to a conventional wheel and tire arrangement, but may also refer to any modification of a conventional wheel and tire arrangement, such as but not limited to a frameless magnetic levitation tire, a ball tire, or any other known means of motorized movement, such as but not limited to a tread, caster, roller, propeller, or gas thrusters, liquid thrusters, or ion drive thrusters.
[0026] As used herein, "road" may refer to conventional driving surface roads, such as, but not limited to, concrete or asphalt roads, even when modified by a descriptive adjective, but may also refer to any driving surface or medium along or through which a vehicle for the transport of cargo or passengers may travel, typically such as, but not limited to, water, ice, snow, dirt, mud, air or other gas or space.
[0027] As used herein, "appendage" may refer to the hands, fingers, forearms, elbows and arms, shoulders, neck, head, thighs, knees, calves, feet, toes, or any other body part, even when modified by a descriptive adjective.
[0028] As used herein, an "observation area" may refer to a point, zone, or area from which at least one sensor or camera is configured to collect data. As a non-limiting example, if a camera is generally directed to collect an image of a handwheel, the "observation area" may refer to the handwheel, but may also refer to any point captured in the image. As another non-limiting example, if a sensor is configured to collect data at a point of the sensor or at a point away from the sensor, the point from which data is collected may be referred to as an "observation area." As used herein, an "observation area" may also refer to an area of an image that at least one module is configured to analyze, or a collection of data collected from a sensor that at least one module is configured to analyze. To this end, as used herein, "observation" may refer to the use of the ability of a camera or sensor to collect data at a point of the sensor or at a point away from the sensor.
[0029] As used herein, "posture" may refer to the posture or position of at least one appendage of a human driver, even when modified by a descriptive adjective.
[0030] In a plurality of illustrative variations, the vehicle may have a steering system that allows the driver to change the direction of the vehicle or to turn it from a path in which it can travel. The steering system may be operated in conjunction with a source such as a pair of drive wheels that drives the motor vehicle to move. As a non-limiting example, the vehicle may be equipped with an internal combustion engine that mechanically drives a pair of rear wheels to propel the vehicle forward along a road. In such an example, the vehicle may additionally be equipped with a set of steerable front wheels that may be manipulated via a steering interface by a steering system such as, but not limited to, a hand wheel that turns the vehicle left and right when the vehicle travels along a road. In such an example, the driven rear wheels are used as a device to drive the motor vehicle to move, and the steerable front wheel pair as manipulated by the steering interface is used as a steering system. Importantly, this is not the only device by which the vehicle is contemplated to be driven or turned in the present disclosure. In fact, in a plurality of illustrative variations, the front wheels may be drive wheels as well as steerable wheels. Similarly, the device that drives the motor vehicle to move does not need to be of the same type as the steering device. That is, if the device that drives the motor vehicle to move includes wheels, the steering device does not need to also include wheels. For this reason, as a non-limiting example, it is envisioned that the snowmobile can be driven by a set of treads toward the rear of the vehicle and steered by a set of steerable skis toward the front of the vehicle. Additionally, it is envisioned that the device that drives the motor vehicle to move (such as but not limited to the drive wheel) and the steering device (such as but not limited to the steerable wheel) can change function or wander in function when in operation. As a non-limiting example, a vehicle including a pair of drive wheels near the rear of the vehicle and also including a pair of steerable wheels near the front of the vehicle can change the driving mode and begin to use each available wheel, including the front steering wheel, as a drive wheel, while still maintaining the steerable nature and steering function of the steerable front wheel. Similarly, it is expected that in some cases, the drive wheel can be used as a steerable wheel intermittently or optionally.
[0031] In a plurality of illustrative variations, the vehicle steering system can also be autonomous, because the vehicle can turn itself toward a predetermined position that has been communicated to it, without assistance or intervention from the driver. The vehicle may have an obstacle avoidance system that allows the vehicle to sense objects in its path and avoid them. In some cases, the human driver of the vehicle may wish to assist the vehicle to avoid objects or change the direction of travel of the vehicle. In this case, the driver can control at least one driver-side steering system control or at least one steering interface and assist the autonomous steering system. When the driver moves to place at least one human driver appendage on the steering interface, an optical sensing system or imaging system including at least one optical sensor or camera can collect images or optical data to be analyzed by the human appendage tracking module to determine whether the human driver intends to commandeer the steering or driving task from the autonomous steering system. If the separate human appendage tracking module cannot determine the intention of the human driver, the human appendage tracking module can associate the analyzed image or optical data with the sensor data from the sensor, and the sensor is embedded or associated with any steering interface that the human driver attempts to commandeer from the analysis of the optical sensor or image data. In some such cases, sensor data from sensors embedded in or associated with any steering interface that the human driver appears to be attempting to commandeer may include sensor data such as, but not limited to, pressure data from a pressure sensor embedded in the steering interface, touch data from a touch sensor embedded in the steering interface, torque data from any torque sensor associated with the steering interface, such as, but not limited to, a torque sensor embedded in or associated with a steering column for a handwheel, a position sensor associated with the steering interface (such as, but not limited to, a position sensor embedded in or associated with a steering column for a handwheel), or any other sensor known to provide useful data regarding manipulation of the steering interface by a human user.
[0032] In a plurality of illustrative variations, an optical sensing system or imaging system including at least one optical sensor or camera can be positioned above or behind a human driver in a vehicle cabin. In some cases, an optical sensing system or imaging system can be positioned at one of the multiple sides of a human driver or in front of a human driver on a roof panel of a vehicle cabin, as long as the observation area for the optical sensing system or imaging system is still properly configured to observe a human appendage gesture made by a human driver or an interaction with a steering interface made by a human driver. In some cases, the optical sensing system or imaging system does not need to be immediately focused on the associated observation area, but can be indirectly focused by using at least one reflector. As a non-limiting example, an optical sensing system or imaging system can be positioned near the rear of a vehicle cabin and aim at a reflector positioned on the roof panel of the vehicle. In this case, the reflector can be positioned like this to allow the optical sensing system or imaging system to properly observe the associated observation area.
[0033] In a number of illustrative variations, to facilitate a collaborative steering system, sensors of an autonomous steering system may be configured to detect and recognize steering inputs as driver inputs. Accurate recognition of driver inputs may include image analysis using a neural network to recognize and track human driver appendage movements, positions, postures, and gestures, as disclosed herein, in combination with simpler sensing methods, such as detecting a human driver grip on a steering interface or a human driver-initiated force, manipulation, or touch on a steering interface using dedicated sensors embedded in or indirectly associated with the steering interface. Correlating image analysis with data from the aforementioned simpler sensing methods results in an advantageously enhanced method of determining human driver intent for accurately interpreting human driver inputs within an autonomous steering system.
[0034] In multiple illustrative variations, the logic of any autonomous steering module can be responsive to autonomous input as well as driver input. Human driver input can be associated with human driver intention. In some cases, if the human driver intends to reach a certain threshold, full steering control can be ceded to the human driver. In some cases, if the steering system or some other system on the steering system determines that the driver input should not be given a precedent or judgment (the precedent or judgment is that the human driver's intention is low or non-existent), full steering control can be reserved only for autonomous input. As a non-limiting example, in the case where it is determined by the autonomous steering module that the human driver is very likely to have accidentally interacted with the steering interface or unintentionally made a recognized gesture, the autonomous input can be given the only control of the steering system to exclude the human driver input.
[0035] In a number of exemplary variations, a vehicle may include a steering system including a steering interface and a set of steerable wheels. The steering system may be of an electric steering type, wherein a physical linkage mechanically transmits manipulation of the steering interface to the steerable wheels. The steering system may be of a steer-by-wire type, wherein a physical mechanism does not mechanically transmit manipulation of the steering interface to the steerable wheels, and wherein manipulation of the steering interface affects associated manipulation of the steerable wheels via communication of electronic devices, such as, but not limited to, sensors, transceivers, and electronically actuated actuators. In such a system, the steering interface manipulation system may include an actuator or electromagnet, etc., for generating at least visual steering feedback when the steering intervention of the manipulation is a mechanically manipulable steering interface, such as, but not limited to, a hand wheel. In a steering system having a digital steering interface, the steering interface manipulation system may include any logic module for determining how to update any digital data displayed on any digital display for the digital steering interface. Thus, if the driver desires to cooperate or intervene in automated steering, the driver may be less disoriented when attempting to commandeer the steering interface because the steering interface has been autonomously manipulated in a manner commensurate with the vehicle's actual steering strategy.
[0036] In a plurality of illustrative variations, the vehicle includes an electronic power steering system. In this case, the steering interface mechanically communicates with any mechanism for steering the vehicle according to the steering ratio. Therefore, if the vehicle including the electronic power steering system includes semi-autonomous steering or fully autonomous steering that allows the driver to cooperate or intervene, the steering system can convey visual feedback from the steering on any interface through which the driver can turn the vehicle. As a non-limiting example, when participating in fully autonomous steering, the vehicle can automatically rotate the hand wheel of the vehicle in conjunction with the autonomous steering system to turn the steerable wheels of the vehicle, so that the non-interventional or non-cooperative driver can associate the automatic rotation of the hand wheel of the vehicle with the rotation of the steerable wheels of the vehicle, and know, infer or estimate the steering command of the autonomous steering system by intuition. Therefore, if the driver expects to cooperate or intervene in the automatic steering, the driver may not lack directional force when trying to levy a steering interface that does not seem to be associated with or misaligned with the steerable wheels of the vehicle.
[0037] In a number of illustrative variations, a steering ratio is defined as a ratio that describes the relationship between the effects of a steering interface and the effects of a steering device. As a non-limiting example, a steering ratio for a ship may be defined as a ratio of the distance that the ship's rudder turns in response to the captain's rotation of the wheel. As another non-limiting example, a steering ratio for an aircraft may be defined as a ratio of the distance that the aircraft's ailerons are raised or lowered in response to the pilot's rotation of a control wheel. As yet another non-limiting example, a steering ratio for a wheeled vehicle may be defined as a ratio of the distance that a steerable wheel of the vehicle turns in response to the rotation of a hand wheel.
[0038] In multiple illustrative variations, any autonomous manipulation of any steering interface can be controlled by an autonomous steering system so as to make the driving and riding experience more comfortable and intuitive for the driver during any transition to or from autonomous driving, cooperative driving or fully driver-controlled driving modes. In such illustrative variations, the autonomous manipulation of the steering interface can serve as visual steering feedback, which allows the driver to intuitively know or infer steering commands (which are conveyed to the autonomous steering system and interpreted by the autonomous steering system), and how those commands affect the steering of the vehicle via the concurrent, associated, and autonomous manipulation of the steering interface. In order to further promote the driver's intuitive driving and riding experience, the autonomous steering command can be interpreted by the system that affects the autonomous manipulation of the steering interface in a manner intended to be particularly compliant, pleasing or ergonomic for human drivers. As a non-limiting example, a damping factor can be applied to any original steering interface command so as to slow down any manipulation of the steering interface when the driver's manipulation of the steering interface stops, and to control the timing of the adjustment of the position and force applied to the steering interface by the steering system via the steering interface manipulation device so that the manipulation of the steering interface is synchronized with the movement of the vehicle. In this way, a driver may be more likely to find comfort in an autonomous steering vehicle if the visual steering feedback provided to the steering interface by the autonomous steering system appears less jerky, robotic, or non-human, and therefore less disconcerting to any human observer, especially when attempting to commandeer the steering interface.
[0039] In a number of illustrative variations, if a vehicle includes a steer-by-wire steering system, it may be desirable to simulate visual feedback from steering for the driver in order to deliver an intuitive, semi-autonomous steering experience to the driver. If a vehicle including a steer-by-wire steering system includes semi-autonomous steering or fully autonomous steering that allows driver collaboration or intervention, the steering system can simulate feedback from steering on any interface through which the driver can steer the vehicle. As a non-limiting example, when fully autonomous steering is involved, the vehicle can automatically rotate the handwheel of the vehicle by turning the steerable wheels of the vehicle in conjunction with the autonomous steering system, so that a non-intervening or non-cooperative driver can associate the automatic rotation of the handwheel of the vehicle with the rotation of the steerable wheels of the vehicle, and know, infer or estimate the steering ratio by intuition. Therefore, if the driver expects to assist or intervene in the automatic steering, the driver may not lack directional force when attempting to levy a steering interface that appears to be unassociated or misaligned with the steerable wheels of the vehicle. This concept can be taken into account in any determination of human driver intent by a human appendage tracking module.
[0040] In a number of illustrative variations, an autonomous steering system may include at least one steering interface, at least one sensor or camera capable of sensing or capturing optical data, and at least one logic module for determining whether a human driver wishes to control at least one steering interface based on any raw or processed sensed or collected optical data. Such data may include data such as, but not limited to, human appendage movement rate, movement speed, movement direction, movement acceleration, appendage position, appendage posture, appendage shape, or any other data about appendage appearance, movement, or position. In some such cases, any tracked human appendage may be identified and classified as a particular human appendage (such as, but not limited to, a hand, knee, finger, elbow, or any other human appendage) by using a neural network or database such as a local or cloud database. Any such identification and classification may be associated with any appendage appearance, movement, or position data to provide accurate context for the data collected by the camera or sensor. Any logic module that provides context or analysis for data collected by any such camera or sensor may be referred to as a human appendage tracking module.
[0041] In a plurality of illustrative variations, in order to promote the cooperative steering system, the sensor of the autonomous steering system can be configured to detect and identify the steering input as a driver input. In some cases, the detection of the human driver input can cause the autonomous steering system to hand over all driving tasks to the human driver, such as but not limited to acceleration, braking and steering. In some cases, the autonomous steering system can hand over one or more but not all driving tasks only in response to the human driver input. In some cases, the detection of the human driver input can cause the steering interface to adjust its position or orientation. As a non-limiting example, in the case where at least one steering interface includes a hand wheel, the hand wheel can automatically adjust its telescoping or tilting configuration so as to get along better for the human driver when the human driver input is detected. It is also envisioned that any change in the control or the surrender of control of the autonomous steering system can be accompanied by changes in the surrender of control or other control in other vehicle systems (such as but not limited to engine control systems, brake systems or drive chain systems).
[0042] In a number of illustrative variations, the autonomous vehicle may include at least one human appendage tracking module for determining whether a human driver wishes to control a steering interface based on any raw or processed sensed or collected optical data. In some such cases, a logic module may determine at least one human driver hand position or hand gesture characteristic, and determine based thereon whether the human driver intends to control the steering interface. In some such illustrative variations, any processed optical data may be processed or associated with any available data regarding pressure applied to the steering interface, capacitive touch data from the steering interface, steering interface torque data, steering interface position data, steering interface movement data, or any other data regarding movement or position of the steering interface.
[0043] In a plurality of illustrative variations, neural networks can be used to learn and build a library of human driver appendage positions or postures indicating driver intentions. In such a case, the neural network can be assigned to the task of image analysis, which has the purpose of identifying and marking at least one human hand in the image. Similarly, the neural network can also be assigned to the task of classifying the posture of the human hand identified and marked, and determining the position of the human hand relative to the steering interface. Such a neural network can be trained by collecting thousands of images of many different human drivers and manually marking the hands of each human driver in each image. The marked image can then be used to train the neural network to identify the hand of any human driver and identify the posture and position of any human driver's hand relative to the steering interface. Any system or module communicating with such a neural network can monitor the identification, classification and determination from image analysis, and can use operational logic to determine the driver's intention and convey the driver's intention to the autonomous driving system. Similarly, the neural network itself can be trained to carry out this communication. As a non-limiting example, the flick of all five fingers on two hands or one hand can be a gesture that will be recognized as the intention of representing that the human driver will hand over the task of steering to the autonomous steering system. In such a non-limiting example, a trained neural network can be used to recognize this special gesture and other gestures within a set of collected optical data, and communicate the following to the autonomous steering system via a logic module: the human driver wants to hand over the task of the steering system so that only the steering of the autonomous steering system is performed. Upon receiving such a communication, the autonomous steering system can only adhere to the control of the task of steering the vehicle. It is also envisioned that the neural network can be trained to recognize and track the position and posture of any other human driver appendages (such as but not limited to knees, elbows or forearms). In multiple variations, the steering system can be combined with torque data to further improve the reliability of hand clutch (on-off) detection. The position of the hand can be used with other sensors in the steering system or other monitoring systems to understand the driver's intention and be ready to formulate an action autonomous driving assistance system (ADAS) function before physical contact with the steering interface (such as steering wheel closure) to reduce delays. Sensor fusion of hand detection and other driver monitoring technologies can also be used to understand the driver's intention to assist in better starting ADAS functions. Adding hand detection based on their position on the driving interface (such as the steering wheel), and because she merges it with other physical steering sensors and driver monitoring systems, it will provide a much better proxy for the driver's intention. Such a system also provides early indication before physical contact with the steering wheel. Currently, the driver or demo recorder requires you to take additional actions such as pressing a button on a graphical user interface or a physical button to start the autonomous mode in a self-driving vehicle. This is not very intuitive and results in a waiting time between the time the driver takes his hands off the steering wheel when the vehicle becomes autonomous.Many variations and gestures may be very similar or identical to the driver taking their hands off the steering interface, and in many variations the system may use the occurrence of this gesture to enter autonomous mode.
[0044] In a number of variations, the position of the driver's hand with respect to a steering interface, such as a steering wheel, may be tracked in order to interpret the driver's intent. Areas of the steering interface will also be detected by similar pre-trained neural networks, and the areas will move with the tilt and telescoping steering system. If the driver's hand is in the area for a particular amount of time and / or has a particular posture, it may be interpreted as the driver's intent to take over or relinquish control of the vehicle including the steering system. This signal may be sensor fused using physical sensors such as torque to pre-prepare for the driver to take over, where torque blocks will be eliminated to allow for a smooth takeover to the driver. By identifying driver intent based on the driver's position relative to the steering wheel, faster reactions may be achieved through ADAS functionality by pre-processing certain functions based on driver intent interpretation.
[0045] In a number of illustrative variations, the human appendage tracking module may be able to analyze optical data to determine that the human driver wishes to control the steering interface by using some other extensions of the human driver's body (such as, but not limited to, the knee or thigh). In some such cases, the human appendage tracking module may include a neural network that has been trained to track the posture or position of at least one appendage of the human driver or can be trained to track the posture or position of at least one appendage of the human driver. Such a human appendage tracking module can be configured to track at least one appendage of the human driver relative to a special observation area in the cabin of the vehicle. As a non-limiting example, the posture and position of at least one appendage of the human driver can be tracked with respect to the center of the hand wheel or with respect to the car logo on the hand wheel. In this way, the human appendage tracking module can be configured to determine that at least one tracked appendage in the human driver has changed position and crossed the center of the hand wheel. The human appendage tracking system can identify this appendage movement of the human driver as an intentional gesture to the human appendage tracking module, which should be interpreted as a request by the human driver to the autonomous steering system to autonomously perform a certain action, such as, but not limited to, autonomously change lanes. In some such cases, if the telescoping, tilting, or position of the hand wheel or the center of the vehicle logo on the hand wheel is moved or adjusted, the field of view used by at least one optical sensor or camera for the purpose of image or data collection and analysis may also be moved with the hand wheel. Where appropriate, the necessity or desirability of adjusting the viewing angle or focus of any such sensor or camera to support the adjusted field of view may be communicated by the human appendage tracking module to any appropriate mechanism, system, or module for adjusting such camera or sensor.
[0046] In a number of illustrative variations, the human driver appendage rate and direction of movement may be tracked by a human appendage tracking module. In some such illustrative variations, at least one of the tracking methods of the human appendage tracking module may include a tracking algorithm that takes into account items that partially or completely cover the tracked appendage, such as, but not limited to, gloves, watches, pants, shoes, or any other clothing, accessories, tools, or objects that may completely or partially obscure the complete view of the optical sensing or imaging system of the human appendage.
[0047] In a number of illustrative variations, an optical sensing or imaging system, such as but not limited to a camera system, an infrared sensing or imaging system, an acoustic sensing or imaging system, a laser sensing or imaging system, or any other optical sensing or imaging system may be used to track at least one of a human driver's appendages, such as but not limited to a hand, forearm, elbow, arm, shoulder, neck, head, thigh, knee, calf, foot, or any other body part. An imaging processing algorithm may be used to remove noise from any image or optical data collected, such as but not limited to lighting differences, coloring differences, or any other image noise that may affect the ability of the human appendage tracking module to identify, classify, or track a human appendage.
[0048] In a number of illustrative variations, while the vehicle is autonomously steering, the driver may attempt to collaborate or intervene in steering the vehicle by interacting with at least one steering interface. In some such cases, the steering system may allow the driver to temporarily intervene or collaborate in steering the vehicle upon determining the driver's intent based at least on analysis of image or optical data as disclosed herein. This temporary intervention may be allowed based on a determination by a human appendage tracking module in conjunction with other sensors associated with at least one steering interface that the driver only intends to temporarily intervene in the steering task. In this case, the autonomous steering system may also calculate changes in the direction or path of vehicle travel, and may continuously calculate a new direction or path of vehicle travel as the driver intervenes, such that if the driver ceases to intervene, the vehicle may begin autonomous steering according to one of the new directions or paths.
[0049] In a number of illustrative variations, a hand wheel serves as a steering interface for an autonomous steering system of a vehicle including steerable wheels. In some such variations, if the driver intervenes or collaborates in the steering, any mechanism for automatically rotating the hand wheel of the vehicle in coordination with the autonomous steering system that rotates the steerable wheels of the vehicle may require the driver to apply steering effort thereto or against it as the autonomous steering system simultaneously continues to steer the vehicle. In some cases, the autonomous steering system may negotiate the driver input and the autonomous steering input to determine a net steering input and utilize the net steering input as a control signal for any rotation of the steerable wheels and associated force feedback for the steering interface, such as, but not limited to, automatic rotation of the hand wheel. As a non-limiting example, if the driver begins to loosely grip the hand wheel of the vehicle while the vehicle is in autonomous steering mode, the autonomous steering system may continue to autonomously steer the vehicle and may indicate its intent to steer to the right of the driver by rotating the hand wheel to the right, along with turning the steerable wheels of the vehicle to the right, however while the driver still has a loose grip on the hand wheel, the driver may sense this attempt to turn to the right by the autonomous system and may choose to intervene by gripping the hand wheel more tightly before increasing the amount of turning. In some such cases, if the driver attempts to stop turning the hand wheel to the right, the autonomous steering system may sense the data and determine the driver's intent from it by correlating the driver's intent (which is determined by analyzing image data or optical data captured within the vehicle cabin related to the driver's grip on the hand wheel) with other sensor data that can be used to determine the driver's intent (such as, but not limited to, pressure data from any pressure sensor embedded in the hand wheel, or from a torque sensor trained in, associated with, or embedded in a steering column associated with the hand wheel, or any similar embedded or associated touch sensor of any kind). As a non-limiting example, in some such cases, the autonomous system can greatly reduce but not eliminate the amount of force it applies to the handwheel as a component of the autonomous steering system, thereby making it easy for the driver to commandeer the vehicle without giving the driver the impression that the autonomous steering system has completely handed over control to the driver. In this way, the driver can ensure that the autonomous steering system is still attempting to control the vehicle and will return to controlling the vehicle after the driver releases the handwheel. As another non-limiting example, in some such cases, the autonomous system can completely eliminate the amount of force it applies to the handwheel as a component of the autonomous steering system, thereby making it easy for the driver to commandeer the vehicle and additionally indicate to the driver that the autonomous steering system has completely handed over control to the driver. In some such cases, the driver can be notified by an audio or visual warning that the autonomous steering system is disabled and no longer attempts to control the vehicle.In addition, in this case, the driver can therefore be notified that the steering system will not return to control the vehicle after the driver releases the hand wheel unless the autonomous steering is reactivated. In some such cases, reactivating autonomous steering or driving can be accomplished by using the recognized appendage movement or posture of the human driver. For example, in some such cases, the human driver can utilize a recognized gesture, such as, but not limited to, a gesture, in which each finger of at least one hand of the human driver has been flicked from a curled or clenched posture to a fully extended position. This five-finger flick can be tracked by a human appendage tracking system and can be recognized by a trained neural network, as well as associated with other sensor data to determine the driver's intention to reactivate autonomous driving or turn from the gesture.
[0050] In a number of exemplary variations, if the vehicle is steering autonomously and without driver intervention or cooperation, and the driver attempts to intervene or cooperate in steering, the steering system may determine that driver intent is low or non-existent, the steering system may ignore any unintentional input to steering by the driver and instead steer the vehicle while ignoring the unintentional driver input. As a non-limiting example, in the case of a steer-by-wire system, in the event that the vehicle sensors and systems or any other system that informs the vehicle determines that a steering maneuver is mandatory for safety reasons, the vehicle may ignore any unintentional driver input and autonomously perform the steering maneuver without regard to any unintentional driver input.
[0051] In a number of illustrative variations, the human appendage tracking system may assign at least one confidence level to the tracked appendage. The confidence level may be related to the confidence of the human appendage tracking system that the tracked appendage is performing or has performed the identified gesture. In such a case, the gesture may be registered only when the confidence level for the specific gesture of the specific appendage exceeds a certain threshold level. Other sensor data such as, but not limited to, sensor data related to the steering interface or sensor data from sensors embedded in the steering interface may be associated with the image or optical data analyzed by the human appendage tracking system and further inform at least one confidence level. As a non-limiting example, it may be a case where a lane change gesture is only identified when the confidence in the gesture exceeds 90%. In these cases, some appendage movements are highly similar to a hand-swipe-across-center-of-the-handwheel lane change gesture, but it is not made by a human driver to indicate a desire to change lanes, such as when a human driver reaches across his own body to grab a drink from the center console but does not intend to make a lane change gesture. In some cases, more than one human appendage may be tracked simultaneously. In some such cases, when more than one human appendage is tracked and a conflicting gesture or the same gesture with a conflicting confidence level is determined to have been made by the human appendage tracking module, the steering system or the human appendage tracking system may use a prioritization algorithm or decision tree to determine which gesture to count as input (if any) and in what order to address the gesture inputs.
[0052] In a number of illustrative variations, the steering interface may include a joystick, trackball, slider, throttle, button, toggle switch, lever, touch screen, mouse, or any other known means of user input. In the case of driver input via a driver interface that produces a non-obvious opportunity for visual or force feedback for machine driving, such as but not limited to involving a change in a mouse or touch screen as the primary means of driver input, the automated steering system may indicate the reaction to the driver input via a visual cue (such as but not limited to a flash or animation on a display), an audio cue (such as but not limited to a click, beep, or an instructive voice recording, generated voice, or live voice), or a tactile cue (such as but not limited to vibration of a screen), or any other cue that may indicate to the driver how the driver input (which may be simultaneous with the autonomous input) is being opposed, coordinated, integrated, or processed by the steering system.
[0053] Reference now Figure 1, an illustrative variation in which the autonomous steering system 101 steers the vehicle 102 on the first autonomous path 103. In this illustrative variation, the human appendage tracking module 115 can analyze and mark images or optical data 116 collected by the optical sensing system or imaging system 104 embedded in the rearview mirror 105. The optical sensing system or imaging system 104 can continuously capture images or optical data 116 of the viewing area 106 via a mirror (not shown) positioned behind the human driver 107. When the human driver 107 attempts to intervene in steering the vehicle 102 by engaging the steering interface 108. The optical sensing system or imaging system 104 can capture images or optical data 116 of this attempt with the intervention of the human driver 107 by capturing images or optical data 116 of the hands 109, 110 of the human driver 107. The human appendage tracking module 115 can analyze the images or optical data 116 for identifiable positions or postures relative to the steering interface 108 and identifiable appendages. In this exemplary variation, the human appendage tracking module 115 has identified the hands 109, 110 of the human driver 107 and has marked them with markers 111, 112 and assigned them pose confidence ratings 113, 114, which is accomplished by correlating image or optical data 116 with knowledge of a neural network 117 of image or optical data of human gestures recognizable in the context of steering a vehicle with human hands via a hand wheel. The human appendage tracking module 115 can correlate these pose confidence ratings 113, 114 with touch or pressure data 118, 119 collected from touch or pressure sensors 121, 122 embedded in the steering interface 108 for the purpose of determining the intent 120 of the human driver 107 while engaging the steering interface 108 in this manner. In this illustrative variation, the human appendage tracking module 115 can determine that the human driver's 107 intention 120 is to intervene in the steering by correlating the pose confidence ratings 113, 114 with the touch or pressure data 118, 119 from the touch or pressure sensors 121, 122. This intention 120 can be communicated by the human appendage tracking module 115 to the autonomous steering system 101, which can then hand over control of the steering vehicle to the human driver 107. The human driver 107 can then manually steer the vehicle 102 via the steering interface 108 without being influenced by the autonomous steering system 101.
[0054] Reference now Figure 2, an illustrative variation in which a human driver 207 first manually steers a vehicle 202 including an autonomous steering system 201. In this illustrative variation, a human appendage tracking module 215 can analyze and mark images or optical data 216 collected by an optical sensing system or imaging system 204 embedded in a rearview mirror 205. The optical sensing system or imaging system 204 can continuously capture images or optical data 216 of a viewing area 206 via a mirror (not shown) positioned behind the human driver 207. The human driver 207 can begin to disengage the steering interface 208 by releasing his or her grip on the steering interface 208. The optical sensing system or imaging system 204 can capture images or optical data 216 of such disengagement by the human driver 207 by capturing images or optical data 216 of the hands 209, 210 of the human driver 207. The human appendage tracking module 215 can analyze the images or optical data 216 for identifiable positions or postures relative to the steering interface 208 and identifiable appendages. In this exemplary variation, the human appendage tracking module 215 has identified the hands 209, 210 of the human driver 207 and has marked them with markers 211, 212 and assigned them pose confidence ratings 213, 214, which is done by correlating image or optical data 216 with knowledge of a neural network 217 of image or optical data of human gestures recognizable in the context of steering a vehicle with human hands via a hand wheel. The human appendage tracking module 215 can correlate these pose confidence ratings 213, 214 with touch or pressure data 218, 219 collected from touch or pressure sensors 221, 222 embedded in the steering interface 208 for the purpose of determining the intention 220 of the human driver 207 while disengaging the steering interface 208 in this manner. In this illustrative variation, the human appendage tracking module 215 may determine that the intention 220 of the human driver 207 is to disengage the steering interface 208 and hand over the task of steering the vehicle 202 to the autonomous steering system 201 by correlating the pose confidence ratings 213, 214 with the touch or pressure data 218, 219 from the touch or pressure sensors 221, 222. This intention 220 may be communicated by the human appendage tracking module 215 to the autonomous steering system 201, which may then take exclusive control of the task of steering the vehicle 202. The autonomous steering system 201 may then begin to autonomously steer the vehicle 202 along the autonomous path 203, and exclude any steering inputs determined to be unintentional by the human driver 207.
[0055] Figure 3A is an illustration of a driver's hand 110 on a steering wheel 108 with a plurality of fingers 123 curled around and / or gripping the steering wheel 108 according to a number of variations.
[0056] Figure 3B 200 according to a number of variations, for example, in the field of view of an optical sensor positioned to observe the front side 200 of the steering wheel. Figure 3A The driver's hand 110 is shown in FIG.
[0057] Figure 3C 1 shows a field of view of an optical sensor positioned to observe the side edge of the steering wheel 108 according to a number of variations, for example Figure 3A The driver's hand 110 is shown in FIG.
[0058] Figure 4A is an illustration of a driver's hand 110 in the field of view of an optical sensor positioned to view a front side 200 of a steering wheel 108 according to multiple variations, wherein the driver is making a gesture indicating the driver's intent to transfer steering control (turnover) to a vehicle autonomous driving system by flicking multiple fingers 123 of the driver's hand, wherein the multiple fingers are extended and not curled around or gripping the steering wheel.
[0059] Figure 4B 1 shows, for example, a view of a front view of a steering wheel in the field of view of an optical sensor positioned to observe the front of the steering wheel according to a number of variations. Figure 4A The driver's hand 110 is shown in FIG.
[0060] Figure 4C is an illustration of a driver's hand 110 in the field of view of an optical sensor positioned to observe a side edge of a steering wheel 108 according to multiple variations, wherein the driver is making a gesture indicating the driver's intent to hand over steering control to a vehicle autonomous driving system by flicking the driver's fingers 123, wherein the fingers are extended and not curled around or gripping the steering wheel 108.
[0061] Figure 4D Data is shown that an optical sensor according to multiple variations may generate for a driver's hand 110 in the optical sensor's field of view, where the optical sensor is positioned to view a side edge of a steering wheel, where the driver is making a gesture that indicates the driver's intent to hand over steering control to a vehicle autonomous driving system by flicking the driver's multiple fingers 123, where the multiple fingers are extended and not curled around or gripping the steering wheel.
[0062] Figure 4E is an illustration of a driver indicating the driver's intent to hand over steering control to a vehicle autonomous driving system by completely removing the driver's hands from steering wheel 108 according to a number of variations.
[0063] Figure 5A A driver's hand 110 is shown in the field of view of an optical sensor positioned to observe the side edge of a steering wheel 108 according to a number of variations, with multiple fingers 123 of the driver's hand 110 curled around the steering wheel, indicating the driver's intent to take over control of the vehicle's steering system.
[0064] Figure 5B Data that an optical sensor may generate for a driver's hand 110 in the optical sensor's field of view is shown, wherein the optical sensor is positioned to observe the side edge of the steering wheel, with multiple fingers 123 of the driver's hand 110 curled around the steering wheel, thereby indicating the driver's intent to take over control of the vehicle's steering system.
[0065] Fig. 6A The driver's hands 109, 110 are shown in the field of view of an optical sensor positioned to observe the side edge of the steering wheel 108 according to multiple variations, wherein multiple fingers 123 of each of the driver's hands 109, 110 are curled around the steering wheel 108, thereby indicating the driver's intention to take over control of the vehicle's steering system.
[0066] Figure 6B Data that an optical sensor may generate for a driver's hands 109, 110 in the optical sensor's field of view is shown, wherein the optical sensor is positioned to observe the side edge of the steering wheel, with multiple fingers 123 of each of the driver's hands 109, 110 curled around the steering wheel, indicating the driver's intent to take over control of the vehicle's steering system.
[0067] Figure 7 The field of view of the optical sensor positioned to view the front face 200 of the steering wheel 108 is shown, and the detection area where the driver has placed his hand is shown, where the placement of the hand indicates the driver's intention to take control of the vehicle's steering system. Figure 7 As shown in , object recognition software can be used to determine within a predetermined level of acceptability / confidence (e.g., within a 99% confidence level that the driver's hands 109, 110 are within the detection area). Figure 7 and Figure 8 As shown in , the detection area can be generally elliptical, but may also be circular, square, rectangular, diamond-shaped, or any other shape that would be large enough so that a predetermined portion or sufficient portion of at least one of the driver's hands placed on the steering interface would be within the detection area to indicate the driver's intent to control the vehicle's steering system.
[0068] Figure 8The field of view of an optical sensor positioned to view the front of the steering wheel is shown according to a number of variations, and shows a detection area where the driver has moved his hand to the outside to indicate the driver's intent to hand over control of the vehicle's steering system to the autonomous driving system. Figure 8 As shown in , object recognition software can be used to determine within a predetermined level of acceptability / confidence (e.g., within a 99% confidence level) that a predetermined portion or sufficient portion of each of the driver's hands 109, 110 is outside the detection area, thereby indicating the driver's intent to hand over control of the vehicle's steering system to the autonomous driving system.
[0069] In a number of exemplary variations, any number of modules may be combined together or broken down into smaller modules.
[0070] The following description of variations only illustrates components, elements, actions, products and methods that are considered to be within the scope of the present invention, and does not limit such scope in any way by specifically disclosed or unexplicitly stated content. Components, elements, actions, products and methods as described herein may be combined and rearranged differently than those explicitly described herein, and are still considered to be within the scope of the present invention.
[0071] Variant 1 may include a product comprising: an autonomous steering system comprising a steering interface and being constructed and arranged to simultaneously accept and interpret human driver steering commands input by a human driver via at least one steering interface and autonomous steering commands input by the autonomous steering system, wherein the autonomous steering system is capable of switching between at least a first steering mode and a second steering mode, the first steering mode being a steering mode in which the steering occurs solely based on autonomous steering commands generated by the autonomous steering system, and the second steering mode being a steering mode in which the steering occurs solely based on human driver steering commands generated by a human driver via manipulation of the at least one steering interface, wherein the product further comprises at least one camera or optical sensor, the at least one camera or optical sensor being configured to track at least one of a position or posture of a human appendage, and wherein the autonomous steering system is configured to switch between steering modes based at least on the tracked position or posture of the at least one human appendage.
[0072] Variation 2 may include the product of any of Variations 1 or 3-11 wherein the autonomous steering system is further capable of switching to a third steering mode wherein the at least one steering interface is manipulated by both the human driver and the autonomous steering system.
[0073] Variation 3 may include the product of any of Variations 1-2 or 4-11 wherein the at least one camera or optical sensor comprises a stereo camera arrangement.
[0074] Variation 4 may include a product of any of Variations 1-3 or 5-11 wherein the product wherein the at least one camera or optical sensor is configured to track a speed of movement of the at least one human appendage.
[0075] Variation 5 may include the product of any of Variations 1-4 or 6-11 wherein the at least one camera or optical sensor is configured to track proximity of the at least one human appendage to the at least one steering interface.
[0076] Variation 6 may include a product of any of Variations 1-5 or 7-11 wherein the product further comprises at least one torque sensor for sensing torque input to the steering interface by the human driver.
[0077] Variation 7 may include the product of Variation 6 wherein the torque input is torque sensed at a steering column.
[0078] Variant 8 may include a product of any of Variants 1-7 or 9-11 wherein the product further comprises at least one pressure sensor coupled to the steering interface for sensing pressure applied to the steering interface by the human driver.
[0079] Variant 9 may include a product of any of Variants 1-8 or 10, wherein the product further includes at least one capacitive touch sensor coupled to the steering interface for sensing a change in capacitance applied at the capacitive touch sensor when the human driver touches the steering interface.
[0080] Variant 10 may include the product of any of Variants 1-9, wherein the configuration of the at least one optical sensor or camera for tracking the position or posture of the at least one human appendage includes at least one camera positioned to capture an image of at least one of the human driver's hands on or near the steering wheel.
[0081] Variant 11 may include the product of variant 10, wherein the configuration of the autonomous steering for switching between steering modes based at least on the tracked position or posture of the at least one human appendage includes using an image analysis system to determine that each finger of the at least one hand of the human driver has been flicked from a curled or clenched posture to a fully extended position, and wherein the autonomous steering system is configured to change the steering mode based at least on this determination.
[0082] Variant 12 may include a method comprising: tracking at least one of a position or posture of at least one appendage of a human driver using at least one camera or optical sensor; generating associated data by associating at least one of the position or posture of the at least one appendage of the human driver with data collected from at least one steering interface or from at least one sensor configured to sense interaction by the human driver with the at least one steering interface; and changing a steering mode of an autonomous steering system based at least on the associated data.
[0083] Variation 13 may include the method of any of Variations 12 or 14-17 wherein the at least one camera or optical sensor comprises a stereo camera arrangement.
[0084] Variant 14 may include the method of any of Variants 12-13 or 15-17, wherein the data collected by the human driver from at least one steering interface or from at least one sensor configured to sense interaction with the at least one steering interface includes data from at least one capacitive touch sensor embedded in the at least one steering interface.
[0085] Variant 15 may include the method of any of Variants 12-14 or 16-17, wherein the data collected by the human driver from at least one steering interface or from at least one sensor configured to sense interaction with the at least one steering interface includes at least one torque sensor for sensing torque input by the human driver to the steering interface.
[0086] Variation 16 may include the method of Variation 15 wherein the torque input is torque sensed at a steering column.
[0087] Variant 17 may include the method of any of Variants 12-16, wherein the data collected by the human driver from at least one steering interface or from at least one sensor configured to sense interaction with the at least one steering interface includes at least one pressure sensor coupled to the steering interface for sensing pressure applied to the steering interface by the human driver.
[0088] Variant 18 may include a product comprising: at least one steering interface for an autonomous steering system for a vehicle; at least one optical sensor or camera for tracking at least one of a posture or position of at least one human driver appendage; and at least one sensor other than the optical sensor embedded in the steering interface.
[0089] Variation 19 may include the product of any of Variations 18 or 20 wherein the at least one sensor other than the optical sensor is a capacitive touch sensor.
[0090] Variation 20 may include the product of any of Variations 18-19 wherein the at least one sensor other than the optical sensor is a pressure sensor.
[0091] Variation 21 may include a product comprising: an autonomous steering system, the autonomous steering system including a steering interface and being constructed and arranged to simultaneously accept and interpret human driver steering commands input by a human driver via at least one steering interface and autonomous steering commands input by the autonomous steering system, wherein the autonomous steering system is capable of switching between at least a first steering mode and a second steering mode, the first steering mode being a steering mode in which steering occurs solely based on autonomous steering commands generated by the autonomous steering system, and the second steering mode being a steering mode in which steering occurs solely based on human driver steering commands generated by a human driver via manipulation of the at least one steering interface, wherein the product further comprises at least one camera or optical sensor, The at least one camera or optical sensor is configured to track at least one of a position or a posture of a human appendage, and wherein the autonomous steering system is configured to switch between steering modes based at least on the tracked position or posture of the at least one human appendage, wherein the at least one camera or optical sensor is positioned to observe a front face of the steering interface, and wherein the autonomous driving system is capable of identifying a detection area using the at least one camera or optical sensor and determining whether a predetermined portion of the human appendage has been placed within the detection area and, if so, switching to the second mode, and wherein the autonomous driving system is capable of determining whether a predetermined portion of the human appendage has been placed outside of the detection area and, if so, switching to the first mode.
[0092] Variation 22 may include a product as set forth in Variation 21 wherein the human appendage comprises a human hand.
[0093] Variation 23 may include a method comprising: providing an autonomous steering system, the autonomous steering system comprising a steering interface and being constructed and arranged to simultaneously accept and interpret human driver steering commands input by a human driver via at least one steering interface and autonomous steering commands input by the autonomous steering system, wherein the autonomous steering system is capable of switching between at least a first steering mode and a second steering mode, the first steering mode being a steering mode in which steering occurs solely based on autonomous steering commands generated by the autonomous steering system, and the second steering mode being a steering mode in which steering occurs solely based on human driver steering commands generated by a human driver via manipulation of the at least one steering interface, wherein the product further comprises at least one camera or an optical sensor, the at least one camera or optical sensor being configured to track at least one of a position or a posture of a human appendage, and wherein the autonomous steering system is configured to switch between steering modes based at least on the tracked position or posture of the at least one human appendage, wherein the at least one camera or optical sensor is positioned to view a front face of the steering interface, and wherein the autonomous driving system is capable of identifying a detection area using the at least one camera or optical sensor, and determining whether a predetermined portion of the human appendage has been placed within the detection area, and if so switching to the second mode, and determining whether a predetermined portion of the human appendage has been placed outside of the detection area, and if so switching to the first mode.
[0094] Variation 24 may include a method as set forth in Variation 23 wherein the human appendage comprises a human hand.
[0095] The foregoing description of selected variations within the scope of the present invention is merely illustrative in nature, and thus variations or modifications thereof are not to be regarded as departing from the spirit and scope of the present invention.
Claims
1. A product for autonomous steering, include: An autonomous steering system comprising a steering interface and constructed and arranged to simultaneously accept and interpret human driver steering commands input by a human driver via at least one steering interface and autonomous steering commands input by the autonomous steering system, wherein the autonomous steering system is capable of switching between at least a first steering mode and a second steering mode, the first steering mode being a steering mode in which steering occurs solely based on autonomous steering commands generated by the autonomous steering system, and the second steering mode being a steering mode in which steering occurs solely based on human driver steering commands generated by a human driver via manipulation of the at least one steering interface, wherein the product further comprises at least one camera or optical sensor configured to track at least one of a position or a posture of a human appendage. The autonomous steering system is configured to switch between steering modes based at least on a tracked position or posture of the at least one human appendage, wherein the configuration of the at least one optical sensor or camera for tracking the position or posture of the at least one human appendage includes at least one camera positioned to capture an image of at least one of the human driver's hands on or near a steering wheel, wherein the configuration of the autonomous steering for switching between steering modes based at least on the tracked position or posture of the at least one human appendage includes using an image analysis system to determine that each finger of the at least one hand of the human driver has been flicked from a curled or clenched posture to a fully extended position, and wherein the autonomous steering system is configured to change the steering mode based at least on this determination.
2. The product of claim 1, wherein the autonomous steering system is further capable of switching to a third steering mode, wherein the at least one steering interface is manipulated by both the human driver and the autonomous steering system.
3. The product of claim 2, wherein the at least one camera or optical sensor comprises a stereo camera arrangement.
4. The product of claim 2, wherein the product wherein the at least one camera or optical sensor is configured to track a speed of movement of the at least one human appendage.
5. The product of claim 2, wherein the at least one camera or optical sensor is configured to track proximity of the at least one human appendage to the at least one steering interface.
6. The product of claim 2, wherein the product further comprises at least one torque sensor for sensing torque input to the steering interface by the human driver.
7. A product as set forth in claim 6 wherein the torque input is torque sensed at a steering column.
8. The product of claim 2, wherein the product further comprises at least one pressure sensor coupled to the steering interface for sensing pressure applied to the steering interface by the human driver.
9. The product of claim 2, wherein the product further comprises at least one capacitive touch sensor coupled to the steering interface for sensing a change in capacitance applied at the capacitive touch sensor when the human driver touches the steering interface.
10. A method for autonomous steering, include: tracking at least one of a position or a posture of at least one appendage of the human driver using at least one camera or optical sensor; generating correlated data by correlating the at least one of the position or posture of the at least one appendage of the human driver with data collected from at least one steering interface or from at least one sensor configured to sense interaction by the human driver with the at least one steering interface; as well as and changing a steering mode of an autonomous steering system based at least on the associated data, wherein tracking the position or posture of the at least one human appendage using at least one optical sensor or camera includes positioning at least one camera to capture an image of at least one of the human driver's hands on or near a steering wheel, determining using an image analysis system that each finger of at least one of the human driver's hands has been flicked from a curled or clenched posture to a fully extended position, and wherein the autonomous steering system is configured to change the steering mode based at least on the determination. The method of claim 10 , wherein the at least one camera or optical sensor comprises a stereo camera arrangement.
12. A method according to claim 10, wherein the data collected by the human driver from at least one steering interface or from at least one sensor configured to sense interaction with the at least one steering interface includes data from at least one capacitive touch sensor embedded in the at least one steering interface.
13. The method of claim 10, wherein the data collected by the human driver from at least one steering interface or from at least one sensor configured to sense interaction with the at least one steering interface includes at least one torque sensor for sensing torque input by the human driver to the steering interface.
14. The method of claim 13, wherein the torque input is a torque sensed at a steering column.
15. The method of claim 10, wherein the data collected by the human driver from at least one steering interface or from at least one sensor configured to sense interaction with the at least one steering interface includes at least one pressure sensor coupled to the steering interface for sensing pressure applied to the steering interface by the human driver.
16. A product for autonomous steering, include: An autonomous steering system comprising a steering interface and constructed and arranged to simultaneously accept and interpret human driver steering commands input by a human driver via at least one steering interface and autonomous steering commands input by the autonomous steering system, wherein the autonomous steering system is capable of switching between at least a first steering mode and a second steering mode, the first steering mode being a steering mode in which the steering occurs solely based on autonomous steering commands generated by the autonomous steering system, and the second steering mode being a steering mode in which the steering occurs solely based on human driver steering commands generated by the human driver via manipulation of the at least one steering interface, wherein the product further comprises at least one camera or optical sensor configured to track at least one of a position or posture of a human appendage, and wherein the autonomous steering system is configured to switch between steering modes based at least on the tracked position or posture of the at least one human appendage, wherein the at least one camera or optical sensor is positioned to view a front face of the steering interface, and wherein the autonomous steering system is configured to determine using an image analysis system that each finger of at least one hand of the human driver has been flicked from a curled or clenched posture to a fully extended position, and wherein the autonomous steering system is configured to change the steering mode based at least on that determination.
17. A method for autonomous steering, include: An autonomous steering system is provided, the autonomous steering system including a steering interface and being constructed and arranged to simultaneously accept and interpret human driver steering commands input by a human driver via at least one steering interface and autonomous steering commands input by the autonomous steering system, wherein the autonomous steering system is capable of switching between at least a first steering mode and a second steering mode, the first steering mode being a steering mode in which the steering occurs solely based on autonomous steering commands generated by the autonomous steering system, and the second steering mode being a steering mode in which the steering occurs solely based on human driver steering commands generated by the human driver via manipulation of the at least one steering interface, wherein at least one camera or optical sensor is configured to track at least one of a position or posture of a human appendage, and wherein the autonomous steering system is configured to switch between steering modes based at least on the tracked position or posture of the at least one human appendage, wherein the at least one camera or optical sensor is positioned to view a front face of the steering interface, and wherein the autonomous steering system is configured to determine using an image analysis system that each finger of at least one hand of the human driver has been flicked from a curled or clenched posture to a fully extended position, and wherein the autonomous steering system is configured to change the steering mode based at least on that determination.
