Gesture-based control for semi-autonomous vehicles

Abstract

Various embodiments include methods and vehicles, such as semi-autonomous vehicles, for safely operating a vehicle based on vehicle control gestures made by an occupant. Exemplary implementations may include: determining a first vehicle action by applying a first passenger profile to a detected first vehicle control gesture performed by the first passenger; determining whether performing the first vehicle action is safe; and, in response to determining that the first vehicle action is safe for the vehicle and the occupant, operating the vehicle to perform the first vehicle action. The first passenger profile may be selected from a plurality of passenger profiles to normalize the vehicle control gestures received from the first passenger. In response to determining that the first vehicle action is unsafe for the vehicle or the occupant, the vehicle control gesture may be ignored, or a vehicle action different from but similar to the first vehicle action may be performed.

Description

[0001] Related applications

[0002] This application claims the benefit of priority to U.S. Patent Application Serial No. 16 / 893,112, filed June 4, 2020, entitled “Gesture-Based Control For Semi-Autonomous Vehicle,” the entire contents of which are incorporated herein by reference for all purposes. Background Technology

[0003] As industry moves towards deploying autonomous and semi-autonomous vehicles, cars and trucks are becoming increasingly intelligent. With the advent of fully autonomous and semi-autonomous vehicles, passengers may no longer need to use a conventional steering wheel for vehicle control. In fact, conventional steering wheels are expensive and pose a danger to passengers in the event of a collision. However, passengers may still occasionally want to steer or control navigation in other ways. Summary of the Invention

[0004] The aspects include methods and vehicles, such as semi-autonomous vehicles, for implementing a method for safely operating a vehicle based on passenger-based vehicle control gestures. The aspects may include: determining a first vehicle action by applying a first passenger profile to a detected first vehicle control gesture performed by a first passenger, wherein the first passenger profile is selected from a plurality of passenger profiles to normalize the vehicle control gesture received from the first passenger; and operating the vehicle to perform the first vehicle action in response to determining that the first vehicle action is safe for the vehicle and the occupants. In some aspects, the plurality of passenger profiles may include a second passenger profile designated for a second passenger, wherein the second passenger profile normalizes the vehicle control gesture differently from the first passenger profile. Some aspects may also include: identifying the first passenger based on at least one of the first passenger's location in the vehicle, the first passenger's input, or the first passenger's identification; and selecting the first passenger profile based on the first passenger's identity. In some aspects, the first passenger profile may be customized for the first passenger based on training input previously received from the first passenger. Some aspects may also include: receiving the first passenger profile from a remote computing device for application to the vehicle control gestures.

[0005] Some aspects may also include: determining whether a first vehicle action associated with the detected first vehicle control gesture is safe for the operation of the vehicle; in response to determining that the first vehicle action is unsafe for the operation of the vehicle, determining a second vehicle action as a safer alternative to the first vehicle action; and operating the vehicle to perform the second vehicle action. Some aspects may also include: determining whether the first vehicle action associated with the detected first vehicle control gesture is safe for the vehicle to operate after a determined delay period; and in response to determining that the first vehicle action associated with the detected first vehicle control gesture is safe for the vehicle to operate after the determined delay period, operating the vehicle to perform the first vehicle action after the determined delay period.

[0006] Some aspects may also include: determining whether the first vehicle action includes an unusual vehicle operation; and in response to determining that the first vehicle action may include the unusual vehicle operation, determining whether the detected first vehicle control gesture may include additional instructions to the first passenger anticipating the unusual vehicle operation, wherein operating the vehicle to perform the first vehicle action is further in response to determining that the detected first vehicle control gesture may include additional instructions to the first passenger anticipating the unusual vehicle operation. In some aspects, the additional instructions may include at least one of the following: an exaggerated gesture, a repetitive gesture, a gesture performed faster than usual, or non-visual input received in conjunction with the detected first vehicle control gesture. Some aspects may also include: determining whether the unusual vehicle operation is safe for the vehicle or the occupant; and in response to determining that the unusual vehicle operation is unsafe for the vehicle or the occupant, operating the vehicle to perform a second vehicle action as a safer alternative to the first vehicle action.

[0007] Other aspects include a vehicle control system having a processor configured to detect and implement vehicle control gestures and perform operations of any of the methods outlined above. Other aspects include a non-transitory processor-readable storage medium storing processor-executable software instructions configured to cause the processor to perform operations of any of the methods outlined above. Other aspects include a vehicle control system having components for performing functions of any of the methods outlined above. Other aspects include a vehicle control gesture system for use in a vehicle, the vehicle including a processor configured with processor-executable instructions to perform operations of any of the methods outlined above. Other aspects include a processing device configured for use in a vehicle control system and configured to perform operations of any of the methods outlined above. Attached Figure Description

[0008] The accompanying drawings, which are incorporated herein and constitute a part of this specification, illustrate exemplary embodiments and, together with the general description given above and the detailed description given below, serve to explain the features of the various embodiments.

[0009] Figure 1A and Figure 2B This is a schematic diagram illustrating a vehicle gesture detection system suitable for implementing various embodiments.

[0010] Figure 2A and 2B This is a block diagram of components for a vehicle suitable for implementing various embodiments.

[0011] Figure 3 This is a component block diagram illustrating components suitable for implementing various embodiments of a vehicle.

[0012] Figure 4A This is a component block diagram illustrating the components of an example vehicle management system according to various embodiments.

[0013] Figure 4B This is a component block diagram illustrating components of another example vehicle management system according to various embodiments.

[0014] Figure 5 This is a block diagram illustrating components of an example on-chip system for use in a vehicle according to various embodiments, the on-chip system being configured to operate based on passenger vehicle control gestures.

[0015] Figure 6 This is a component block diagram of an example vehicle computing system according to various embodiments, the vehicle computing system being configured to operate a vehicle based on passenger vehicle control gestures.

[0016] Figure 7A , Figure 7B and Figure 7C Examples of vehicle operation based on passenger vehicle control gestures according to various embodiments are shown.

[0017] Figure 8A and Figure 8B Examples of vehicle operation based on vehicle control gestures according to various embodiments are shown.

[0018] Figure 9A , Figure 9B , Figure 9C and Figure 9D This is a process flowchart of an example method for operating a vehicle based on passenger-based vehicle control gestures, according to various embodiments. Detailed Implementation

[0019] Various aspects will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. Reference to specific examples and embodiments is for illustrative purposes and is not intended to limit the scope of the aspects or the claims.

[0020] Various embodiments provide methods for recognizing vehicle control gestures from passengers, executed by a processor in a semi-autonomous vehicle, intended for steering and speed control of the autonomous vehicle. The processor can select a passenger profile from multiple passenger profiles to normalize the vehicle control gestures received from the passenger. The processor can determine vehicle actions by applying the passenger profile or using generic parameters applicable to anyone to the detected vehicle control gestures performed by the passenger, and confirm that the detected vehicle control gestures are safe for vehicle operation. The processor can execute safe vehicle commands in a safe manner, regardless of the emphasis used by the passenger when making the vehicle control gestures.

[0021] Various embodiments include a vehicle system configured to recognize gestures made by passengers for controlling a semi-autonomous vehicle. The vehicle system may include sensors (including one or more cameras) and sensor processing capabilities (e.g., motion and / or image processing) to recognize certain passenger movements (i.e., gestures) as vehicle control gestures made by the passenger. For example, a vehicle occupant waving her hand from left to right in mid-air, pretending to turn the wheels to the right, pointing to the right side of the vehicle, and / or giving other predefined hand signals / movements may be specified and recognized as vehicle control gestures for turning right. Once the sensor processing capabilities recognize the specified vehicle control gesture, the translation of this recognized vehicle control gesture, representing vehicle movement, can be processed by a vehicle control unit for controlling the operation of the vehicle.

[0022] The process of recognizing vehicle control gestures can take into account the history of passenger input. This history can serve as the basis for one or more passenger profiles reflecting characteristics of how a passenger actually performs a given vehicle control gesture, including consistency deviations from model gestures. Each passenger profile can be specific to a particular passenger and associated with one or more specific seats within the vehicle (e.g., a traditional driver's seat), or it can be generalized to any passenger and / or any seating position. Passenger profiles can be used to verify that detected gestures are consistent with the passenger's typical or previous inputs. Furthermore, passenger profiles can be used to better identify valid inputs and reduce the chance of detecting erroneous gestures.

[0023] As used herein, the term "passenger" refers to an occupant of a vehicle, particularly a semi-autonomous vehicle, which may include one or more drivers and one or more other occupants.

[0024] As used herein, the term “passenger profile” refers to a set of data that reflects key characteristics of one or more passengers making vehicle control gestures or the manner in which they make particular vehicle control gestures.

[0025] As used herein, the term "gesture" refers to a movement or posture using body parts, particularly the hands and / or arms, to express an idea or meaning. More specifically, the term "vehicle control gesture" refers to a movement or posture configured to control one or more actions of a semi-autonomous or autonomous vehicle.

[0026] As used herein, the term "safe" or "secure" is used synonymously and refers to a situation / condition that is protected from or not significantly exposed to danger. In various embodiments, a situation / condition may be considered safe when the probability of damage or injury to one or more passengers, vehicles, and / or parts of vehicles is less than a threshold. While operating a vehicle may always involve some level or risk, a predetermined low-risk level (i.e., the safety threshold) may be considered safe (i.e., acceptable). Additionally, the processor may calculate the probability of damage or injury to more than one thing (e.g., passengers, vehicles, and / or parts of vehicles) and the probability of individual occurrence of more than one type of damage or injury to these things. Furthermore, the same or different safety thresholds may be applied to calculate the risk of each of these things.

[0027] As used herein, the term "computing device" refers to an electronic device equipped with at least one processor, a communication system, and memory configured with a contact database. For example, a computing device may include any or all of the following: cellular phones, smartphones, portable computing devices, personal or mobile multimedia players, laptop computers, tablet computers, 2-in-1 laptop / desktop computers, smartbooks, ultrabooks, handheld computers, wireless email receivers, cellular phones supporting multimedia internet, wearable devices including smartwatches, entertainment devices (e.g., wireless game controllers, music and video players, satellite radios, etc.), and similar electronic devices including memory, wireless communication components, and programmable processors. In various embodiments, the computing device may be configured with memory and / or storage devices. Additionally, the computing devices mentioned in the various example embodiments may be coupled to or include wired or wireless communication capabilities implementing various embodiments, such as network transceivers and antennas configured to communicate with wireless communication networks.

[0028] The term "System-on-a-Chip" (SOC) is used herein to refer to a set of interconnected electronic circuits, typically but not exclusively including one or more processors, memory, and communication interfaces. An SOC can include a variety of different types of processors and processor cores, such as general-purpose processors, central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), accelerated processing units (APUs), subsystem processors, auxiliary processors, single-core processors, and multi-core processors. An SOC can further embody other hardware and hardware combinations, such as field-programmable gate arrays (FPGAs), configuration and status registers (CSRs), application-specific integrated circuits (ASICs), other programmable logic devices, discrete gate logic, transistor logic, registers, performance monitoring hardware, watchdog hardware, counters, and time references. An SOC can be an integrated circuit (IC) configured such that the components of the IC reside on the same substrate, such as a monolithic semiconductor material (e.g., silicon).

[0029] Autonomous and semi-autonomous vehicles, such as cars, trucks, and buses, are becoming a reality on the roads. Furthermore, autonomous and semi-autonomous vehicles typically include multiple sensors, including cameras, radar, and lidar, that collect information about the vehicle's surroundings. For example, the information collected in this way can enable the vehicle to recognize roads, identify objects to avoid, and track the movement and future positions of other vehicles to achieve partial or full autonomous navigation. According to various embodiments, the use of such sensors can be extended inside the vehicle to detect vehicle control gestures, allowing passengers to control the vehicle's movements without the use of traditional steering wheels, joysticks, or similar direct mechanical controls.

[0030] Various embodiments can be implemented in various semi-autonomous vehicles that include a gesture detection system, wherein example vehicle gesture detection system 50 is... Figure 1A and Figure 1B As shown in the image. (Reference) Figure 1A and Figure 1B The vehicle gesture detection system 50 may include one or more sensors 101, a profile database 142, and a gesture recognition engine 144, which are coupled to and / or work with the control unit 140 of the vehicle 100. The vehicle gesture detection system 50 may be configured to detect predefined gestures 61, 62 performed by passengers 11, 12 within the vehicle 100.

[0031] By using gestures, one or more passengers can provide control input to the processor of vehicle 100 without physically touching the steering wheel or other mechanical controllers. Sensor 101 may include motion sensors, proximity sensors, cameras, microphones, impact sensors, radar, lidar, satellite geolocation system receivers, tire pressure sensors, etc., which can be configured to capture images or other characteristics of movements and / or postures performed by passengers. Movements and / or postures can include all movements and / or postures, including predefined gestures associated with one or more vehicle control gestures, as well as other movements or postures unrelated to vehicle control. In this way, sensor 101 does not need to distinguish the type of movement or posture, but can instead transmit data about the captured images or other characteristics to gesture recognition engine 144 for analysis.

[0032] The profile database 142 can maintain multiple passenger profiles, which are customized to normalize vehicle control gestures for specific passengers, passenger groups, and / or passengers in specific seats within the vehicle 100. The profile database 142 can be configured to transmit data about one or more of the multiple passenger profiles to the gesture recognition engine 144 for analysis therein.

[0033] Passenger profiles may include data relating to how a particular person makes gestures and / or the specific gestures that person chooses to use. Therefore, profile database 142 may include different passenger profiles assigned to different people. Each particular person may be identified based on input from the passenger or by identification of the passenger via sensor 101. Thus, input from the passenger can provide an identity-based and / or personalized passenger profile, which can be added to or updated in profile database 142. Input from the passenger may come from a user interface associated with vehicle 100, the passenger's personal computing device (e.g., a smartphone), and / or another input device. Therefore, passenger profiles may be selected based on the passenger's identity and / or may be specific to a unique individual. Furthermore, passenger profiles can be customized for identified passengers based on training input previously received from the passenger. Additionally, more than one passenger profile may be associated with a single unique individual. Therefore, individuals may select specific passenger profiles to suit their mood (e.g., tired, excited, thrilled, etc.), time of day, and / or other criteria.

[0034] Alternatively, multiple passenger profiles can be customized to standardize vehicle control gestures for passengers in one or more specific seats within the vehicle. For example, a passenger profile can be assigned to any individual sitting in a designated "driver's seat," and one or more other passenger profiles can be assigned to any individual sitting in other seats. As another alternative, a single passenger profile can be assigned to all passengers, regardless of who they are or where they are sitting. Similarly, the profile database 142 can maintain one or more default passenger profiles for use in cases where a passenger is not identified or a customized passenger profile has not been set for that passenger.

[0035] In some embodiments, gesture recognition engine 144 may use neural network processing and / or artificial intelligence methods to determine whether the movement and / or posture captured by sensor 101 matches a predefined gesture associated with vehicle control. Additionally, gesture recognition engine 144 may use data from passenger profiles received from profile database 142 to determine the expected vehicle action of the movement and / or posture captured by sensor 101. Therefore, taking into account the movement or posture captured by sensor 101 and the passenger profiles conveyed from profile database 142, gesture recognition engine 144 may transmit instructions or data to control unit 140 to cause control unit 140 to operate vehicle 100 in a manner that implements a specified vehicle action. In various embodiments, before initiating any vehicle action, control unit 140 may ensure that the specified vehicle action is safe for vehicle 100 and occupants (e.g., passengers 11, 12).

[0036] Vehicle actions such as acceleration, deceleration, and lane changes can vary in amount or magnitude. In some implementations, in addition to matching the captured movement and / or posture with predefined gestures associated with vehicle control, the gesture recognition engine 144 can also evaluate measurable parameters of the captured movement (e.g., angle, distance, etc.) to interpret the intended vehicle control input. For ease of description, the rotation angle or degree of the gesture, distance or sweep, speed of movement of the gesture, acceleration during the gesture, and other measurable parameters of the observed gesture are referred to herein as the “degree” of the captured movement. The gesture recognition engine 144 can interpret the degree of the captured movement to determine the magnitude of the vehicle movement expected by the person making the gesture. For example, a passenger may wave her hand in the air at a 15-degree arc to convey a command for lane changing, but only one lane can be changed along the direction of the wave. A similar wave across 45 degrees or more may indicate a command for changing two lanes along the direction of the wave. Thus, the degree of the captured movement and / or posture can correspond to the amount or magnitude of the resulting vehicle action. In some embodiments, the degree of movement and / or posture captured by the gesture recognition engine 144 can vary depending on the type of action. For example, the degree (e.g., the degree or amount of movement) of a gesture used for turning a vehicle can differ from the degree or amount of movement of a gesture used to stop or slow down a vehicle.

[0037] In some embodiments, the degree to which passenger movement is interpreted as vehicle control gestures, particularly the degree to which detected gestures are interpreted to determine anticipated vehicle actions or commands, may vary from passenger to passenger and may be stored in passenger profiles or historical data, reflected in training data, etc. For example, the degree of gesture may vary with an individual's body size. Furthermore, the degree of gesture may vary from person to person, as some people may make more expressive or exaggerated gestures, while others may make smaller gestures.

[0038] In some embodiments, the amount of vehicle redirection or speed change corresponding to a passenger gesture may be linear or non-linear depending on the degree of the captured movement. For example, a 5-degree movement of a finger, hand, or arm may result in a single-lane change, a 15-degree movement may correspond to a two-lane change (i.e., because this is a more drastic maneuver), and a 60-degree movement may correspond to a three-lane change (i.e., even more drastic).

[0039] In some embodiments, to determine the corresponding command vehicle redirection or speed change, the captured movement may be limited to a defined extent (i.e., not interpreted as extending beyond a maximum extent, distance, or speed). In some embodiments, this limitation or maximum value is always imposed on the interpretation in certain situations, or only for certain types of gestures. For example, while 5-degree, 10-degree, and 15-degree movements may correspond to single-lane changes, two-lane changes, and three-lane changes, respectively, if a lane-changing gesture is limited to a 15-degree movement, the gesture recognition engine 144 may not interpret a 20-degree turn as a four-lane change.

[0040] In some embodiments that take into account the degree of captured movement and / or posture, the measured movement may be rounded (down / up) to a specific degree increment suitable for a particular gesture. For example, if a 5-degree hand movement is associated with a single-lane change and a 10-degree hand movement with a two-lane change, then a 7-degree hand movement may only be associated with a single-lane change. In some embodiments, the safer option of either a single-lane change or a two-lane change will be selected. Alternatively or additionally, the passenger may provide some additional indication of the anticipated partial lane change (i.e., thus the vehicle eventually crosses two lanes), such as indicating that a 7-degree hand movement should be interpreted as a half-lane change.

[0041] In some embodiments, passenger input other than movement can be interpreted as vehicle control gestures by the gesture recognition engine 144 as a substitute for or supplement to movement gestures. For example, the gesture recognition engine 144 can recognize (but is not limited to) voice commands, touch controls on any touch-sensitive surface (e.g., steering wheel, touchscreen, armrest, seat belt, etc.), and / or remote controller inputs, such as inputs on mobile device applications, game controllers, detachable steering wheels, or other remote controllers that can be passed to any passenger who wants to input commands.

[0042] In some embodiments, gestures or other commands can be received from outside the vehicle (e.g., via radio module 172). Therefore, gesture recognition engine 144 can recognize such remote commands for controlling the vehicle. Alternatively, a device (e.g., wireless communication device 190) can be configured with gesture recognition engine 144 to recognize gestures or commands for controlling the vehicle and then send the corresponding command to the vehicle for execution. For example, the vehicle owner (a person away from the vehicle) can provide gestures or commands to change lanes, change speed, etc. As another example, passengers can leave the vehicle via a detachable steering wheel (or other input device) to control the vehicle from outside (e.g., to provide a different view when parking in a confined space).

[0043] In some embodiments, a passenger may perform a series of vehicle control gestures to cause the vehicle to perform a series of corresponding maneuvers. In some embodiments, the gesture recognition engine 144 may indicate (e.g., with tone of voice, light, vibration, etc.) that a gesture has been recognized, and the passenger may wait for a gesture to be acknowledged or performed before performing the next gesture. In some embodiments, the gesture recognition engine 144 or the control unit 140 may cause the vehicle to perform the series of maneuvers in the order in which they are presented. Alternatively, the gesture recognition engine 144 or the control unit 140 may cause the vehicle to perform each maneuver when it becomes safe or after receiving further passenger input (e.g., after a prompt). Additionally, or as a further alternative, if the execution of a vehicle action becomes unsafe before the vehicle action is performed, the gesture recognition engine 144 may prompt the user before performing a vehicle action such as one of a series of actions. For example, if a passenger performs a series of vehicle control gestures designed to decelerate and then turn, the vehicle control unit 140 may not perform the turn if the vehicle is prevented from turning after deceleration.

[0044] refer to Figure 1A and Figure 1B Specifically, the first passenger 11 is shown performing a first vehicle control gesture 61 and the second passenger 12 is shown performing a second vehicle control gesture 62. The first vehicle control gesture 61 involves the first passenger 11 moving from left to right in the air following an arched movement angle or waving an open hand. In this scenario, sensor 101 detects a 30° (30-degree) wave from left to right (i.e., a 30° wave to the right). The second vehicle control gesture 62 involves the second passenger 12 moving in the air from left to right following an arc-shaped movement angle or waving their hand with a pointing finger. Compared to the first vehicle control gesture 61, which uses an open hand and extends only across a 30-degree angle, the second vehicle control gesture 62 uses a partially closed hand with a pointing finger extending across a 90-degree angle. However, after normalizing each of the first and second vehicle control gestures 61 and 62 using the first passenger profile A and the second passenger profile B, the gesture recognition engine 144 can determine that passengers 11 and 12 both wish for the vehicle to change lanes to the right. Therefore, in both instances, the gesture recognition engine 144 can output the same vehicle action to the control unit 140 for a lane change; a change to the right lane.

[0045] The gesture recognition engine 144 can output many different types and degrees of vehicle actions to the control unit 140 to operate the vehicle 100 to perform vehicle actions associated with the recognized gestures 61, 62.

[0046] Various embodiments can be implemented in various vehicles, in Figure 2A and Figure 2B The example vehicle 100 is shown in the image. (Reference) Figure 2A and Figure 2B The vehicle 100 may include a control unit 140 and multiple sensors. Figure 2A and Figure 2B The diagram separately describes multiple sensors in relation to the general description in Figure 1. Specifically, the multiple sensors may include occupancy sensors 102, 104, 106, 108, 110 (e.g., motion and / or proximity sensors), cameras 112, 114, microphones 116, 118, impact sensor 120, radar 122, lidar 124, satellite geolocation system receiver 126, and tire pressure sensor 128. The multiple sensors 102 to 128 disposed in or on the vehicle can be used for various purposes, such as vehicle gesture detection and / or autonomous and semi-autonomous navigation and control, collision avoidance, location determination, vehicle control gesture detection, etc., and providing sensor data about objects and people in or near the vehicle 100. Sensors 102 to 128 may include one or more of a variety of sensors capable of detecting various information for receiving inputs about the internal and external environment of the vehicle 100, as well as navigation control and collision avoidance. Each of the sensors 102 to 128 can communicate wirelessly with the control unit 140 or with each other. Specifically, sensors 102 to 128 may include one or more cameras 112, 114, which include or consist of other optical, photoelectric, and / or motion sensors. Sensors 102 to 128 may also include other types of object detection and ranging sensors, such as radar 122, lidar 124, IR sensors, and ultrasonic sensors. Sensors 102 to 128 may also include tire pressure sensors 128, humidity sensors, temperature sensors, satellite geolocation system receivers 126, accelerometers, vibration sensors, gyroscopes, gravimeters, impact sensors 120, force gauges, stress gauges, strain sensors, microphones 116, 118, occupancy sensors 102, 104, 106, 108, 110, which may include motion and / or proximity sensors, as well as various environmental sensors.

[0047] According to various embodiments, the vehicle control unit 140 can be configured to operate the vehicle 100 based on vehicle actions interpreted from vehicle control gestures and passenger profiles of one or more passengers 11, 12, 13. Additionally, the control unit 140 can have default settings for one or more passenger profiles. For example, based on the currently loaded passenger profile, the default settings may allow the vehicle to operate more or less smoothly, efficiently, quickly, slowly, etc. For example, when the control unit 140 does not recognize a passenger (i.e., no passenger profile matches), the default settings can be followed. Alternatively, when the control unit 140 does not recognize a passenger, a new profile can be created for the unknown person, for example, based on the person's actions (e.g., gestures / postures).

[0048] The vehicle control unit 140 may be configured with processor-executable instructions to perform various embodiments using information received from various sensors, particularly cameras 112, 114, microphones 116, 118, and occupancy sensors 102, 104, 106, 108, 110. In some embodiments, the control unit 140 may supplement the processing of camera images with distance and relative position (e.g., relative azimuth) obtainable from radar 122, lidar 124, and / or other sensors. The control unit 140 may also be configured to control the direction, braking, speed, acceleration / deceleration, etc., of the vehicle 100 when operating in autonomous or semi-autonomous mode using information about other vehicles and / or vehicle control gestures from passengers determined using various embodiments.

[0049] Figure 3 This is a component block diagram illustrating system 300 with components and supporting systems suitable for implementing various embodiments. (See reference...) Figures 1A to 3 The vehicle 100 may include a control unit 140, which may include various circuits and devices for controlling the operation of the vehicle 100. Figure 3 In the example shown, control unit 140 includes processor 164, memory 166, input module 168, output module 170, and radio module 172. Control unit 140 may be coupled to driving control component 154, navigation component 156, and one or more sensors 101 of vehicle 100 and is configured to control driving control component 154, navigation component 156, and one or more sensors 101 of vehicle 100.

[0050] As used herein, the terms “component,” “system,” “unit,” “module,” etc., include computer-related entities such as, but not limited to, hardware, firmware, combinations of hardware and software, software, or software in execution configured to perform a particular operation or function. For example, a component can be, but is not limited to, a process executing on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or a computer. By way of description, both an application running on a communication device and the communication device itself can be referred to as a component. One or more components may reside within a process and / or a thread of execution, and components may reside on a processor or core and / or be distributed across two or more processors or cores. Furthermore, these components may execute from various non-transitory computer-readable media having various instructions and / or data structures stored thereon. Components may communicate via local and / or remote procedures, function or procedure calls, electronic signals, data packets, memory read / write, and other known computer, processor, and / or process-related communication methods.

[0051] Control unit 140 may include processor 164, which may be configured with processor-executable instructions to determine vehicle control gestures and / or alternatives, and to control the maneuvering, navigation, and / or other operations of vehicle 100, including operations according to various embodiments. Processor 164 may be coupled to memory 166. Control unit 162 may include input module 168, output module 170, and radio module 172.

[0052] Radio module 172 can be configured for wireless communication. Radio module 172 can exchange signals 182 (e.g., command signals for control maneuvers, signals from navigation facilities, etc.) with communication network 180. Radio module 172 can provide signals 182 to processor 164 and / or navigation component 156. In some embodiments, radio module 172 can enable vehicle 100 to communicate with wireless communication device 190 via wireless communication link 187. Wireless communication link 187 can be a bidirectional or unidirectional communication link. Wireless communication link 187 can use one or more communication protocols. In some embodiments, radio module 172 can enable vehicle 100 to communicate with another vehicle via wireless communication link 192. Wireless communication link 192 can be a bidirectional or unidirectional communication link, and wireless communication link 192 can use one or more communication protocols.

[0053] The input module 168 can receive sensor data from one or more vehicle sensors 101, as well as electronic signals from other components including the driving control component 154 and the navigation component 156. The output module 170 can be used to communicate with or activate the various components of the vehicle 100, including the driving control component 154, the navigation component 156, and the sensors 101.

[0054] Control unit 140 may be coupled to driving control component 154 to control physical elements of the vehicle related to the handling and navigation of vehicle 100, such as (but not limited to) engine, motor, throttle, steering elements, flight control elements, braking or deceleration elements, etc. Driving control component 154 may also, or alternatively, include components for controlling other devices of the vehicle, including (but not limited to) environmental controls (e.g., air conditioning and heating), exterior and / or interior lighting, interior and / or exterior information displays (which may include displays or other devices for displaying information), safety devices (e.g., haptic devices, auditory alarms, etc.), and other similar devices.

[0055] Control unit 140 may be coupled to navigation component 156 and may receive data from navigation component 156 and be configured to use this data to determine the current position and orientation of vehicle 100. In various embodiments, navigation component 156 may include or be coupled to a Global Navigation Satellite System (GNSS) receiver system (e.g., one or more Global Positioning System (GPS) receivers) enabling vehicle 100 to use GNSS signals to determine its current position. Alternatively or additionally, navigation component 156 may include a radio navigation receiver for receiving navigation beacons or other signals from radio nodes such as Wi-Fi access points, cellular network sites, radio stations, telecomputing devices, other vehicles, etc. Through control of driving control component 154, processor 164 may control vehicle 100 for navigation and maneuvering. Processor 164 and / or navigation component 156 may be configured to communicate with server 184 on network 186 (e.g., the Internet) using wireless connection signal 182 with cellular data communication network 180 to receive commands for control maneuvering, receive data useful during navigation, provide real-time position reports, and evaluate other data. The control unit 140 may be coupled to one or more sensors 101, which may be configured to provide various data to the processor 164.

[0056] Although the control unit 140 is described as comprising separate components, in some embodiments, some or all of the components (e.g., processor 164, memory 166, input module 168, output module 170, and radio module 172) may be integrated into a single device or module, such as a system-on-a-chip (SOC) processing device. Such an SOC processing device may be configured for use in a vehicle and configured to, for example, have processor-executable instructions that execute in processor 164 to perform operations according to various embodiments when installed in a vehicle.

[0057] Figure 4A Examples of subsystems, computing elements, computing devices, or units within a vehicle management system 400 that can be utilized within a vehicle (e.g., 100) are shown. Reference Figures 1A to 4A In some embodiments, various computing elements, computing devices, or units within the vehicle management system 400 may be implemented within a system of interconnected computing devices (i.e., subsystems) that transmit data and commands to each other (e.g., by...). Figure 4A (The arrows in the diagram indicate this). In other embodiments, various computing elements, computing devices, or units within the vehicle management system 400 may be implemented within a single computing device (such as a separate thread, process, algorithm, or computing element). Therefore, Figure 4AEach subsystem / computing element shown herein is also generally referred to herein as a "layer" within the computing "stack" constituting the vehicle management system 400. However, the use of the terms layer and stack in describing various embodiments is not intended to imply or require the implementation of corresponding functionality within a single autonomous (or semi-autonomous) vehicle management system computing device, but rather as a potential embodiment. Rather, the use of the term "layer" is intended to encompass subsystems with independent processors, computing elements (e.g., threads, algorithms, subroutines, etc.) running in one or more computing devices, and combinations of subsystems and computing elements.

[0058] In various embodiments, the vehicle management system 400 may include a radar perception layer 402, a camera perception layer 404, a localization engine layer 406, a map fusion and arbitration layer 408, a route planning layer 410, a sensor fusion and road world model (RWM) management layer 412, a motion planning and control layer 414, and a behavior planning and prediction layer 416. Layers 402 to 416 are merely examples of some layers in one example configuration of the vehicle management system 400. In other configurations consistent with the various embodiments, one or more additional layers may be included, such as (but not limited to) additional layers for other perception sensors (e.g., camera perception layers, etc.), additional layers for planning and / or control, additional layers for modeling, etc., and / or some of the layers 402 to 416 may be excluded from the vehicle management system 400.

[0059] Each of layers 402 to 416 can exchange data, calculation results, and commands (such as...) Figure 4A (As indicated by the arrows in the diagram). Furthermore, the vehicle management system 400 can receive and process data from sensors (e.g., radar, lidar, cameras, inertial measurement units (IMUs), etc.), navigation systems (e.g., GPS receivers, IMUs, etc.), vehicle networks (e.g., Controller Area Network (CAN) bus), and databases in memory (e.g., digital map data). The vehicle management system 400 can output vehicle control commands or signals to the drive-by-wire (DBW) system / control unit 420, which is a system, subsystem, or computing device that directly interfaces with vehicle steering, throttle, and braking controls. Figure 4A The configurations of the vehicle management system 400 and DBW system / control unit 420 shown are merely example configurations, and other configurations of the vehicle management system and other vehicle components can be used in various embodiments. As an example, Figure 4A The configurations of the vehicle management system 400 and DBW system / control unit 420 shown can be used in vehicles configured for autonomous or semi-autonomous operation, while different configurations can be used in non-autonomous vehicles.

[0060] The radar perception layer 402 may receive data from one or more detection and ranging sensors, such as radar (e.g., 122) and / or lidar (e.g., 124). The radar perception layer 402 may process the data to identify and determine the location of other vehicles and objects near vehicle 100. In some embodiments, the radar perception layer 402 may include the use of neural network processing and artificial intelligence methods to identify objects and vehicles. The radar perception layer 402 may pass such information to the sensor fusion and RWM management layer 412.

[0061] The camera perception layer 404 may receive data from one or more cameras (such as cameras (e.g., 112, 114)) and process that data to detect vehicle control gestures, as well as identify and determine the location of other vehicles and objects near vehicle 100. In some embodiments, the camera perception layer 404 may include using neural network processing and artificial intelligence methods to identify objects and vehicles and pass such information to the sensor fusion and RWM management layer 412.

[0062] The positioning engine layer 406 can receive and process data from various sensors to determine the position of the vehicle 100. These sensors may include, but are not limited to, GPS sensors, IMUs, and / or other sensors connected via a CAN bus. The positioning engine layer 406 can also utilize input from one or more cameras (such as cameras (e.g., 112, 114)) and / or any other available sensors (such as radar (e.g., 122), lidar (e.g., 124)).

[0063] The map fusion and arbitration layer 408 can access data within the high-definition (HD) map database and receive output from the positioning engine layer 406, processing the data to further determine the vehicle's position within the map, such as its position within a traffic lane, its position within a street map, etc. The HD map database can be stored in memory (e.g., memory 166). For example, the map fusion and arbitration layer 408 can convert latitude and longitude information from GPS into a position within a surface map of a road contained in the HD map database. GPS location positioning includes errors, so the map fusion and arbitration layer 408 can be used to arbitrate between GPS coordinates and HD map data to determine the best guess of the vehicle's position within the road. For example, while GPS coordinates might place the vehicle near the middle of a two-lane road in the HD map, the map fusion and arbitration layer 408 can determine, based on the direction of travel, that the vehicle is most likely aligned with the lane in the same direction of travel. The map fusion and arbitration layer 408 can then pass the map-based position information to the sensor fusion and RWM management layer 412.

[0064] Route planning layer 410 can utilize HD maps and input from operators or dispatchers to plan routes for vehicles to follow to specific destinations. Route planning layer 410 can pass map-based location information to sensor fusion and RWM management layer 412. However, the use of prior maps by other layers (such as sensor fusion and RWM management layer 412) is not required. For example, other stacks can operate and / or control the vehicle based solely on perception data, without requiring provided maps, and construct concepts of lanes, boundaries, and local maps upon receiving perception data.

[0065] The sensor fusion and RWM management layer 412 can receive data and outputs generated by the radar perception layer 402, camera perception layer 404, map fusion and arbitration layer 408, and route planning layer 410, and use some or all of these inputs to estimate or refine the vehicle's position and state relative to the road, other vehicles on the road, and other objects near the vehicle. For example, the sensor fusion and RWM management layer 412 can combine image data from the camera perception layer 404 with arbitration map position information from the map fusion and arbitration layer 408 to refine the vehicle's determined position within the traffic lane.

[0066] As another example, the sensor fusion and RWM management layer 412 can combine object recognition and image data from the camera perception layer 404 with object detection and ranging data from the radar perception layer 402 to determine and refine the relative positions of other vehicles and objects near the vehicle. As another example, the sensor fusion and RWM management layer 412 can receive information about the position and direction of travel of other vehicles from vehicle-to-vehicle (V2V) communication (such as via a CAN bus) and combine this information with information from the radar perception layer 402 and the camera perception layer 404 to refine the position and motion of other vehicles. The sensor fusion and RWM management layer 412 can output refined position and status information of the vehicle, as well as refined position and status information of other vehicles and objects near the vehicle, to the motion planning and control layer 414 and / or the behavior planning and prediction layer 416.

[0067] As another example, the sensor fusion and RWM management layer 412 can use dynamic traffic control commands that guide the vehicle to change speed, lane, direction of travel, or other navigation elements, and combine this information with other received information to determine refined location and status information. The sensor fusion and RWM management layer 412 can output the refined location and status information of the vehicle, as well as the refined location and status information of other vehicles and objects near the vehicle, to the motion planning and control layer 414, the behavior planning and prediction layer 416, and / or remote devices such as data servers, other vehicles, etc., via wireless communication (such as via C-V2X connections, other wireless connections, etc.).

[0068] As another example, the sensor fusion and RWM management layer 412 can monitor perception data from various sensors, such as those from the radar perception layer 402, camera perception layer 404, other perception layers, etc., and / or information from one or more sensors themselves, to analyze the conditions in the vehicle sensor data. The sensor fusion and RWM management layer 412 can be configured to detect conditions in the sensor data, such as sensor measurements being equal to, above, or below thresholds, or the occurrence of certain types of sensor measurements, and can output the sensor data as part of refined location and status information of the vehicle provided via wireless communication (such as via C-V2X connections, other wireless connections, etc.) to the behavior planning and prediction layer 416 and / or to devices remote to the vehicle (such as data servers, other vehicles, etc.).

[0069] Detailed location and status information may include vehicle descriptors associated with the vehicle and its owner and / or operator, such as: vehicle specifications (e.g., size, weight, color, type of onboard sensors, etc.); vehicle location, speed, acceleration, direction of travel, attitude, orientation, destination, fuel / power level, and other status information; vehicle emergency status (e.g., whether the vehicle is an emergency vehicle or a private vehicle); vehicle restrictions (e.g., heavy / wide load, turning restrictions, high-occupancy vehicle (HOV) authorization, etc.); vehicle capabilities (e.g., all-wheel drive, four-wheel drive, snow tires, chain, supported connection types, onboard sensor operating status, onboard sensor resolution level, etc.); equipment issues (e.g., low tire pressure, weak braking, sensor malfunction, etc.); owner / operator driving preferences (e.g., preferred lanes, roads, routes, and / or destinations, preference for avoiding tolls or highways, preference for the fastest route, etc.); permission to provide sensor data to a data broker server (e.g., 184); and / or owner / operator identification information.

[0070] The behavior planning and prediction layer 416 of the autonomous vehicle management system 400 can use detailed position and state information of the vehicle, as well as position and state information of other vehicles and objects output from the sensor fusion and RWM management layer 412, to predict the future behavior of other vehicles and / or objects. For example, the behavior planning and prediction layer 416 can use such information to predict the future relative positions of other vehicles near the vehicle based on its own vehicle position and speed, as well as the positions and speeds of other vehicles. This prediction can take into account information from HD maps and route planning to predict changes in relative vehicle positions as the primary vehicle and other vehicles follow the road. The behavior planning and prediction layer 416 can output the predicted behavior and positions of other vehicles and objects to the motion planning and control layer 414.

[0071] Additionally, the behavior planning and prediction layer 416 can combine position prediction with object behavior to plan and generate control signals for controlling vehicle motion. For example, based on route planning information, refined position data in road information, and the relative positions and movements of other vehicles, the behavior planning and prediction layer 416 can determine that the vehicle needs to change lanes and accelerate, such as maintaining or achieving a minimum distance from other vehicles, and / or preparing to turn or leave. As a result, the behavior planning and prediction layer 416 can calculate or otherwise determine the wheel steering angle and the throttle setting changes to be commanded to the motion planning and control layer 414 and the DBW system / control unit 420, as well as various parameters required to achieve such lane changes and acceleration. One such parameter could be the calculated steering wheel command angle.

[0072] The motion planning and control layer 414 can receive data and information output from the sensor fusion and RWM management layer 412, as well as other vehicle and object behaviors and position predictions from the behavior planning and prediction layer 416. The motion planning and control layer 414 can use at least some of this information to plan and generate control signals for controlling vehicle motion, and verify whether such control signals meet the vehicle's safety requirements. For example, based on route planning information, refined location data in road information, and the relative positions and movements of other vehicles, the motion planning and control layer 414 can verify various control commands or instructions and transmit them to the DBW system / control unit 420.

[0073] The DBW system / control unit 420 can receive commands or instructions from the motion planning and control layer 414 and translate this information into mechanical control signals for controlling the vehicle's wheel angles, brakes, and throttle valves. For example, the DBW system / control unit 420 can respond to the calculated steering wheel command angle by sending a corresponding control signal to the steering wheel controller.

[0074] In various embodiments, the vehicle management system 400 may include functionality to perform safety checks or oversight of various commands, plans, or other decisions at various layers that may affect vehicle and occupant safety. Such safety check or oversight functionality may be implemented within a dedicated layer or distributed across layers and included as part of the functionality.

[0075] In some embodiments, various safety parameters may be stored in memory, and safety check or monitoring functions may compare determined values ​​(e.g., relative distance to nearby vehicles, distance from the road centerline, etc.) with corresponding safety parameters and issue warnings or commands if safety parameters are violated or will be violated. For example, the safety or monitoring function in the behavior planning and prediction layer 416 (or a separate layer) may determine the current or future distance between the vehicle and another vehicle (defined by the sensor fusion and RWM management layer 412) based on a world model refined by the sensor fusion and RWM management layer 412, compare this distance with a safe distance parameter stored in memory, and issue an acceleration, deceleration, or turning command to the motion planning and control layer 414 if the current or predicted distance violates the safe distance parameter. As another example, the safety or monitoring function in the motion planning and control layer 414 (or a separate layer) may compare a determined or commanded steering wheel command angle with a safe wheel angle limit or parameter and issue an override command and / or alarm in response to the commanded angle exceeding the safe wheel angle limit.

[0076] According to various implementations, some safety parameters stored in memory can be static (i.e., do not change over time), such as maximum vehicle speed. For example, operating a vehicle at speeds exceeding 120 mph (or some other value that may be set by the manufacturer, owner, etc.) may be considered dangerous, so the vehicle (e.g., gesture recognition engine 144 and / or control unit 140) will apply a speed-related maximum safety parameter. For autonomous vehicles, operating at speeds exceeding 120 mph may still be safe, so higher values ​​can be used. Alternatively, some safety parameters (e.g., maximum speed) can be dynamic and vary depending on location, driving conditions (e.g., traffic volume), and / or external inputs (e.g., signals from intelligent transportation systems). For example, the maximum speed may have one value for driving in densely populated areas or on more restricted roads (i.e., city driving), and another value for driving in less populated areas or on roads with many lanes (i.e., highway driving). Other safety parameters, such as proximity to objects (e.g., other vehicles), people, living beings, or other elements, can be used. Additionally, other safety parameters stored in memory can be dynamic, as they are continuously or periodically determined or updated based on vehicle status information and / or environmental conditions. Non-limiting examples of safety parameters may include maximum speed, minimum speed, maximum deceleration (e.g., braking speed), maximum acceleration, and maximum wheel angle limits. Any or all of these may be a function of occupant, road, and / or weather conditions. For example, if an occupant is sleeping, intoxicated, or distracted (e.g., reading, not facing forward / looking ahead, etc.), they are less likely to give appropriate commands and / or properly navigate the vehicle compared to other times when, for example, the occupant is alert and not distracted.

[0077] Figure 4B Examples of subsystems, computing elements, computing devices, or units within a vehicle management system 450 that can be utilized within a vehicle (e.g., 100) are shown. Reference Figures 1A to 4B In some embodiments, layers 402, 404, 406, 408, 410, 412, and 416 of the vehicle management system 400 may be similar to those referenced in the reference document. Figure 4A The layers described. The vehicle management system 450 can operate similarly to the vehicle management system 400, except that the vehicle management system 450 can transmit various data or instructions to the vehicle safety and collision avoidance system 452 instead of the DBW system / control unit 420. For example, Figure 4B The configuration of the vehicle management system 450 and the vehicle safety and collision avoidance system 452 shown can be used in non-autonomous, semi-autonomous, or fully autonomous vehicles. Additionally, the functionality of the vehicle management system 450 and / or the vehicle safety and collision avoidance system 452 can be reduced or disabled (e.g., turned off).

[0078] In various embodiments, the behavior planning and prediction layer 416 and / or the sensor fusion and RWM management layer 412 can output data to the vehicle safety and collision avoidance system 452. For example, the sensor fusion and RWM management layer 412 can output sensor data as part of refined position and state information of vehicle 100 provided to the vehicle safety and collision avoidance system 452. The vehicle safety and collision avoidance system 452 can use the refined position and state information of vehicle 100 to make safety determinations regarding vehicle 100 and / or its occupants. As another example, the behavior planning and prediction layer 416 can output behavior models and / or predictions related to the motion of other vehicles to the vehicle safety and collision avoidance system 452. The vehicle safety and collision avoidance system 452 can use the behavior models and / or predictions related to the motion of other vehicles to make safety determinations regarding vehicle 100 and / or its occupants (e.g., 11, 12, 13).

[0079] In various embodiments, the vehicle safety and collision avoidance system 452 may include functions for performing various commands, planning, or other decisions on various layers, as well as safety checks or supervision of human driver actions and / or vehicle control gestures that may affect the safety of the vehicle and its occupants. In some embodiments, various safety parameters may be stored in memory, and the vehicle safety and collision avoidance system 452 may compare determined values ​​(e.g., relative distance to nearby vehicles, distance from the center line of the road, etc.) with corresponding safety parameters, and issue warnings, commands, or safe alternative vehicle actions if safety parameters are violated or are about to be violated.

[0080] For example, the vehicle safety and collision avoidance system 452 can (e.g., based on a road world model refined by sensor fusion and RWM management layer 412) determine the current or future distance between itself and another vehicle (defined by sensor fusion and RWM management layer 412), compare this distance with a safe distance parameter stored in memory, and if the current or predicted distance violates the safe distance parameter, issue instructions to the driver and / or propose safe alternative vehicle actions to accelerate, decelerate, or turn. As another example, the vehicle safety and collision avoidance system 452 can compare a determined first vehicle action (e.g., by applying a first passenger profile to a detected first vehicle control gesture performed by the first passenger) with safe vehicle action limits or parameters, and issue an override command and / or warning in response to a proposed vehicle action exceeding the safe vehicle action limits or parameters.

[0081] Figure 5 An example system-on-a-chip (SoC) architecture suitable for implementing various embodiments of the processing device SOC 500 in a vehicle is shown. References Figures 1A to 5 The processing device SOC 500 may include several heterogeneous processors, such as a digital signal processor (DSP) 503, a modem processor 504, an image and object recognition processor 506, a mobile display processor 507, an application processor 508, and a resource and power management (RPM) processor 517. The processing device SOC 500 may also include one or more coprocessors 510 (e.g., vector coprocessors) connected to one or more of the heterogeneous processors 503, 504, 506, 507, 508, and 517. Each of these processors may include one or more cores and an independent / internal clock. Each processor / core may perform operations independently of the other processors / cores. For example, the processing device SOC 500 may include a processor running a first type of operating system (e.g., FreeBSD, LINUX, OS X, etc.) and a processor running a second type of operating system (e.g., Microsoft Windows). In some embodiments, the application processor 508 may be the main processor of the SOC 500, a central processing unit (CPU), a microprocessor unit (MPU), an arithmetic logic unit (ALU), a graphics processing unit (GPU), etc.

[0082] The processing device SOC 500 may include analog circuitry and custom circuitry 514 for managing sensor data, analog-to-digital conversion, wireless data transmission, and performing other specialized operations, such as processing encoded audio and video signals for rendering in a web browser. The processing device SOC 500 may also include system components and resources 516, such as voltage regulators, oscillators, phase-locked loops, peripheral bridges, data controllers, memory controllers, system controllers, access ports, timers, and other similar components for supporting processors and software clients (e.g., web browsers) running on computing devices.

[0083] The processing device SOC 500 also includes dedicated circuitry for Camera Actuation and Management (CAM) 505, which includes, provides, controls, and / or manages the operation of one or more cameras (e.g., 101, 112, 114; main camera, webcam, 3D camera, etc.), video display data from camera firmware, image processing, video preprocessing, video front-end (VFE), in-line JPEG, high-definition video encoder, etc. CAM 505 may be a separate processing unit and / or include an independent clock or an internal clock.

[0084] In some embodiments, the image and object recognition processor 506 may be configured with processor-executable instructions and / or be configured with dedicated hardware to perform the image processing and object recognition analysis involved in the various embodiments. For example, the image and object recognition processor 506 may be configured to perform operations such as processing images received from a camera (e.g., 136) via CAM 505 to identify and / or recognize vehicle control gestures, other vehicles, and otherwise perform the functions of the camera perception layer 404 as shown in the figure. In some embodiments, the processor 506 may be configured to process radar or lidar data and perform the functions of the radar perception layer 402 as shown in the figure.

[0085] System components and resources 516, analog and custom circuit systems 514, and / or CAM 505 may include circuitry for interfacing with peripheral devices such as camera 136, radar 122, lidar 124, electronic displays, wireless communication devices, external memory chips, etc. Processors 503, 504, 506, 507, and 508 may be interconnected to one or more memory elements 512, system components and resources 516, analog and custom circuit systems 514, CAM 505, and RPM processor 517 via interconnect / bus module 524, which may include an array of reconfigurable logic gates and / or implement a bus architecture (e.g., CoreConnect, AMBA, etc.). Communication may be provided via advanced interconnects such as high-performance networks-on-chip (NoC).

[0086] The processing device SOC 500 may also include input / output modules (not shown) for communicating with external resources (such as clock 518 and voltage regulator 520). External resources (e.g., clock 518, voltage regulator 520) may be shared by two or more of the internal SOC processors / cores (e.g., DSP 503, modem processor 504, image and object recognition processor 506, MDP, application processor 508, etc.).

[0087] In some embodiments, the processing device SOC 500 may be included in a control unit (e.g., 140) for use in a vehicle (e.g., 100). As described, the control unit may include a communication link for communicating with a communication network (e.g., 180), the Internet, and / or a web server (e.g., 184).

[0088] The processing device SOC 500 may also include additional hardware and / or software components suitable for collecting sensor data from sensors including: motion sensors (e.g., accelerometers and gyroscopes of an IMU), user interface elements (e.g., input buttons, touchscreen displays, etc.), microphone arrays, sensors for monitoring physical conditions (e.g., position, orientation, motion, orientation, vibration, pressure, etc.), cameras, compasses, GPS receivers, and communication circuitry (e.g., ...). WLAN, WiFi, and other components of well-known modern electronic devices.

[0089] Figure 6 A component block diagram is shown illustrating a system 600 configured, according to various embodiments, to collaboratively operate a vehicle (e.g., 100) based on passenger vehicle control gestures. In some embodiments, system 600 may include one or more vehicle computing systems 602 and one or more other vehicle computing systems 604 communicating via a wireless network. References Figures 1A to 6 Vehicle computing system 602 may include a processor (e.g., 164), processing device (e.g., 500), and / or control unit (e.g., 140) of a vehicle (e.g., 100) (referred to as a "processor" in different contexts). Other vehicle computing systems 604 may include a processor (e.g., 164), processing device (e.g., 500), and / or control unit (e.g., 140) of a vehicle (e.g., 100) (referred to as a "processor" in different contexts).

[0090] The vehicle computing system 602 can be configured by machine-executable instructions 606. Machine-executable instructions 606 may include one or more instruction modules. Instruction modules may include computer program modules. Instruction modules may include (but are not limited to) one or more of the following: a passenger identification module 607, a passenger profile determination module 608, a vehicle control gesture determination module 610, a vehicle action determination module 612, a passenger profile receiving module 616, a vehicle action safety determination module 618, an alternative vehicle action determination module 620, a delay period assessment module 622, an unusual operation determination module 624, an additional instruction determination module 626, an unusual operation safety assessment module 628, a vehicle operation module 629, and / or other instruction modules.

[0091] Passenger identification module 607 can be configured to identify passengers (i.e., one or more occupants of the vehicle). In some embodiments, passenger identification module 607 can be configured to identify passengers based on at least one of (but not limited to) the passenger's location in the vehicle, passenger input, or passenger identification. For example, passenger input and / or passenger identification can be determined based on the occupant's use of a portable computing device (e.g., a smartphone) or based on identification using sensors (e.g., 101) (such as (but not limited to) the use of biometric sensors and / or facial recognition systems). As a non-limiting example, the processor (e.g., 164) of a processing device (e.g., 500) can use electronic storage device 635, other vehicle computing system 604, external resources 630, one or more sensors (e.g., 101), and profile database (e.g., 142) to identify passengers or the location of passenger seating.

[0092] The passenger profile determination module 608 can be configured to determine one or more profiles that should be applied to identified gestures performed by a passenger. Determining an appropriate passenger profile will more effectively normalize passenger gestures to translate them into executable vehicle actions. As a non-limiting example, the processor (e.g., 164) of a processing device (e.g., 500) can use electronic storage device 635, a profile database (e.g., 142), and passenger identification information to determine which passenger profile to use based on the current situation. The profile database and / or passenger profile determination module 608 can maintain multiple passenger profiles, including passenger profiles for specific individuals. Some or all of the multiple passenger profiles can normalize vehicle control gestures in different ways because an individual making a given control gesture may move their fingers, hands, and / or arms at different speeds, through different angles, and through differences in motion. Therefore, a second passenger profile can normalize vehicle control gestures differently from a first passenger profile. Some implementations may not employ the passenger profile determination module 608, but instead apply predetermined passenger profiles more directly. Alternatively, the passenger profile determination module 608 can be selectively turned on or off as needed or desired, which can be determined manually or automatically based on the environment / conditions.

[0093] According to various embodiments, in cases where more than one passenger occupies the vehicle, the passenger profile determination module 608 may also be configured to determine that one or more passengers are considered to be designated as the driver, and therefore only receive vehicle control gestures from the designated driver. In this way, the passenger profile determination module 608 can ignore gestures from one or more passengers who are not designed to be the driver. In other embodiments, if more than one driver is designated as the driver (e.g., in student driver situations or driver education applications), the passenger profile determination module 608 may have a manner that determines the hierarchy among conflicting vehicle control gestures. Using sensors (e.g., 101), the passenger profile determination module 608 can determine that the vehicle has multiple occupants and also determine who or which occupants(s) are responsible for control (i.e., the designated driver). In some embodiments, the passenger occupying the conventional driver's seat (or other preset position in the vehicle) may be defaulted to being the designated driver unless overridden. Alternatively, both front seat occupants may be designated drivers (because they tend to have better road visibility). In a particular embodiment, in such a case, vehicle control gestures from the driver's seat passenger may overridden vehicle control gestures from other front seat passengers.

[0094] In some embodiments, a designated driver can be selected following input from an occupant (e.g., input from a relevant mobile device or direct input to the vehicle). In some embodiments, input regarding the designated driver can also be received automatically from the occupant, such as via the passenger identification module 607. In some embodiments, when a designated driver is determined by identification, the passenger profile determination module 608 can automatically apply a hierarchy. For example, the vehicle owner (the most frequently or most recently designated driver) can have the highest priority as the designated driver. Similarly, the hierarchy list can be programmed or defined by the user (e.g., father > mother > eldest child, or father and mother > eldest child). In other embodiments, a hierarchy may not exist, where vehicle control gestures are received from all or some of the occupants.

[0095] In some implementations where a designated driver or level of authority exists among occupants for receiving commands, undesignated drivers or lower-level occupants may be allowed to input selected vehicle control gestures, such as, but not limited to, non-navigation commands and / or commands that do not jeopardize vehicle or occupant safety (e.g., controlling cabin temperature, entertainment system volume). Alternatively, vehicle control gestures from undesignated drivers or lower-level occupants may be accepted, but with higher safety parameters (e.g., greater distance required between vehicles, lower maximum speed, etc.) or lower magnitudes in vehicle input (e.g., limited to single-lane changes, increasing / decreasing speed to only 5 mph increments, etc.). For example, the vehicle control gesture determination module 610 may recognize gestures from a "rear seat driver" limited to decreasing vehicle speed or increasing vehicle spacing, as such commands would be safe and would make such occupants feel safer, but may not recognize all other vehicle handling controls. Therefore, undesignated drivers or lower-level occupants may be allowed a lower degree of control compared to a designated driver or a higher-priority designated driver.

[0096] In some embodiments, restrictions on recognized gestures by non-designated drivers or lower-level occupants can be overrided in certain situations. In some embodiments, the vehicle control gesture determination module 610 can recognize overrided gestures or situations where gestures by non-designated drivers should be recognized and implemented to accommodate situations where the designated driver becomes incapacitated. For example, if the vehicle control gesture determination module 610 detects that the designated driver has fallen asleep or passed out (obviously passed out) and another passenger is making a recognized vehicle control gesture, the vehicle can implement such a gesture.

[0097] In some embodiments, the prioritization of designated drivers or the selection of a newly designated driver can occur in response to a triggering event, such as automatically. A non-limiting example of such a triggering event could be the detection, by one or more sensors and / or the vehicle control system, of behavioral changes in the currently designated driver (or the designated driver with the highest priority), such as due to fatigue, distraction, intoxication, etc. For example, a camera system tracking the eyes of a designated driver could detect drooping eyelids due to fatigue, the driver not looking at the road due to distraction, or the designated driver's eyes moving slowly due to intoxication. Another non-limiting example of such a triggering event could be the detection of commands (e.g., verbal commands or command gestures) received via a wireless communication link by the currently designated driver (or the designated driver with the highest priority), other occupants, or a remote party (e.g., the vehicle owner). Therefore, the passenger profile determination module 608 can be configured to assess and determine (e.g., automatically) whether the currently designated driver or other passengers providing vehicle control gestures are exhibiting behavioral changes that may impair that individual's ability to give appropriate or timely vehicle control gestures. The passenger profile determination module 608 can receive input from sensors (e.g., 101) such as cameras, alcohol sensors, motion detectors, etc., and apply passenger motion pattern recognition to determine the passenger's injury level. In response to the passenger profile determination module 608 detecting that a designated driver (or the designated driver with the highest priority) is injured, the passenger profile determination module 608 can be configured to select a new designated driver or change the priority of a designated driver.

[0098] In some embodiments, in response to the passenger profile determination module 608 detecting that a designated driver (or the designated driver with the highest priority) is impaired, some degree of control can still be provided to the impaired designated driver. For example, the impaired designated driver can be restricted to providing only non-navigation commands and / or commands that do not endanger the safety of the vehicle or occupants (e.g., controlling the cabin temperature, entertainment system volume). Alternatively, the impaired designated driver can be restricted to less vehicle control or a lower degree of vehicle control (e.g., limiting speed, speed changes, the number of lanes that can be changed in a single continuous maneuver, etc.). Additionally or alternatively, the impaired designated driver can be allowed to provide navigation commands and / or controls, provided that they meet a safety threshold, which may be higher than the safety threshold used for drivers not considered impaired. For example, the impaired designated driver may be allowed to make two-lane changes on an open highway, but not on a congested highway.

[0099] The vehicle control gesture determination module 610 can be configured to determine when a passenger (e.g., 11, 12, 13) performs a vehicle control gesture in a manner recognizable by the vehicle gesture detection system (e.g., 50). As a non-limiting example, the processor (e.g., 164) of the processing device (e.g., 500) can use electronic storage device 635, one or more sensors (e.g., 101), and a gesture recognition engine (e.g., 144) to determine whether one or more passengers have performed a vehicle control gesture for operating the vehicle (e.g., 100). Once the vehicle control gesture determination module 608 detects that a passenger has performed one or more vehicle control gestures, information about the one or more vehicle control gestures can be transmitted to the control unit (e.g., 140).

[0100] The vehicle motion determination module 612 can be configured to determine which vehicle motions or actions are associated with the detected vehicle control gestures. The vehicle motion determination module 612 can also be configured to determine alternative vehicle motions when the detected vehicle control gestures are associated with actions that are unsafe for the vehicle and / or passengers or unusual in some respects. As a non-limiting example, the vehicle motion determination module 612 may use a processor (e.g., 164) of a processing device (e.g., 500), an electronic storage device 635, and a vehicle management system (e.g., 400, 450) to determine vehicle motions.

[0101] Passenger profile receiving module 616 can be configured to receive and store passenger profiles. Passenger profile receiving module 616 can receive a passenger profile customized for a passenger from training input. Alternatively or concurrently, passenger profile receiving module 616 can receive passenger profiles as such input data via a vehicle user interface or from another computing device providing input data. For example, a passenger can follow a training protocol in which the passenger practices and / or performs gestures and the passenger's movements are recorded and analyzed to generate a passenger profile for that passenger. As another example, a remote computing device can provide one or more passenger profiles to passenger profile receiving module 616 for application to vehicle control gestures. As a non-limiting example, the processor (e.g., 164) of a processing device (e.g., 500) can use electronic storage device 635, one or more sensors (e.g., 101), and input devices for receiving passenger profiles.

[0102] The vehicle action safety determination module 618 can be configured to determine whether a first vehicle action associated with a detected first vehicle control gesture is safe for the operation of the vehicle. Determining safety ensures that no damage or injury will be caused to the vehicle or passengers. Generally, what is safe for the vehicle is also safe for passengers, and vice versa (i.e., the vehicle safety risk level is equal to or close to the passenger safety risk level), but this may not always be the case. For example, extremely rapid deceleration may cause a passenger to twist their ankle, while the vehicle may not be damaged. Therefore, safety usually takes precedence over passenger safety. As a non-limiting example, the processor (e.g., 164) of a processing device (e.g., 500) may use electronic storage device 635 and a vehicle management system (e.g., 400, 450) to determine or evaluate the safety of various vehicle actions.

[0103] In some implementations, the alternative vehicle action determination module 620 may be configured to determine one or more alternative vehicle actions (e.g., changing speed, changing lanes, etc.), which may be a safer alternative to the first vehicle action associated with the received vehicle control gesture. The alternative vehicle action determination module 620 may be used in response to determining that the first vehicle action is unsafe for vehicle operation. In another embodiment, the probability of the determined alternative vehicle action causing harm to the vehicle and / or passengers may be at least a certain threshold amount lower than that of the first vehicle action. For example, the first vehicle action may involve changing lanes so that the vehicle will be behind another vehicle. This first vehicle action may be relatively safe, but there may be a statistical chance (i.e., a relatively small chance) that the leading vehicle may slam on its brakes shortly after the lane change (i.e., a first safety risk level). Meanwhile, the alternative vehicle action may include overtaking the leading vehicle before the lane change, which may be associated with a lower statistical chance that the vehicle becomes involved in an accident due to the open road ahead of the leading vehicle (i.e., a second safety risk level, which is lower than the first safety risk level). As a non-limiting example, the processor (e.g., 164) of the processing device (e.g., 500) may use electronic storage device 635 and vehicle management system (e.g., 400, 450) to determine alternative vehicle actions.

[0104] The delay period assessment module 622 can be configured to determine whether a reasonable delay period would make an otherwise unsafe vehicle maneuver safe enough for the vehicle to operate. For example, a first vehicle control gesture might instruct the vehicle to make an unsafe lane change due to another vehicle traveling very close in the lane. The delay period assessment module 622 can determine that, because the other vehicle is traveling at a different speed, a delay of up to five (5) seconds before initiating the lane change could change the previously unsafe maneuver into a safe one. In other embodiments, if the maneuver is still unsafe, at the end of the delay period, the delay period assessment module 622 can reassess whether an additional delay period would make an otherwise unsafe vehicle maneuver safe enough for the vehicle to operate. Alternatively, the vehicle can notify the user (driver) to determine that the maneuver should not be performed and / or request further user input.

[0105] In various embodiments, a maximum delay threshold (such as (but not limited to) 5 to 10 seconds) can be used to limit the duration of delay periods that the delay period evaluation module 622 can consider. The maximum delay threshold can be set and / or changed by the passenger, vehicle owner, and / or manufacturer. Additionally, the maximum delay threshold may be different for each passenger (i.e., associated with a passenger profile). Alternatively, the maximum delay threshold may be common to all passengers. As another alternative, while individual passengers may have different maximum delay thresholds, the vehicle may also have a final maximum delay threshold that individual maximum delay thresholds cannot exceed. As a non-limiting example, the processor (e.g., 164) of the processing device (e.g., 500) may use electronic storage device 635 and vehicle management system (e.g., 400, 450) to determine or evaluate delay periods used to perform various vehicle actions.

[0106] The unusual operation determination module 624 can be configured to determine whether a vehicle action associated with a vehicle control gesture includes an unusual vehicle operation. Unusual operations may include vehicle actions that the vehicle is not accustomed to or does not frequently perform (e.g., compared to actions of the same vehicle in the past, actions of other or similar vehicles, or actions of the vehicle under similar conditions (e.g., location, time of day / year, weather conditions, etc.) such as sudden movements or movements significantly more extreme than those the vehicle typically performs). Whether a vehicle action is considered unusual may depend on whether the action has been performed previously, and the speed, acceleration / deceleration, degree, and range of the action. In the case of a vehicle action that has been performed previously but only at a different speed, degree, and / or range, the processor can use a threshold that makes the vehicle action unusual once it exceeds the threshold.

[0107] Furthermore, whether vehicle behavior is considered unusual may depend on the current circumstances. For example, changing lanes from the center lane to the right on a highway may not be unusual, but changing lanes from the right lane to the right (i.e., guiding the vehicle onto the highway shoulder) may be assessed as unusual by the unusual operation determination module 624. Similarly, the unusual operation determination module 624 may consider other factors such as (but not limited to) location, weather conditions, time of day / year, lighting conditions, etc.

[0108] The determination made by the unusual operation determination module 624 may be based on historical records from passenger files and / or a knowledge base of unusual / unusual vehicle actions under specified conditions (i.e., expected specifications). In some cases, the unusual operation determination module 624's determination that a vehicle action associated with a vehicle control gesture is unusual may indicate an erroneous input. For example, an erroneous input may be caused by a passenger gesture (e.g., such as gesturing or sneezing) that is not performed correctly or is not intended as a vehicle control gesture. Similarly, a passenger under the influence of controlled substances or distracted may perform a vehicle control gesture incorrectly, inappropriately, and / or unintentionally. When assessing whether a vehicle control gesture has been followed or whether an alternative vehicle action has been determined, the determination that the vehicle control gesture is unusual (i.e., erroneous input or otherwise) may be communicated to the vehicle action determination module 612. As a non-limiting example, the processor (e.g., 164) of a processing device (e.g., 500) may use electronic storage device 635 and a vehicle management system (e.g., 400, 450) to determine or assess whether a vehicle action is considered unusual.

[0109] In various embodiments, the additional instruction determination module 626 may be configured to detect whether a passenger has already performed an additional instruction in conjunction with an unusual operation. In some embodiments, the additional instruction may have been performed by the passenger along with a vehicle control gesture associated with the unusual operation (e.g., performed before, during, or after the unusual operation). The additional instruction may be performed within a window of the unusual operation (e.g., within 2 to 3 seconds before / after the unusual operation). In some embodiments, the additional instruction may be performed in response to a prompt associated with the unusual operation (e.g., the vehicle notifies the driver of an unusual operation requesting confirmation or clarification).

[0110] Additional instructions may suggest that a passenger who has performed an unusual vehicle control gesture actually intends to perform an unusual action. For example, such additional instructions may include at least one of the following: exaggerated gestures, repetitive gestures, gestures performed faster than usual, or non-visual input (e.g., audio) received in conjunction with the detected vehicle control gesture.

[0111] In some embodiments, in response to the unusual operation determination module 624 determining that the received vehicle control gesture is unusual, the system may prompt the passenger for confirmation or clarification, and the user's response as an additional instruction (e.g., a gesture of an audio command) may confirm or clarify the vehicle control gesture. The prompt to the passenger may be simple (e.g., a ringtone or a series of short tones) or may include more detailed feedback (e.g., verbal feedback) informing the passenger that additional instructions are needed. Thus, verbal feedback or a different tone sequence may inform the passenger that the received vehicle control gesture will result in unusual vehicle operation and therefore requires confirmation. As a non-limiting example, the processor (e.g., 164) of the processing device (e.g., 500) may use electronic storage device 635, one or more sensors (e.g., 101), and gesture recognition engine (e.g., 144) to determine whether the passenger has performed an additional instruction associated with the vehicle control gesture.

[0112] In some embodiments, even when the user confirms that the unusual action is expected, the vehicle operation module 629 may operate the vehicle to perform a less extreme version of the unusual action (e.g., if the unusual action exceeds a safety threshold). A less extreme version of the unusual action might involve a smaller degree of maneuvering (e.g., a single-lane change instead of a two- or three-lane change; increasing speed by 5 mph instead of 25 mph, etc.). In some embodiments, if the unusual action is too extreme (i.e., exceeds a safety threshold), the unusual action can be ignored and the passenger can be notified that the unusual action is unsafe. For example, the vehicle control system may verbally explain that the unusual action is being ignored.

[0113] The unusual operation safety assessment module 628 can be configured to determine whether any unusual vehicle operation detected by the unusual operation determination module 624 is safe for the vehicle or its occupants. As a non-limiting example, the processor (e.g., 164) of a processing device (e.g., 500) can use electronic storage device 635 and a vehicle management system (e.g., 400, 450) to determine or evaluate whether a vehicle action is considered unusual.

[0114] Vehicle operation module 629 can be configured to operate the vehicle to perform a vehicle action, such as in response to determining that a particular vehicle action is safe for the vehicle and passengers. Vehicle operation module 629 can also operate the vehicle as needed to perform alternative and / or other vehicle actions. For example, vehicle operation module 629 can operate the vehicle to perform a second vehicle action in response to determining that a second vehicle action is available. Similarly, vehicle operation module 629 can operate the vehicle to perform the first vehicle action after a determined delay period in response to determining that a first vehicle action associated with a detected first vehicle control gesture is safe for the vehicle to operate after a determined delay period. Furthermore, vehicle operation module 629 can operate the vehicle to perform a second vehicle action as a safer alternative to the first vehicle action in response to determining that an unusual vehicle action is unsafe for the vehicle or occupants (i.e., unsafe). Thus, vehicle operation module 629 can perform alternative vehicle actions determined by alternative vehicle action determination module 620. As a non-limiting example, the vehicle operation module 629 may use the processor (e.g., 164) of the processing device (e.g., 500), the electronic storage device 635, and the vehicle management system (e.g., 400, 450) to operate the vehicle (e.g., perform vehicle actions).

[0115] According to various embodiments, the vehicle operation module 629 can autonomously perform vehicle actions. Therefore, although the performed vehicle actions are caused by passenger input (i.e., from received vehicle control gestures), the vehicle operation module 629 can perform the necessary functions to safely operate the vehicle without disengaging it from autonomous mode. In some cases, the vehicle operation module 629 can perform one or more alternative vehicle actions determined to be safer than vehicle actions more strictly associated with one or more received vehicle control gestures.

[0116] In some embodiments, vehicle computing system 602 and other vehicle computing systems 604 may communicate with each other via a wireless network (e.g., 180) such as a V2V wireless communication link. Additionally, vehicle computing system 602 and other vehicle computing systems 604 may connect to a wireless communication network that provides access to external resource 630. For example, such an electronic communication link may be established at least in part via a network such as the Internet and / or other networks. It should be understood that this is not intended to be limiting, and the scope of this disclosure includes embodiments in which vehicle computing system 602, other vehicle computing systems 604, and / or external resource 630 can be operatively linked via some other communication medium.

[0117] Other vehicle computing systems 604 may also include one or more processors configured to execute computer program modules configured by machine-executable instructions 606. Machine-executable instructions 606 may include one or more instruction modules, which may include one or more of the following: a passenger profile determination module 608, a vehicle control gesture determination module 610, a vehicle action determination module 612, a passenger identification module 607, a passenger profile receiving module 616, a vehicle action safety determination module 618, an alternative vehicle action determination module 620, a delay period evaluation module 622, an unusual operation determination module 624, an additional instruction determination module 626, an unusual operation safety evaluation module 628, a vehicle operation module 629, and / or other instruction modules similar to those of the vehicle computing system 602 of the first vehicle described.

[0118] External resource 630 may include information sources outside system 600, external entities participating in system 600, and / or other resources. For example, external resource 630 may include map data resources, highway information (e.g., traffic, buildings, etc.) systems, weather forecast services, etc. In some embodiments, some or all of the functionality attributable to external resource 630 herein may be provided by resources included in system 600.

[0119] The vehicle computing device 602 may include an electronic storage device 635, one or more processors 164, and / or other components. The vehicle computing system 602 may include communication lines or ports to enable information exchange with networks and / or other vehicle computing systems. Figure 6 The illustration of vehicle computing system 602 is not intended to be limiting. Vehicle computing system 602 may include multiple hardware, software, and / or firmware components that operate together to provide the functionality attributed to vehicle computing system 602 herein. For example, vehicle computing system 602 may be implemented by a vehicle computing system cloud that operates as part of vehicle computing system 602.

[0120] Electronic storage device 635 may include a non-transitory storage medium that electronically stores information. The electronic storage medium of electronic storage device 635 may include one or both of a system storage device and / or a removable storage device provided integrally with (i.e., substantially non-removable) the vehicle computing system 602, which is removably connected to the vehicle computing system 602 via, for example, a port (e.g., a Universal Serial Bus (USB) port, a FireWire port, etc.) or a drive (e.g., a disk drive, etc.). Electronic storage device 635 may include one or more of optically readable storage media (e.g., optical discs, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard disk drives, floppy disk drives, etc.), charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash memory drives, etc.) and / or other electronically readable storage media. Electronic storage device 635 may store software algorithms, information determined by processor 164, information received from vehicle computing system 602, information received from other vehicle computing systems 604, and / or other information that enables vehicle computing system 602 to function as described herein.

[0121] Processor 164 can be configured to provide information processing capabilities in vehicle computing system 602. Therefore, processor 164 may include one or more of the following: digital processor, analog processor, digital circuitry designed to process information, analog circuitry designed to process information, state machine, and / or other mechanisms for electronically processing information. Although processor 164 in Figure 6 The processor 164 is shown as a single entity, but this is for illustrative purposes only. In some embodiments, the processor 164 may include multiple processing units. These processing units may be physically located within the same device, or the processor 164 may represent the processing capabilities of multiple devices operating in cooperation. The processor 164 may be configured to execute modules 608, 607, 610, 612, 616, 618, 620, 622, 624, 626, 628 and / or 629, and / or other modules. The processor 164 may be configured to execute modules 608, 607, 610, 612, 616, 618, 620, 622, 624, 626, 628 and / or 629, and / or other modules via: software; hardware; firmware; a combination of software, hardware and / or firmware; and / or other mechanisms for configuring processing capabilities on the processor 164. As used herein, the term "module" can refer to any component or set of components that performs the functionality belonging to a module. This may include one or more physical processors during the execution of processor-readable instructions, circuitry, hardware, storage media, or any other component.

[0122] It should be understood that although modules 608, 607, 610, 612, 616, 618, 620, 622, 624, 626, 628 and / or 629 are in Figure 6 While illustrated as being implemented within a single processing unit, in embodiments where processor 164 includes multiple processing units, one or more of modules 608, 607, 610, 612, 616, 618, 620, 622, 624, 626, 628, and / or 629 may be implemented remotely from other modules. The description of the functionality provided by the various modules 608, 607, 610, 612, 616, 618, 620, 622, 624, 626, 628, and / or 629 described below is for illustrative purposes and is not intended to be limiting, as any of modules 608, 607, 610, 612, 616, 618, 620, 622, 624, 626, 628, and / or 629 may provide more or less functionality than described. For example, one or more of modules 608, 607, 610, 612, 616, 618, 620, 622, 624, 626, 628, and / or 629 may be eliminated, and some or all of their functionality may be provided by other modules among modules 608, 607, 610, 612, 616, 618, 620, 622, 624, 626, 628, and / or 629. As another example, processor 164 may be configured to execute one or more additional modules that can perform some or all of the functionality of one of modules 608, 607, 610, 612, 616, 618, 620, 622, 624, 626, 628, and / or 629.

[0123] Figure 7A , Figure 7B and / or Figure 7C Operation of methods 700, 703, and 705 for operating a vehicle based on passenger-based vehicle control gestures, according to various embodiments, is illustrated respectively. Reference Figures 1A to 7CMethods 700, 703, and 705 can be implemented in a processor (e.g., 164), processing device (e.g., 500), and / or control unit (e.g., 140) (collectively referred to as processors) of a vehicle (e.g., 100). In some embodiments, methods 700, 703, and 705 can be executed by one or more layers within a vehicle management system stack, such as a vehicle management system (e.g., 400, 450). In some embodiments, methods 700, 703, and 705 can be executed by a processor independent of but in conjunction with a vehicle control system stack, such as a vehicle management system. For example, methods 700, 703, and 705 can be implemented as standalone software modules or within dedicated hardware that monitors data and commands from / within the vehicle management system and is configured to take actions and store data as described.

[0124] Figure 7A A method 700 for operating a vehicle based on passenger-based vehicle control gestures is shown according to various embodiments.

[0125] In block 702, the vehicle processor can determine a first vehicle action by applying a first passenger profile to a detected first vehicle control gesture performed by the first passenger. The first passenger profile can be selected from a plurality of passenger profiles to normalize the vehicle control gesture received from the first passenger. For example, the processor can detect a vehicle control gesture in the form of a passenger's outstretched palm (i.e., a stop instruction). After applying the passenger profile assigned to the current passenger, the processor can determine that the gesture indicates the passenger wishes the vehicle to stop. In some embodiments, components for performing the operations of block 702 may include a processor (e.g., 164) coupled to one or more sensors (e.g., 101) and electronic storage devices (e.g., 635). To make the determination in block 702, the processor may use a vehicle action determination module (e.g., 612).

[0126] In block 704, the vehicle processor may, in response to determining that a first vehicle action is safe for the vehicle and its occupants, operate the vehicle to perform that first vehicle action. For example, the processor may stop the vehicle. In some embodiments, components for performing the operations of block 704 may include a processor (e.g., 164) coupled to an electronic storage device (e.g., 635) and a vehicle management system (e.g., 400, 450). To make the determination in block 704, the processor may use a vehicle operation module (e.g., 629).

[0127] In some embodiments, the processor may repeat the operations in blocks 702 and 704 to operate the vehicle periodically or continuously based on the passenger's vehicle control gestures.

[0128] Figure 7BA method 703 for operating a vehicle based on passenger-based vehicle control gestures is shown according to various embodiments.

[0129] In block 706, the processor of the first vehicle can identify the first passenger based on at least one of the first passenger's position in the vehicle, the first passenger's input, or the first passenger's identification. For example, the processor can identify that the passenger is seated in the front left seat, can receive a passenger profile or at least identification information from the passenger, or facial recognition software can identify the passenger using in-vehicle imaging. In some embodiments, components for performing the operations of block 706 may include a processor (e.g., 164) coupled to an electronic storage device (e.g., 635) and a passenger identification module (e.g., 607).

[0130] In block 708, the processor may select a first passenger profile based on the identity of the first passenger. For example, the processor may select a unique passenger profile for the current occupant based on the identification card presented by the passenger when the vehicle starts. In some embodiments, the components for performing the operations of block 708 may include a processor (e.g., 164) coupled to an electronic storage device (e.g., 635) and a passenger profile determination module (e.g., 608). Following the operations in block 708 of method 703, the processor may perform the operations of block 702 of method 700 as described.

[0131] In some embodiments, the processor may repeat any or all of the operations in blocks 706 and 708 to repeatedly or continuously select passenger profiles as needed.

[0132] Figure 7C A method 705 for operating a vehicle based on passenger-based vehicle control gestures is shown according to various embodiments.

[0133] In block 710, the processor may perform operations including receiving a first passenger profile from a remote computing device for use with vehicle control gestures. For example, the processor may receive the first passenger profile as data received via a radio module (e.g., 172) or an input module (e.g., 168). In some embodiments, components for performing the operations of block 710 may include a processor (e.g., 164) coupled to an electronic storage device (e.g., 635) and a passenger profile receiving module (e.g., 616). Following the operations in block 710 of method 705, the processor may perform the operations of block 702 of method 700 as described.

[0134] In some embodiments, the processor may repeat the operations in block 710 to repeatedly or continuously receive passenger profiles or updates to passenger profiles.

[0135] Another embodiment includes a method and apparatus comprising: determining whether a first vehicle action associated with a detected and identified first vehicle control gesture is safe for the operation of the vehicle; in response to determining that the first vehicle action is unsafe for the operation of the vehicle, determining whether a second vehicle action, as a safer alternative to the first vehicle action, is available for the vehicle to operate; and in response to the availability of the second vehicle action, operating the vehicle to perform the second vehicle action.

[0136] In some embodiments, the vehicle's processor may determine whether a first vehicle action associated with a detected first vehicle control gesture is safe for the vehicle to operate after a determined delay period. Furthermore, the processor may, in response to determining that the first vehicle action or an alternative second vehicle action is safe for the vehicle to operate after the determined delay period, operate the vehicle to perform the first vehicle action or an alternative second vehicle action after the determined delay period.

[0137] In some embodiments, the vehicle's processor may determine whether a first vehicle action includes unusual vehicle operation. In some embodiments, the processor may, in response to determining that the first vehicle action includes unusual vehicle operation, determine whether a detected first vehicle control gesture includes additional instructions for a first passenger to perform unusual vehicle operation, wherein operating the vehicle to perform the first vehicle action is further in response to determining that the detected first vehicle control gesture includes additional instructions for a first passenger to perform unusual vehicle operation. In some embodiments, the additional instructions may include at least one of the following: an exaggerated gesture, a repetitive gesture, a gesture performed faster than usual, or non-visual input (e.g., audio) received in conjunction with the detected first vehicle control gesture. In some embodiments, the processor may, in response to determining that the first vehicle action is unsafe for the vehicle or occupants and determining that the detected first vehicle control gesture includes additional instructions for a first passenger to perform drastic vehicle operation, operate the vehicle to perform the first vehicle action.

[0138] Figure 8A and Figure 8B Example scenarios 800, 801 according to various embodiments are shown, in which the processor of vehicle 100 is approaching two other vehicles 804, 806, and the processor overrides passenger vehicle control gestures. References Figures 1A to 8B Three vehicles, 804, 806, and 100, are all traveling in the same direction on road 802. Road 802 is a three-lane road. According to various embodiments, the third vehicle 100 may be a semi-autonomous vehicle. The methods and systems of various embodiments can be applied to any path, regardless of whether it is a paved and clearly marked road.

[0139] refer to Figure 8AWhen the third vehicle 100 approaches a collision in the middle lane, the two leading vehicles 804 and 806 have already collided, blocking the middle lane and the right lane (i.e., the middle and farthest right lanes in the direction shown in Figure 8). According to various embodiments, the processor of the third vehicle 100 has detected the collision caused by passenger 13 (e.g., Figure 2B The vehicle control gesture 808 is performed by a rear seat passenger 13 in the vehicle 100. Although passenger 13 is in a rear seat, in some embodiments, the processor can recognize and accept vehicle control gestures from passengers in any seat within the vehicle 100. For example, a generic rear seat passenger profile can (but is not required) be applied by the processor to determine a first vehicle action 810 corresponding to the vehicle control gesture 808 performed by passenger 13. For example, since the visibility of rear seat passenger 13 may be limited or even partially obstructed, a generic rear seat passenger profile may be more readily available to overtake commands from that type of passenger. In this case, the first vehicle action 810 would cause the third vehicle 100 to veer into the right lane, which is blocked by a collision.

[0140] exist Figure 8A In the illustrated scenario, the processor of the third vehicle 100 can determine whether the first vehicle action 810 associated with the detected first vehicle control gesture 808 is safe for the operation of vehicle 100. Considering that the first vehicle action 810 would either result in a collision involving the third vehicle 100 or require the third vehicle 100 to stop very abruptly to avoid an impact, the processor can conclude that the first vehicle action 810 indicated by passenger 13 is unsafe for vehicle 100 and / or the passenger. In response to determining that the first vehicle action 810 is unsafe for the operation of vehicle 100, the processor can determine a second vehicle action 820 as a safer alternative to the first vehicle action 810. In the illustrated example, the second vehicle action 820 involves the third vehicle 100 turning into the left lane, which avoids an accident.

[0141] According to some embodiments, the processor of the third vehicle 100 can determine whether the first vehicle action 810 includes an unusual vehicle maneuver. In the illustrated example, although the first vehicle action 810 would require the vehicle to decelerate very abruptly to a stop (i.e., an unusual maneuver), it can do so if the processor can safely perform the vehicle action. For example, in addition to changing to the far right lane, the vehicle might need to move slightly or completely to the right shoulder, but could still stop behind the second leading vehicle 806. Alternatively, the vehicle could automatically perform the second vehicle action 820 and then stop in the right lane in front of the two leading vehicles 804, 806. Thus, regardless of whether passenger 13 provides additional instructions for passenger 13 to perform the first vehicle control gesture 808, which would result in a sudden and dangerous stop, the processor can operate the third vehicle 100 to implement the second vehicle action 820 as a safer alternative to the first vehicle action 810 in response to determining that the unusual vehicle maneuver is unsafe for vehicle 100 or passenger 13. In addition, the processor can determine that such sudden braking maneuvers require additional vehicle actions, such as turning on hazard lights, locking seat belts, changing headlight settings, and lowering the entertainment system volume, and the processor can automatically execute these actions.

[0142] Figure 8B This illustrates a situation where two leading vehicles 804 and 806 are traveling in the far right lane, with the second leading vehicle 806 following the first leading vehicle 804 (i.e., approaching dangerously close to point P, considering the speeds of the two leading vehicles 804 and 806). Simultaneously, a third vehicle 100 is approaching the two leading vehicles in the middle lane, and the processor of the third vehicle 100 has detected a passenger 13 (e.g., ...). Figure 2B The vehicle control gesture 808 is performed by the rear seat passenger 13. In some embodiments, the processor of the vehicle control gesture determination module 610 may not use a passenger profile (e.g., if no passenger profile is available for the designated driver and / or other passengers), but instead use a generic setting to normalize the vehicle control gesture for determining the third vehicle action 830 corresponding to the vehicle control gesture 808 performed by passenger 13. Alternatively, the processor of the vehicle control gesture determination module 610 may use a default passenger profile or a profile pre-selected for the designated driver or other passengers providing the vehicle control gesture. In this case, the third vehicle action 830 would cause the third vehicle 100 to steer into the right lane, immediately following the second leading vehicle 806.

[0143] exist Figure 8BIn the illustrated scenario, the processor of the third vehicle 100 can determine whether the third vehicle action 830 associated with the detected first vehicle control gesture 808 is safe for the operation of the third vehicle 100. Considering that the third vehicle action 830 will cause the third vehicle 100 to closely follow the second leading vehicle 806, such that if the first leading vehicle 804 or the second leading vehicle 806 suddenly stops, the third vehicle 100 may be unable to avoid a collision. In this case, the processor can determine that the third vehicle action 830 indicated by the passenger 13 is unsafe for the third vehicle 100 and / or the passenger. In response to determining that the third vehicle action 830 is unsafe for the operation of the third vehicle 100, the processor can determine a fourth vehicle action 840 as a safer alternative to the third vehicle action 830. In the illustrated example, the fourth vehicle action 840 involves the third vehicle 100 delaying the lane-changing maneuver associated with the first vehicle action 830 (e.g., delaying for 10 seconds) until the vehicle 100 has safely overtaken the two leading vehicles 804, 806. Thus, the processor of the third vehicle 100 can determine a first safety level associated with the third vehicle action 830, which may be below a safety threshold. Therefore, the processor of the third vehicle 100 can determine whether the added delay before performing the lane-changing maneuver is not only safer than the third vehicle action, but also associated with a second safety level above the safety threshold. In the illustrated example, the processor can determine a fourth vehicle action 840 similar to the third vehicle action 830, but including the delay, so that the third vehicle 100 overtakes the two leading vehicles 804, 806.

[0144] In some implementations, the processor determines Figure 8B If the third vehicle action 830 may be unsafe, the processor can prompt passenger 13 for input (e.g., verbally asking the passenger, "Would it be safer to overtake the vehicle in front in the right lane before changing lanes? Would you like to overtake the vehicle in front before changing lanes?"). In response, passenger 13 can provide input (e.g., a verbal response or another vehicle control gesture), which can be interpreted by the processor (e.g., using the additional instruction determination module 626). Thus, passenger 13 can provide input agreeing to the proposed fourth vehicle action 840, disagreeing and providing additional instructions for the need for the third vehicle action 830 (e.g., emphasizing the repetition of vehicle control gesture 808), or providing another input (e.g., a new vehicle control gesture or simply canceling the third vehicle action 830).

[0145] In some implementations, besides in Figure 8BIn addition to determining that the third vehicle action 830 is unsafe, the vehicle may also perform a different maneuver than that anticipated by the original vehicle control gesture 808 (rather than simply adding a delay). For example, the processor may detect that one or both of the leading vehicles 804, 806 are driving erratically (e.g., making a sharp turn in the lane) or in some other situation that makes it questionable whether it is safe to continue in the middle lane. Therefore, the processor may prompt the passenger 13 for input regarding the fifth vehicle action 850 (i.e., the alternative vehicle action) or execute the fifth vehicle action automatically, rather than simply suggesting a delayed action, such as the fourth vehicle action 840. For example, the fifth vehicle action 850 might involve changing lanes to the far left lane before overtaking two leading vehicles (and temporarily increasing speed in some cases), and then changing lanes to the far right lane (and in some cases returning to the original speed or other speed before overtaking other vehicles), thus still performing the maneuver anticipated by the original vehicle control gesture 808, but in a safer manner. Passenger 13 can again provide input that can be interpreted by the processor (e.g., using the additional instruction determination module 626) (e.g., a verbal response or another vehicle control gesture) to confirm the fifth vehicle action 850, cancel the original control gesture 808, provide another input, etc. Alternatively, the processor does not need to wait for passenger input (e.g., additional instructions) after determining that the initial vehicle control gesture is unsafe, and can perform the safest vehicle action that closely approximates the original control gesture 808 and ensures safe vehicle operation.

[0146] Figure 9A , Figure 9B , Figure 9C and / or Figure 9D Operation of methods 900, 903, 905, and 907 for operating a vehicle based on passenger-based vehicle control gestures, according to various embodiments, is illustrated respectively. Reference Figures 1A to 9D Methods 900, 903, 905, and 907 can be implemented in a processor (e.g., 164), processing device (e.g., 500), and / or control unit (e.g., 140) (collectively referred to as processors) of a vehicle (e.g., 100). In some embodiments, methods 900, 903, 905, and 907 can be executed by one or more layers within a vehicle management system stack, such as a vehicle management system (e.g., 400, 450). In some embodiments, methods 900, 903, 905, and 907 can be executed by a processor independent of but in conjunction with a vehicle control system stack, such as a vehicle management system. For example, methods 900, 903, 905, and 907 can be implemented as standalone software modules or within dedicated hardware that monitors data and commands from / within the vehicle management system and is configured to take actions and store data as described.

[0147] Figure 9A A method 900 for operating a vehicle based on passenger-based vehicle control gestures is shown according to various embodiments.

[0148] In some embodiments of method 900, the determination in determination block 902 may follow the operations in block 702 (as described with respect to method 700). However, in alternative embodiments of method 900, the determination in determination block 902 may follow the operations in alternative block 901.

[0149] In alternative block 901, the vehicle processor can determine a first vehicle action based on a detected first vehicle control gesture performed by a first passenger. For example, the processor can detect a vehicle control gesture that takes the form of a passenger drawing a circle upwards across their face with their finger / hand and then repeatedly drawing a circle downwards (i.e., an acceleration instruction) or other gestures or inputs. Using a knowledge base to recognize this movement, the processor can determine that this gesture means the passenger desires the vehicle to accelerate. In some embodiments, components for performing the operations of alternative block 901 may include a processor (e.g., 164) coupled to one or more sensors (e.g., 101) and electronic storage devices (e.g., 635). To make the determination in alternative block 901, the processor may use a vehicle action determination module (e.g., 612).

[0150] In determination block 902, following the operations in block 702 (as described with respect to method 700), the vehicle processor can determine whether a first vehicle action associated with the detected first vehicle control gesture is safe for vehicle operation. For example, as described above with respect to FIG8, the processor can detect a dangerous, unsafe, and / or highly unusual vehicle control gesture. After applying a passenger profile assigned to the rear seat where the passenger is seated, the processor can determine that the gesture means the passenger desires a sudden stop of the vehicle. In some embodiments, components for performing the operations of determination block 902 may include a processor (e.g., 164) coupled to one or more sensors (e.g., 101), electronic storage devices (e.g., 635), and vehicle management systems (e.g., 400, 450). To make the determination in determination block 902, the processor may use a vehicle action safety determination module (e.g., 618).

[0151] In response to the processor determining that the first vehicle action associated with the detected first vehicle control gesture is safe for vehicle operation (i.e., determining block 902 = "yes"), the processor may follow the operation in block 704 as described above with respect to method 700.

[0152] In response to the processor determining that a first vehicle action associated with a detected first vehicle control gesture is unsafe for vehicle operation (i.e., determination block 902 = "No"), the processor may determine a second vehicle action in block 908 as a safer alternative to the first vehicle action for vehicle operation. In some embodiments, components for performing the operation of determination block 902 may include a processor (e.g., 164) coupled to one or more sensors (e.g., 101), electronic storage devices (e.g., 635), vehicle management systems (e.g., 400, 450), and a vehicle action safety determination module (e.g., 618).

[0153] In block 908, the vehicle processor may determine a second vehicle action as a safer alternative to the first vehicle action for vehicle operation. For example, the processor may determine the alternative vehicle action. In some embodiments, components for performing the operations of block 908 may include a processor (e.g., 164) coupled to an electronic storage device (e.g., 635) and a vehicle management system (e.g., 400, 450). To make the determination in block 908, the processor may use an alternative vehicle action determination module (e.g., 620).

[0154] In block 910, in response to determining that a second vehicle action is a safer alternative to a first vehicle action, the vehicle processor may operate the vehicle to perform the second vehicle action. For example, the processor may determine that the first vehicle action is dangerous and / or could lead to a collision. In some embodiments, components for performing the operations of block 910 may include a processor (e.g., 164) coupled to an electronic storage device (e.g., 635) and a vehicle management system (e.g., 400, 450). To make the determination in block 910, the processor may use a vehicle operation module (e.g., 629).

[0155] In some embodiments, the processor may repeatedly determine the operations in blocks 902, 908, and 910 to operate the vehicle periodically or continuously based on the passenger's vehicle control gestures.

[0156] Figure 9B A method 903 for operating a vehicle based on passenger-based vehicle control gestures is shown according to various embodiments.

[0157] In response to the processor determining that a first vehicle action associated with a detected first vehicle control gesture is unsafe for vehicle operation (i.e., determination block 902 = "No"), the vehicle processor may determine in determination block 904 whether the first vehicle action associated with the detected first vehicle control gesture is safe for the vehicle to operate after a delay period. For example, as described above with respect to Figure 8, it would be dangerous for the vehicle to immediately turn into the right lane, but if the two leading vehicles (804, 806) move into the two left lanes or the leftmost lane, turning into the right lane after a short delay (e.g., one or two seconds of pumping the brakes) may be the safest route. Therefore, the processor may determine whether the delay period changes the determination made in determination block 902 of method 900. In some embodiments, components for performing the operation of determination block 904 may include a processor (e.g., 164) coupled to one or more sensors (e.g., 101), electronic storage devices (e.g., 635), and vehicle management systems (e.g., 400, 450). In order to make a determination in determination block 904, the processor can use a latency period evaluation module (e.g., 622).

[0158] In response to the processor determining that the first vehicle action associated with the detected first vehicle control gesture (e.g., 63) is safe for the vehicle to operate after a determined delay period (i.e., determining block 904 = "yes"), the processor may operate the vehicle to implement the first vehicle action after the determined delay period in block 906.

[0159] In response to the processor determining that a first vehicle action associated with the detected first vehicle control gesture is unsafe for vehicle operation (i.e., determination block 902 = "No"), the processor may follow the operation in block 908 of method 900 as described. In some embodiments, components for performing the operation of determination block 904 may include a processor (e.g., 164) coupled to one or more sensors (e.g., 101), electronic storage devices (e.g., 635), vehicle management systems (e.g., 400, 450), a vehicle action safety determination module (e.g., 618), and a delay period evaluation module 622.

[0160] In some embodiments, the processor may repeatedly determine any or all of the operations in blocks 904 and 906 to repeatedly or continuously determine how to operate the vehicle as needed.

[0161] Figure 9C A method 905 for operating a vehicle based on passenger-based vehicle control gestures is shown according to various embodiments.

[0162] Following the operations in block 702 of method 700 or alternative block 901 of method 900, the vehicle processor may determine in determination block 912 whether the first vehicle action includes an unusual vehicle operation. For example, if the first vehicle action includes a previously unperformed, irregular, and / or dangerous maneuver, the processor may determine that the maneuver is unusual. Therefore, the processor may evaluate and determine in determination block 912 of method 905 that the first vehicle action includes an unusual vehicle operation. In some embodiments, components for performing the operations of determination block 912 may include a processor (e.g., 164) coupled to one or more sensors (e.g., 101), electronic storage devices (e.g., 635), and vehicle management systems (e.g., 400, 450). To make the determination in determination block 912, the processor may use an unusual operation safety assessment module (e.g., 628).

[0163] In response to the processor determining that the first vehicle action includes an unusual vehicle action (i.e., determining block 912 = "No"), the processor may operate the vehicle in block 704 of the described method 700 to perform the first vehicle action.

[0164] In response to the processor determining that the first vehicle action includes an unusual vehicle operation (i.e., determination block 912 = "yes"), the processor may determine in determination block 914 whether the detected first vehicle control gesture includes additional instructions for the unusual vehicle operation expected by the first passenger.

[0165] In some embodiments, the components for performing the operation of the determination block 912 may include a processor (e.g., 164) coupled to one or more sensors (e.g., 101), electronic storage devices (e.g., 635), vehicle management systems (e.g., 400, 450), and additional indication determination modules (e.g., 626).

[0166] In determination block 914, the vehicle processor can determine whether the detected first vehicle control gesture includes additional instructions for the first passenger to perform unusual vehicle operations. For example, the passenger may exaggerate the gesture, repeat the gesture, perform the gesture faster than usual, or provide non-visual input (e.g., audio input) in conjunction with the detected first vehicle control gesture.

[0167] In response to the processor determining that the detected first vehicle control gesture includes an additional indication that the first passenger expects unusual vehicle operation (i.e., determining block 914 = "yes"), the processor may operate the vehicle to perform the first vehicle action in block 704 of the described method 700.

[0168] In response to the processor determining that the detected first vehicle control gesture does not include additional instructions for the first passenger to perform unusual vehicle operations (i.e., determining block 914 = "No"), the processor may prompt the passenger (e.g., the first passenger) for additional instructions in block 916.

[0169] In block 916, one or more passengers may be prompted with additional instructions (e.g., tone of voice, lights, vibration, etc.) regarding whether the detected first vehicle control gesture is anticipated or is currently anticipated. As described with respect to additional instruction determination module 626, prompting passengers may include details of informing passengers or otherwise corresponding to the type of additional instruction required. In some embodiments, the processor may process only responses from passengers who made the detected first vehicle control gesture. Alternatively, the processor may accept responses from any designated driver or any passenger. To give passengers time to respond, prompting for additional instructions may assign an allocated response time to the passenger, or at least the passenger who made the detected first vehicle control gesture, to respond before the processor initiates and to determine alternative actions in block 908. For example, the allocated time may be 3 to 5 seconds. In some embodiments, the duration may depend on the current circumstances. For example, when the vehicle is traveling at high speed, the processor may wait a much shorter amount of time before taking action compared to when the vehicle is traveling at low speed without additional instructions. Similarly, other factors such as location, weather conditions, time of day / year, lighting conditions, etc., may be considered when determining how long the processor should wait for additional instructions from one or more passengers. Components used to prompt passengers to perform operations on block 916 may include a processor (e.g., 164), a speaker (e.g., via a vehicle entertainment system), other vehicle computing systems 604, external resources (e.g., 630), electronic storage devices (e.g., 635), vehicle management systems (e.g., 400, 450), and additional instruction determination modules (e.g., 626).

[0170] During the allocated time period (i.e., before the allocated time expires), the processor may determine in determination block 918 whether an additional instruction has been received. Components for performing the operation of determination block 918 may include a processor (e.g., 164), an electronic storage device (e.g., 635), a vehicle management system (e.g., 400, 450), and an additional instruction determination module (e.g., 626).

[0171] In response to receiving an additional instruction during the allocated time period (i.e., determining block 918 = "Yes"), the processor may operate the vehicle in block 704 of the described method 700 to perform a first vehicle action.

[0172] In response to the absence of any additional instruction during the allocated time period (i.e., determining block 918 = "No"), the processor may follow the operation in block 908 of the described method 900.

[0173] In some embodiments, the components for performing the operations of determining block 914 may include a processor (e.g., 164) coupled to one or more sensors (e.g., 101), electronic storage devices (e.g., 635), vehicle management systems (e.g., 400, 450), and an unusual operation safety assessment module (e.g., 628).

[0174] In some embodiments, the processor may repeatedly determine any or all of the operations in blocks 912 and 914 to repeatedly or continuously determine how to operate the vehicle as needed.

[0175] Figure 9D A method 907 for operating a vehicle based on passenger-based vehicle control gestures is shown according to various embodiments.

[0176] In response to determining that the detected first vehicle control gesture includes an additional indication to the first passenger of an anticipated unusual vehicle operation (i.e., determination block 914 = "yes"), the vehicle processor may determine in determination block 920 whether the unusual vehicle operation is safe for the vehicle or the occupants. For example, even if the passenger gives an indication that the unusual vehicle operation is intentional, the processor may refuse or prevent the vehicle operation if it is unsafe. Therefore, the processor may evaluate and determine the degree of safety of the unusual vehicle operation for the vehicle and the occupants in determination block 920 of method 907. In some embodiments, components for performing the operation of determination block 920 may include a processor (e.g., 164) coupled to one or more sensors (e.g., 101), electronic storage devices (e.g., 635), and vehicle management systems (e.g., 400, 450). To make a determination in determination block 920, the processor may use an unusual operation safety assessment module (e.g., 628).

[0177] In response to the processor determining that unusual vehicle operation is unsafe (i.e., determining block 920 = "No"), the processor may determine a second vehicle action in block 908 as a safer alternative to the first vehicle action of the vehicle operation, as described above with respect to method 700.

[0178] In response to the processor determining that the unusual vehicle operation is safe (i.e., determining block 920 = "yes"), the processor can operate the vehicle in block 922 to perform the first vehicle action.

[0179] In some embodiments, the components for performing the operations of determining block 922 may include a processor (e.g., 164) coupled to one or more sensors (e.g., 101), electronic storage devices (e.g., 635), vehicle management systems (e.g., 400, 450), and an unusual operation safety assessment module (e.g., 628).

[0180] In some embodiments, the processor may repeatedly determine any or all of the operations in blocks 920 and 922 to repeatedly or continuously determine how to operate the vehicle as needed.

[0181] The foregoing method descriptions and process flowcharts are provided as illustrative examples only and are not intended to require or imply that the blocks of the various embodiments must be performed in the provided order. As will be understood by those skilled in the art, the blocks in the foregoing embodiments may be performed in any order. Words such as “afterward,” “then,” and “next” are not intended to limit the order of the blocks; these words are merely used to guide the reader through the description of the method. Furthermore, any reference to an element of the claim in the singular form (e.g., using the articles “a” or “the”) is not construed as limiting that element to the singular form.

[0182] The various illustrative logic blocks, modules, circuits, and algorithm blocks described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and frames have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application; however, such implementation decisions should not be construed as causing a departure from the scope of the various embodiments.

[0183] Hardware for implementing the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed by any of the following: a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of communication devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Alternatively, some blocks or methods may be performed by a circuit system specific to a given function.

[0184] In various embodiments, the described functionality can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on a non-transitory computer-readable medium or a non-transitory processor-readable medium. The operation of the methods or algorithms disclosed herein can be embodied in a processor-executable software module that may reside on a non-transitory computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable storage medium can be any storage medium accessible by a computer or processor. For example, but not limited to, such a non-transitory computer-readable or processor-readable medium may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically reproduce data, while optical discs optically reproduce data by means of lasers. Combinations of the foregoing are also included within the scope of non-transitory computer-readable and processor-readable media. Additionally, the operation of a method or algorithm may reside as code and / or instructions, or any combination of code and / or instructions, or a set of code and / or instructions on a non-transitory processor-readable and / or computer-readable medium that may be incorporated into a computer program product.

[0185] The prior description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the embodiments of the invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the scope of the embodiments. Therefore, the various embodiments are not intended to be limited to those shown herein, but are to be accorded the widest scope consistent with the appended claims and the principles and novel features disclosed herein.

Claims

1. A method executed by a vehicle processor for operating the vehicle in response to a passenger's vehicle control gesture, the method comprising: A first vehicle action is determined by applying a first passenger profile to a detected first vehicle control gesture performed by the first passenger, wherein the first passenger profile is selected from a plurality of passenger profiles to normalize the vehicle control gesture received from the first passenger. Determine whether the first vehicle action includes unusual vehicle operation; In response to determining that the first vehicle action includes the unusual vehicle operation, determine whether the detected first vehicle control gesture includes additional instructions for the first passenger to anticipate the unusual vehicle operation; as well as In response to determining that the detected first vehicle control gesture includes the additional instruction to the first passenger anticipating the unusual vehicle operation, the vehicle is operated to perform the first vehicle action.

2. The method of claim 1, wherein the plurality of passenger profiles includes a second passenger profile designated for a second passenger, wherein the second passenger profile standardizes vehicle control gestures differently from the first passenger profile.

3. The method according to claim 2, further comprising: The first passenger is identified based on at least one of the following: the first passenger's location in the vehicle, the first passenger's input, or the first passenger's identification. as well as Select the first passenger profile based on the first passenger's identity.

4. The method of claim 1, wherein the first passenger profile is customized for the first passenger based on training input previously received from the first passenger.

5. The method according to claim 1, further comprising: Receive the first passenger profile from the remote computing device for use in vehicle control gestures.

6. The method according to claim 1, further comprising: Determine whether the first vehicle action associated with the detected first vehicle control gesture is safe for the operation of the vehicle; In response to determining that the first vehicle action is unsafe for the operation of the vehicle, a second vehicle action is determined as a safer alternative to the first vehicle action; as well as Operate the vehicle to perform the second vehicle action.

7. The method according to claim 1, further comprising: Determine whether the first vehicle action associated with the detected first vehicle control gesture is safe for the vehicle to operate after a determined delay period; as well as In response to determining that the first vehicle action associated with the detected first vehicle control gesture is safe for the vehicle to operate after a determined delay period, the vehicle is operated to perform the first vehicle action after the determined delay period.

8. The method of claim 1, wherein the additional indication includes at least one of: an exaggerated gesture, a repetitive gesture, a gesture performed faster than usual, or non-visual input received in conjunction with the detected first vehicle control gesture.

9. The method according to claim 1, further comprising: Determine whether the unusual vehicle operation is safe for the vehicle or its occupants; as well as In response to determining that the unusual vehicle operation is unsafe for the vehicle or its occupants, the vehicle is operated to perform a second vehicle operation as a safer alternative to the first vehicle operation.

10. A vehicle control system configured for autonomous vehicles, comprising: A sensor configured to detect vehicle control gestures performed by a passenger; as well as A processor, coupled to the sensor and configured with processor-executable instructions to operate the vehicle in response to a passenger's vehicle control gestures, to: A first vehicle action is determined by applying a first passenger profile to a detected first vehicle control gesture performed by the first passenger, wherein the first passenger profile is selected from a plurality of passenger profiles to normalize the vehicle control gesture received from the first passenger. Determine whether the first vehicle action includes unusual vehicle operation; In response to determining that the first vehicle action includes the unusual vehicle operation, determine whether the detected first vehicle control gesture includes additional instructions for the first passenger to anticipate the unusual vehicle operation; as well as In response to determining that the detected first vehicle control gesture includes the additional instruction to the first passenger anticipating the unusual vehicle operation, the vehicle is operated to perform the first vehicle action.

11. The vehicle control system of claim 10, wherein the processor is further configured with processor-executable instructions such that the plurality of passenger profiles include a second passenger profile designated for a second passenger, wherein the second passenger profile normalizes vehicle control gestures differently from the first passenger profile.

12. The vehicle control system of claim 11, wherein the processor is further configured with processor-executable instructions to: The first passenger is identified based on at least one of the following: the first passenger's location in the vehicle, the first passenger's input, or the first passenger's identification; and Select the first passenger profile based on the first passenger's identity.

13. The vehicle control system of claim 10, wherein the processor is further configured with processor-executable instructions such that the first passenger profile is customized for the first passenger based on training input previously received from the first passenger.

14. The vehicle control system of claim 10, wherein the processor is further configured with processor-executable instructions to: Receive the first passenger profile from the remote computing device for use in vehicle control gestures.

15. The vehicle control system of claim 10, wherein the processor is further configured with processor-executable instructions to: Determine whether the first vehicle action associated with the detected first vehicle control gesture is safe for the operation of the vehicle; In response to determining that the first vehicle action is unsafe for the operation of the vehicle, a second vehicle action is determined as a safer alternative to the first vehicle action; as well as Operate the vehicle to perform the second vehicle action.

16. The vehicle control system of claim 10, wherein the processor is further configured with processor-executable instructions to: Determine whether the first vehicle action associated with the detected first vehicle control gesture is safe for the vehicle to operate after a determined delay period; and In response to determining that the first vehicle action associated with the detected first vehicle control gesture is safe for the vehicle to operate after a determined delay period, the vehicle is operated to perform the first vehicle action after the determined delay period.

17. The vehicle control system of claim 10, wherein the processor is further configured with processor-executable instructions such that the additional indication includes at least one of: an exaggerated gesture, a repetitive gesture, a gesture executed faster than usual, or non-visual input received in conjunction with a detected first vehicle control gesture.

18. The vehicle control system of claim 10, wherein the processor is further configured with processor-executable instructions to: Determine whether the unusual vehicle operation is safe for the vehicle or its occupants; and In response to determining that the unusual vehicle operation is unsafe for the vehicle or its occupants, the vehicle is operated to perform a second vehicle operation as a safer alternative to the first vehicle operation.

19. A non-transitory processor-readable medium having stored thereon processor-executable instructions configured to cause a processor of a vehicle control system to operate a vehicle in response to a passenger's vehicle control gesture by performing operations including: A first vehicle action is determined by applying a first passenger profile to a detected first vehicle control gesture performed by the first passenger, wherein the first passenger profile is selected from a plurality of passenger profiles to normalize the vehicle control gesture received from the first passenger. Determine whether the first vehicle action includes unusual vehicle operation; In response to determining that the first vehicle action includes the unusual vehicle operation, determine whether the detected first vehicle control gesture includes additional instructions for the first passenger to anticipate the unusual vehicle operation; as well as In response to determining that the detected first vehicle control gesture includes the additional instruction to the first passenger anticipating the unusual vehicle operation, the vehicle is operated to perform the first vehicle action.

20. The non-transitory processor-readable medium of claim 19, wherein the stored processor-executable instructions are configured to cause a processor of the vehicle control system to perform operations such that the plurality of passenger profiles include a second passenger profile designated for a second passenger, wherein the second passenger profile normalizes vehicle control gestures differently from the first passenger profile.

21. The non-transitory processor-readable medium of claim 20, wherein the stored processor-executable instructions are configured to cause the processor of the vehicle control system to perform operations further comprising: The first passenger is identified based on at least one of the following: the first passenger's location in the vehicle, the first passenger's input, or the first passenger's identification; and Select the first passenger profile based on the first passenger's identity.

22. The non-transitory processor-readable medium of claim 19, wherein the stored processor-executable instructions are configured to cause the processor of the vehicle control system to perform operations further comprising: Receive the first passenger profile from the remote computing device for use in vehicle control gestures.

23. The non-transitory processor-readable medium of claim 19, wherein the stored processor-executable instructions are configured to cause the processor of the vehicle control system to perform operations further comprising: Determine whether the first vehicle action associated with the detected first vehicle control gesture is safe for the operation of the vehicle; In response to determining that the first vehicle action is unsafe for the operation of the vehicle, a second vehicle action is determined as a safer alternative to the first vehicle action; as well as Operate the vehicle to perform the second vehicle action.

24. The non-transitory processor-readable medium of claim 19, wherein the stored processor-executable instructions are configured to cause the processor of the vehicle control system to perform operations further comprising: Determine whether the first vehicle action associated with the detected first vehicle control gesture is safe for the vehicle to operate after a determined delay period; and In response to determining that the first vehicle action associated with the detected first vehicle control gesture is safe for the vehicle to operate after a determined delay period, the vehicle is operated to perform the first vehicle action after the determined delay period.

25. The non-transitory processor-readable medium of claim 19, wherein the stored processor-executable instructions are configured to cause the processor of the vehicle control system to perform operations such that the additional instructions include at least one of: an exaggerated gesture, a repetitive gesture, a gesture executed faster than usual, or non-visual input received in conjunction with a detected first vehicle control gesture.

26. The non-transitory processor-readable medium of claim 19, wherein the stored processor-executable instructions are configured to cause the processor of the vehicle control system to perform operations further comprising: Determine whether the unusual vehicle operation is safe for the vehicle or its occupants; and In response to determining that the unusual vehicle operation is unsafe for the vehicle or its occupants, the vehicle is operated to perform a second vehicle operation as a safer alternative to the first vehicle operation.

27. A vehicle control system, comprising: A component for determining a first vehicle action by applying a first passenger profile to a detected first vehicle control gesture performed by the first passenger, wherein the first passenger profile is selected from a plurality of passenger profiles to normalize the vehicle control gesture received from the first passenger. Components used to determine whether the first vehicle action includes unusual vehicle operation; A component for determining whether a detected first vehicle control gesture includes additional instructions to the first passenger anticipating the unusual vehicle operation in response to determining that the first vehicle action includes the unusual vehicle operation; as well as Components for operating the vehicle to perform the first vehicle action in response to determining that the detected first vehicle control gesture includes the additional instruction to the first passenger that the unusual vehicle operation is anticipated.

Images