System and method for detecting gestures and completing actions outside of a vehicle

Abstract

Systems, methods, and other embodiments described herein relate to predicting an obstacle to completion of a task from a detected gesture outside of a vehicle and completing the task upon satisfaction of a corrective response.SOLUTION: In one embodiment, a method includes detecting, using sensor data, a gesture command for an action from a user outside of a vehicle. The method also includes predicting, using the sensor data, a fault regarding an uncompleted task of the operation from the vehicle state and notifying a user. The method also includes executing the incomplete task for the operation when the corrective response to the fault satisfies the parameter.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The subject matter described herein relates generally to detecting gestures by a vehicle for operation, and more particularly to predicting obstacles to task completion from detected gestures outside the vehicle. [Background technology]

[0002] A vehicle may be equipped with sensors that facilitate the perception of other vehicles, obstacles, pedestrians, and additional aspects of the surrounding environment. For example, a vehicle may be equipped with a light detection and ranging (LIDAR) sensor that uses light to scan the surrounding environment, while logic associated with the LIDAR analyzes the acquired data to detect the presence of objects and other features of the surrounding environment. In a further example, additional / alternative sensors, such as cameras, may be implemented to acquire information about the surrounding environment from which the system derives its perception of aspects of the surrounding environment. This sensor data may be useful in various situations to improve perception of the surrounding environment, so that a system, such as an autonomous driving system, can perceive the indicated aspects and accurately plan and navigate a route.

[0003] In various implementations, vehicles use sensor data to improve and personalize convenience features associated with access and comfort. For example, vehicle systems use data from proximity sensors for parking assistance by alerting the driver. Data from ambient light sensors automatically adjust interior lighting based on external conditions, which improves comfort. Vehicle access and interior settings (e.g., seat position) can change according to remote data. Yet, these vehicle systems lack easier interaction with the vehicle, including factoring in improving context, thereby reducing system satisfaction. Summary of the Invention

[0004] In one embodiment, exemplary systems and methods relate to predicting obstacles to task completion from detected gestures outside the vehicle and completing the task when corrective responses are met. In various implementations, a vehicle system detects commands from an operator and passengers to control systems such as access and convenience. For example, the access system detects a unique code associated with the vehicle operator transmitted using radio frequency from a key fob. The access system retrieves settings associated with the vehicle operator and unlocks the operator's door while leaving the other doors locked. However, upon detecting a different code from a mobile application controlling the vehicle, the access system unlocks all doors. While this customization may increase convenience, the vehicle system is limited by command types and by incorporating contexts that increase intelligence. For example, when the vehicle system receives a command during heavy rain, it automatically opens a window according to the operator's preferences, thereby damaging the vehicle floor. Thus, the vehicle system performing tasks associated with convenience and access may lack the capability and information to cause damage.

[0005] Thus, in one embodiment, the detection system estimates contextual interactions from the user after exiting the vehicle and, upon mitigating the obstacle, completes the unfinished task related to the operation associated with the vehicle command. For example, the vehicle command is a gesture command to open a window by the vehicle in an environment with an increased temperature after the vehicle is parked. Here, the detection system may predict an obstacle to completing the operation from sensor data and alert the user about a corrective response. In one approach, the detection system instructs the automated system to automatically execute a corrective response to avoid the obstacle. For example, the detection system receives a weather forecast of a rainstorm as an obstacle and delays opening the window until the rainstorm has passed. The detection system may also wait for a command from the user regarding a corrective response, such as partially opening a window instead of fully opening it.

[0006] Furthermore, in one embodiment, the detection system satisfies a parameter for a corrective response before completing the action. The parameter may be a safe area around the vehicle that is free of obstacles. Thus, once an obstacle (e.g., a pedestrian) exceeds the safe area, the vehicle completes the action. Thus, the detection system effectively completes delayed actions and tasks associated with vehicle commands from outside the vehicle through the corrective response, thereby improving the safety and reliability of the system.

[0007] In one embodiment, a detection system is disclosed that predicts an obstacle to completing a task from a detected gesture outside a vehicle and completes the task when a corrective response is satisfied. The detection system includes a memory that stores instructions that, when executed by a processor, cause the processor to detect a gesture command related to an operation from a user outside the vehicle using sensor data. The instructions also include instructions for using the sensor data to predict an obstacle related to an uncompleted task of the operation from a vehicle state and notifying the user. The instructions also include instructions for performing the uncompleted task of the operation when a corrective response to the obstacle satisfies a parameter.

[0008] In one embodiment, a non-transitory computer-readable medium is disclosed that includes instructions for predicting an obstacle to completing a task from detected gestures outside a vehicle, completing the task when a corrective response is satisfied, and causing the processor to perform one or more functions when executed by the processor. The instructions include instructions for detecting a gesture command related to an operation from a user outside the vehicle using sensor data. The instructions also include instructions for predicting an obstacle related to an uncompleted task of the operation from a vehicle state using the sensor data and notifying the user. The instructions also include instructions for performing the uncompleted task of the operation when a corrective response to the obstacle satisfies a parameter.

[0009] In one embodiment, a method is disclosed for predicting an obstacle to completing a task from detected gestures outside a vehicle and completing the task when a corrective response is satisfied. In one embodiment, the method includes detecting a gesture command related to an operation from a user outside the vehicle using sensor data. The method also includes predicting an obstacle related to an incomplete task of the operation from a vehicle state using the sensor data and notifying the user. The method also includes performing the incomplete task of the operation when a corrective response to the obstacle satisfies a parameter. [Brief explanation of the drawings]

[0010] The accompanying drawings, which are incorporated herein and constitute a part of this specification, illustrate various systems, methods, and other embodiments of the present disclosure. It will be understood that the boundaries of elements shown in the figures (e.g., boxes, groups of boxes, or other shapes) represent one embodiment of the boundaries. In some embodiments, one element may be designed as multiple elements, or multiple elements may be designed as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component, and vice versa. Additionally, elements may not be drawn to scale.

[0011] [Figure 1] FIG. 1 illustrates one embodiment of a vehicle in which the systems and methods disclosed herein may be implemented. [Figure 2] FIG. 2 illustrates one embodiment of a detection system that predicts obstacles to task completion from detected gestures outside the vehicle and associates task completion with fulfilling corrective responses. [Figure 3] FIG. 3 illustrates an example of automatically parking and exiting a vehicle safely through detecting an obstacle and completing actions to mitigate the obstacle. [Figure 4] FIG. 4 illustrates one embodiment of a method associated with predicting a fault related to an incomplete task from an operation associated with a vehicle state and eliminating the fault through a corrective response. DETAILED DESCRIPTION OF THE INVENTION

[0012] Disclosed herein are systems, methods, and other embodiments associated with predicting obstacles to task completion from detected gestures outside the vehicle and completing the task when a corrective response is met. In various implementations, a system executing vehicle commands from a user outside the vehicle lacks understanding and awareness of certain environmental scenarios, reducing reliability. For example, the user transmits a vehicle command for the vehicle to automatically exit a parking spot. While exiting the parking spot, a pedestrian may occasionally cross the path, but the vehicle may lack the intelligence to safely avoid the pedestrian while following the vehicle command. Thus, the vehicle may terminate operation to mitigate a potential collision, thereby reducing user satisfaction and reliability with automated parking.

[0013] Thus, in one embodiment, the detection system assists in controlling the vehicle from the outside using gesture commands related to the operation while mitigating and correcting obstacles (e.g., walls) associated with the operation. In particular, the detection system may sense various vehicle conditions (e.g., a garage in which the vehicle is parked) that require countermeasures to ensure a safe and secure state in order to complete a task associated with the operation (e.g., an access operation, a parking operation, etc.). In one approach, an automated driving system (ADS) automatically executes corrective responses to avoid obstacles as instructed by the detection system. For example, the detection system receives a gesture command to park the vehicle. When the perception system identifies a limited gap, the ADS retracts the side-view mirrors midway through the parking spot. In another example, the detection system alerts the user to the obstacle, suggests corrective responses, and waits for another vehicle command before completing the task. In this way, the detection system effectively and safely mitigates barriers to completion of the operation, either through automatic assistance from the ADS or through user commands, thereby avoiding aborting the operation.

[0014] Furthermore, in one embodiment, the detection system satisfies parameters associated with corrective responses to obstacles before proceeding with an uncompleted task associated with the operation. For example, the parameters may instruct the ADS to automatically stop the vehicle from entering a parking spot when the detection system, using sensor data, detects an animal crossing. Here, the detection system may automatically wait until the animal leaves a safe area (e.g., 3 feet) around the vehicle using sonar data, camera data, etc., as a corrective response that satisfies the parameters. Thus, the detection system completes the operation once the animal leaves the safe area. Thus, the detection system uses sensor data to predict obstacles to completing the operation associated with the vehicle and satisfies parameters for corrective responses that appropriately overcome the obstacles, thereby improving external and remote vehicle control.

[0015] Referring to FIG. 1 , an example of a vehicle 100 is shown. As used herein, a “vehicle” is any form of motorized transportation. In one or more implementations, the vehicle 100 is an automobile. While mechanisms related to automobiles are described herein, it will be understood that the embodiments are not limited to automobiles. In some implementations, the detection system 170 uses a roadside unit (RSU), consumer electronics (CE), mobile device, robot, drone, etc. that would benefit from the functionality described herein associated with predicting obstacles to task completion from detected gestures outside the vehicle and completing the task upon satisfying a corrective response.

[0016] Vehicle 100 also includes various elements. It will be understood that in various embodiments, vehicle 100 may have fewer elements than those shown in FIG. 1 . Vehicle 100 may have any combination of the various elements shown in FIG. 1 . Furthermore, vehicle 100 may have additional elements to those shown in FIG. 1 . In some arrangements, vehicle 100 may be implemented without one or more of the elements shown in FIG. 1 . While various elements are shown as being located within vehicle 100 in FIG. 1 , it will be understood that one or more of the elements may be located external to vehicle 100. Furthermore, the elements shown may be physically separated by large distances. For example, as discussed, one or more components of the system of the present disclosure may be implemented within the vehicle, while additional components of the system are implemented in a cloud computing environment or other system remote from vehicle 100.

[0017] Some of the possible elements of vehicle 100 are shown in FIG. 1 and described in conjunction with subsequent figures. However, a description of many of the elements in FIG. 1 is provided following the description of FIGS. 2-4 for brevity of this description. Furthermore, it will be understood that, for ease and clarity of illustration, reference numerals are appropriately repeated among different figures to indicate corresponding or similar elements. Additionally, the description outlines numerous specific details to provide a thorough understanding of the embodiments described herein. However, those skilled in the art will appreciate that the embodiments described herein may be implemented using various combinations of such elements. In either case, vehicle 100 includes detection system 170, which is implemented to perform the methods and other functions as disclosed herein with respect to predicting obstacles to task completion from detected gestures outside vehicle 100 and completing the task upon satisfying corrective responses associated with vehicle 100. In various embodiments, detection system 170 is implemented in part within vehicle 100 as a cloud-based service. For example, in one approach, functionality associated with at least one module of detection system 170 is implemented within vehicle 100, while additional functionality is implemented within a cloud-based computing system.

[0018] Referring to FIG. 2 , one embodiment of the detection system 170 of FIG. 1 is further illustrated. The detection system 170 is illustrated as including the processor 110 of the vehicle 100 of FIG. 1 . Thus, the processor 110 may be part of the detection system 170, the detection system 170 may include a processor separate from the processor 110 of the vehicle 100, or the detection system 170 may access the processor 110 through a data bus or another communication path. In one embodiment, the detection system 170 includes a memory 210 that stores an estimation module 220. The memory 210 may be a random access memory (RAM), a read-only memory (ROM), a hard disk drive, a flash memory, or other suitable memory that stores the estimation module 220. The estimation module 220 includes, for example, computer-readable instructions that, when executed by the processor 110, cause the processor 110 to perform various functions disclosed herein.

[0019] 2 is generally an abstract form of the detection system 170. Moreover, the detection system 170 and the estimation module 220 generally include instructions that function to control the processor 110 to receive data input from one or more sensors of the vehicle 100. In one embodiment, the input is observations of one or more objects in an environment proximate the vehicle 100 and / or other aspects of the surroundings. As provided herein, in one embodiment, the detection system 170 and / or the estimation module 220 acquires sensor data 250 including at least camera images. In a further mechanism, the detection system 170 and / or the estimation module 220 acquires sensor data 250 from additional sensors, such as the radar sensor 123, the lidar sensor 124, and other sensors, as may be suitable for identifying the vehicle and its location.

[0020] Thus, in one embodiment, detection system 170 and / or estimation module 220 control respective sensors to provide data input in the form of sensor data 250. Further, although detection system 170 and / or estimation module 220 are described as controlling various sensors to provide sensor data 250, in one or more embodiments, detection system 170 and / or estimation module 220 may employ other techniques, either active or passive, for acquiring sensor data 250. For example, detection system 170 passively discovers sensor data 250 from streams of electronic information provided by various sensors to additional components within vehicle 100. Further, estimation module 220 may perform various techniques to fuse data from multiple sensors when providing sensor data 250 and / or from sensor data acquired over a wireless communication link. Thus, in one embodiment, sensor data 250 represents a combination of perceptions acquired from multiple sensors.

[0021] In addition to the locations of surrounding vehicles, sensor data 250 may also include, for example, information about lane markings, etc. Furthermore, in one embodiment, detection system 170 controls sensors to acquire sensor data 250 for an area encompassing 360 degrees around vehicle 100 to provide a comprehensive assessment of the surrounding environment. Of course, in alternative embodiments, detection system 170 may acquire sensor data 250 for only the forward direction, for example, when vehicle 100 is not equipped with additional sensors to include additional areas around the vehicle and / or the additional areas are not scanned for other reasons.

[0022] Additionally, in one embodiment, detection system 170 includes data store 230. In one embodiment, data store 230 is a database. In one embodiment, a database is an electronic data structure stored in memory 210 or another data store that is comprised of routines that can be executed by processor 110 to analyze, present, organize, and the like the stored data. Thus, in one embodiment, data store 230 stores data used by estimation module 220 in performing various functions. In one embodiment, data store 230 includes sensor data 250 along with metadata that characterizes various aspects of sensor data 250, for example. For example, metadata may include location coordinates (e.g., longitude and latitude), relative map coordinates or tile identifiers, time / date stamps from when the separate sensor data 250 was generated, and the like. In one embodiment, data store 230 further includes gestures 240, which are vehicle commands detected from the user's body movements. For example, a vehicle command may be associated with one of parking and exiting vehicle 100 using a beckoning hand motion as gesture 240. An access action of unlocking a vehicle door signal with a side hand motion may be another gesture. Additionally, as described below, detection system 170 may implement a learning model that infers gesture 240 using data from one or more cameras 126, an infrared (IR) camera, one or more LIDAR sensors 124, a distance estimator, etc.

[0023] In one approach, the learning model uses a machine learning algorithm, such as a convolutional neural network (CNN), embedded within detection system 170 to perform semantic segmentation on sensor data 250 from which further information is derived. Of course, in further aspects, detection system 170 may employ a different machine learning algorithm or implement a different approach to performing associated functions that may include a deep convolutional encoder-decoder architecture, or another suitable approach to generating semantic labels for distinct object classes represented in an image. In any particular approach implemented by detection system 170, the learning model may output semantic labels that identify objects represented in sensor data 250, including gesture commands.

[0024] 3 , an example is shown of automatically parking and exiting vehicle 100 safely through detecting obstacles and completing actions to mitigate the obstacles. While the example includes parking vehicle 100, detection system 170 may detect any obstacles to completing the action and automatically mitigate the obstacles, such as by using autonomous driving module 160 or requesting user assistance. In one embodiment, detection system 170 is further configured to perform additional tasks beyond controlling the respective sensors that acquire and provide sensor data 250. For example, detection system 170 includes instructions that use sensor data 250 to cause processor 110 to detect gesture commands related to an action from a user outside vehicle 100. Estimation module 220 may use sensor data 250 to predict obstacles related to the incomplete task of the action from the vehicle state and notify the user. For example, the detection system 170 perceives the vehicle state as one of an open window, an object left in the vehicle 100, a person occupying the vehicle 100, an operator walking away from the vehicle 100, and an authorized person outside the vehicle 100. Additionally, in one approach, the detection system 170 executes outstanding tasks related to operation when a corrective response to an obstacle meets a parameter such as a change from the vehicle state.

[0025] Regarding details regarding the detection of gesture commands, the detection system 170 may implement a vision model for perception (e.g., Toyota Sense) and use the sensor data 250 to identify gesture commands. In one approach, the vision model is a data-driven, trained learning model. Here, the vision model is trained to recognize user-specific body gestures using motions of the head, hands, feet, arms, legs, etc. For example, the learning model is trained with data related to the user and vehicle state. Thus, the learning model may use the sensor data 250 during implementation to infer features of gesture commands with improved accuracy. In one approach, the gesture commands are contextually related to the vehicle state while the user is outside the vehicle 100. For example, an up gesture closes the vehicle 100 windows when the user exits. A cut and off gesture with hand motions turns off systems (e.g., lights) of the vehicle 100 when the user is outside. However, the cut and off actions may be ignored within the cabin. The locking motion (e.g., twisting and pushing the wrist) can be opening the trunk or locking the doors of the vehicle 100. For added security, the detection system 170 can authenticate the user before accepting the gesture command. Authentication can include detecting a token from a key fob, two-factor verification through a mobile application, facial recognition using data from one or more cameras 126, etc.

[0026] The estimation module 220 may use the sensor data 250 to predict obstacles related to the incomplete task of the operation from the vehicle state as follows: In one approach, the obstacle related to the vehicle state is one of a wall and a person entering a boundary area around the vehicle 100, representing a hazard associated with the parking operation. Detecting these and other obstacles may include a perception model, such as a vision model trained on a driving scene, using sonar data, ultrasound data, or the like from the sensor data 250. An obstacle and another vehicle state may include safety during an access operation where an object is near one of the doors and tailgate associated with the vehicle 100. Other examples of obstacles related to the vehicle state may include an open window, an object (e.g., an animal) left in the vehicle 100, a person occupying the vehicle 100, an operator walking away from the vehicle 100, an authorized person outside the vehicle 100, weather forecasts, local crime, and the like, which affect safety and indicate various contexts.

[0027] Furthermore, in another embodiment, the inference module 220 identifies an object (e.g., valuables) left on a seat and a window left open when a user exits the vehicle 100 as a hazard derived from the vehicle state. Thus, upon identifying the hazard, the detection system 170 pauses and abandons operation with an uncompleted task. In this way, the detection system 170 avoids safety hazards and factors in the context associated with the vehicle state.

[0028] 3 , detection system 170 identifies a gesture command to automatically park from outside vehicle 100. While vehicle 100 attempts to automatically park using self-driving module 160, estimation module 220 may use sensor data 250 and global positioning device (GPS) data from navigation system 147 to predict that parking spot 310 has limited clearance. The limited clearance may be due to vehicles 1001 and 1002 being parked near a parking boundary, which estimation module 220 perceives using sensor data 250. The limited clearance may obstruct vehicle 100, causing discomfort to the user when exiting, as vehicles 1001 and 1002 may get in the way of opening the doors. Thus, detection system 170 may pause parking operations when entering parking spot 310 with an uncompleted task until the obstacle is cleared, trigger evasive action, request assistance from the user, etc.

[0029] Furthermore, in one embodiment, detection system 170 generates an alert to notify the user of a response action to the obstacle before executing the pending task. The alert may indicate an upcoming response action automatically by vehicle 100. A user response action associated with executing the pending task may also be triggered by the alert. In one approach, the alert is one of flashing headlights, a horn, a verbal alert, an audible alert, a signal to the user's wireless device, and a visual to the wireless device. Additionally, the alert may be generated contextually using location derived from GPS information and knowledge of local conditions, such as crime and weather. For example, detection system 170 may receive a gesture command outside vehicle 100 to leave a window open in the summer and notify the user of a crime-prone area upon detecting valuables inside vehicle 100. In this manner, the alert informs the user of the response action and context associated with the obstacle, thereby improving user interaction and situational awareness.

[0030] Details regarding detection system 170 performing outstanding tasks related to operation when corrective responses to a fault meet parameters may include the following: For parking spot 310, detection system 170 may pause ongoing automated parking and delay outstanding tasks related to automated parking until vehicle conditions change. For example, detection system 170 may wait a predetermined time, anticipating that vehicle 1002 will soon exit, based on detecting brake lights using sensor data 250, which mitigates a safety hazard associated with limited clearance for parking spot 310. In one approach, detection system 170 performs a default action after a period of time expires without receiving a corrective response, such as from autonomous driving module 160 or a user.

[0031] In another example, the detection system 170 identifies a beckoning gesture 320 that automatically parks the vehicle 100 into a parking spot 330 between the other vehicles 1001 and 1002. The estimation module 220 uses the sensor data 250 (e.g., images) to predict obstacles such as the side mirrors of the other vehicles 1001 and 1002 using a learning model when entering the parking spot 330. The detection system 170 may automatically prevent the automatic parking from being aborted and avoid delays by automatically folding the side view mirrors of the vehicle 100 without pausing the turn. This may be a corrective response that meets the parameters when sufficient clearance (e.g., 4 feet) around a safety area around the vehicle 100 is provided to avoid a collision with the other vehicles 1001 and 1002. The sufficient clearance also allows occupants to exit the vehicle 100 comfortably.

[0032] Additionally, detection system 170 may also prevent the automated parking from being aborted by using vehicle commands from automated driving module 160 to stop vehicle 100 and estimating a different path to enter parking spot 330 as a corrective response. If the different path does not meet safety parameters, detection system 170 may use sensor data 250 to search for another parking spot as a corrective response.

[0033] In various implementations, the detection system 170 performs an unfinished task related to the exit gesture by navigating the vehicle 100 currently within the parking area. For example, the estimation module 220 predicts an obstacle as an approaching pedestrian 340 within the safety area of ​​the vehicle 100 exiting the parking spot 360. The autonomous driving module 160 may momentarily stop the vehicle 100 from backing out and alert the user. Here, the corrective response may be a hand gesture 350 that commands the vehicle to wait a predetermined time until the obstacle is clear. For example, the pedestrian 340 exceeding a certain distance (e.g., 15 feet) from the safety area satisfies the parameter. The corrective response may also include receiving an additional vehicle command from the autonomous driving module 160, such as slightly pushing the vehicle 100 further into the parking spot 360, when the likelihood of meeting the parameter is very low.

[0034] The outstanding task may also be associated with an access operation involving the vehicle 100. For example, the user commands the vehicle 100 to automatically open the operator's door using hand gestures and voice commands that are validated by the detection system 170 with a learning model. Here, the vehicle state is parked, and the estimation module 220 uses the sensor data 250 to predict an obstacle as the vehicle 1001. For example, the vehicle 1001 is within a surrounding safety area, such as in front of, to the side of, or behind the vehicle 100. A corrective response may be for the detection system 170 to wait until the vehicle 1001 exits an adjacent parking spot before completing the access operation. Another corrective response may be for the detection system 170 to automatically pull the vehicle 1001 out of the parking spot until it fills the safety area to open the operator's door. Thus, the detection system 170 intelligently predicts obstacles to the gesture command for the operation and selects a corrective response to complete the operation that avoids the injury.

[0035] In one approach, the obstacles are physical injury and theft associated with the detected vehicle command for the convenience operation. For example, the gesture command is for the user to round up to close a window while exiting the vehicle 100. Here, the estimation module 220 uses the sensor data 250 to detect that an animal (e.g., a dog), a child, etc., has been left behind in the vehicle 100. Thus, the detection system 170 pauses the convenience operation and alerts the user accordingly with a corrective response. For example, the corrective response is for the user to remove the animal, child, etc. from the vehicle 100, which satisfies the safety parameters. The detection system 170 then performs the unfinished task for the convenience operation.

[0036] Similarly, rolling down a window to open it while exiting vehicle 100 can be a nuisance when leaving valuables in a parking area. This behavior is particularly problematic when vehicle 100 is located in a crime-prone area. In response, a corrective response may be to close the window, lock the doors, and / or activate a security system for vehicle 100. Furthermore, activating a security system may include avoiding audible warnings to avoid noise pollution, neighborhood nuisance, etc.

[0037] Another example of a hazard predicted by the inference module 220 is future environmental damage related to a convenience operation. For example, a gesture command may be to turn the sunroof and windows to the side while exiting the vehicle 100 to leave them open when the ambient temperature rises. Here, the inference module 220 uses GPS information and sensor data 250 to infer inclement weather from a forecast for the area. For example, rain may damage the seats, floor, electrical components, etc., inside the cabin of the vehicle 100. A corrective response may be to open the sunroof of the vehicle 100, succumbing to bad weather, a changing weather forecast, etc. Thus, the detection system 170 mitigates environmental damage and physical injury associated with vehicle commands from a user outside the vehicle 100 while efficiently and intelligently completing the vehicle operation.

[0038] With reference to Figure 4, one embodiment of a method 400 associated with predicting a fault related to an incomplete task from an operation associated with a vehicle state and eliminating the fault through a corrective response is shown. Method 400 is described in terms of detection system 170 of Figures 1 and 2. While method 400 is described in conjunction with detection system 170, it should be understood that method 400 is not limited to implementation within detection system 170, but is an example of a system in which method 400 may be implemented.

[0039] At 410, detection system 170 uses sensor data 250 to detect gesture commands related to actions (e.g., access actions, parking actions, etc.) from a user outside vehicle 100. A perceptual vision model (e.g., Toyota Sense) detects features from sensor data 250 (e.g., images) to derive gesture commands. As previously described, the vision model can be a learning model that is data-driven and trained to recognize motions from body gestures related to a particular user, thereby increasing accuracy. For example, the learning model is trained with data related to a user during a particular vehicle state (e.g., entering a parking spot, exiting a parking spot, etc.). While this example describes gesture commands, detection system 170 can perceive other vehicle commands related to actions.

[0040] Additionally, in one approach, gesture commands form a relationship to the vehicle state while the user is outside of vehicle 100. For example, a cut and off gesture with a hand motion may turn off systems (e.g., lights) of vehicle 100 when the user is facing toward vehicle 100 and vehicle 100 is parked outside a garage. However, the hand motion is ignored when the user is detected walking away from vehicle 100. Similarly, a lock motion may lock the doors of vehicle 100 when parked and the user is facing the side of vehicle 100.

[0041] At 420, the estimation module 220 uses the sensor data 250 to predict an obstacle related to the incomplete task of the operation from the vehicle state, where in one embodiment the obstacle related to the vehicle state is one of a wall and a person entering a boundary area around the vehicle 100. The obstacle serves as a hazard associated with the parking operation, for example, posed by the size of the vehicle 100 and the parking position of an adjacent vehicle. As previously described, the detection system 170 may identify one of the wall and the person using the sensor data 250 and sonar data, ultrasound data, or the like from a perception model.

[0042] Additionally, an obstacle may exist for an access operation where an object is near one of the doors, tailgate, etc. associated with vehicle 100, representing another vehicle state. Other examples of obstacles related to vehicle states may include an open window, an object (e.g., an animal) left behind, a person occupying vehicle 100, an operator walking away from vehicle 100, an authorized person outside vehicle 100, etc., which impact safety. Obstacles also include weather forecasts, local crime, etc., which represent context for intelligent and insightful operation.

[0043] In one embodiment, detection system 170 generates an alert automatically by vehicle 100, such as an alert informing the user of a responsive action to an upcoming obstacle or an alert requesting a responsive action by the user. The alert may be one of a headlight flash, a horn, a verbal alert, an audible alert, a signal to the user's wireless device, and a video to the wireless device. As previously described, the alert may be generated contextually using location derived from GPS information and knowledge of local conditions, such as crime and weather. Thus, the alert informs the user of the responsive action and context associated with the obstacle, thereby increasing situational awareness.

[0044] At 430, detection system 170 determines whether the corrective response satisfies a parameter, such as a safety area, clearance, or cabin temperature associated with vehicle 100. For an incomplete task involving a parking operation by autonomous driving module 160, detection system 170 may pause the ongoing automated parking. The incomplete automated parking task may be paused until a vehicle state changes. For example, detection system 170 may detect brake lights using sensor data 250 and predict that a vehicle parked in an adjacent spot will soon exit. An unoccupied adjacent spot may reduce the safety risk associated with the limited parking clearance, thereby satisfying the parking parameter. Therefore, detection system 170 may delay automated parking for a predetermined time, independent of user input, as a corrective response. Detection system 170 may also take a default action if a certain period of time expires without receiving a corrective response from autonomous driving module 160, a user, or the like.

[0045] An access operation involving the vehicle 100 may also encounter an obstacle. Here, the user may use hand gestures to command the vehicle 100 to automatically unlock and open the operator door using an actuator motor while parked. The estimation module 220 predicts an obstacle within the safety area of ​​the vehicle 100 as an approaching bicycle. A corrective response may be for the detection system 170 to automatically wait until the perception system identifies that a bicycle has passed the operator door before completing the access operation. This satisfies the safety area for the operator door opening as a parameter. The user may also communicate a gesture command to the vehicle 100 to notify the cyclist of the access operation through flashing lights as another corrective response.

[0046] At 440, detection system 170 executes the outstanding task for the operation if the parameters are met. For a parking operation, the outstanding task may be moving from a partial park position to a fully parked position within the spot. Similarly, autonomous driving module 160 may continue to pull vehicle 100 out of the parking spot after an ongoing pause if it detects a pedestrian within the safety area. Another outstanding task may be for vehicle 100 to complete the opening of a power door using an actuator after unlocking the door due to a potential collision. In other cases, the estimation module continues to predict an obstacle for the outstanding task until the parameters are met with a corrective response. Thus, detection system 170 completes the operation associated with the vehicle and gesture commands from outside the vehicle by adapting the operation parameters to the corrective response, thereby improving the intelligence and safety of the system.

[0047] 1 will now be described in greater detail as an exemplary environment in which the systems and methods disclosed herein may operate. In some examples, vehicle 100 is configured to selectively switch between different modes of operation / control according to the orientation of one or more modules / systems of vehicle 100. In one approach, the modes include 0, no automation; 1, driver assistance; 2, partial automation; 3, conditional automation; 4, high automation; and 5, full automation. In one or more mechanisms, vehicle 100 may be configured to operate in a subset of the possible modes.

[0048] In one or more embodiments, vehicle 100 is an automated or autonomous vehicle. As used herein, "autonomous vehicle" refers to a vehicle capable of operating in an autonomous mode (e.g., Category 5, fully automated). "Autonomous mode" or "autonomous mode" refers to using one or more computing systems to control vehicle 100 with minimal or no input from a human driver to navigate and / or steer vehicle 100 along a travel route. In one or more embodiments, vehicle 100 is highly automated or fully automated. In one embodiment, vehicle 100 is configured with one or more semi-autonomous operating modes in which one or more computing systems perform a portion of the navigation and / or steering of the vehicle along a travel route, and a vehicle operator (i.e., the driver) provides input to the vehicle to perform a portion of the navigation and / or steering of vehicle 100 along the travel route.

[0049] Vehicle 100 may include one or more processors 110. In one or more arrangements, processor 110 may be the main processor of vehicle 100. For example, processor 110 may be an electronic control unit (ECU), an application specific integrated circuit (ASIC), a microprocessor, etc. Vehicle 100 may include one or more data stores 115 that store one or more types of data. Data store 115 may include volatile memory and / or non-volatile memory. Examples of suitable data stores 115 include RAM, flash memory, ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, and hard drives. Data store 115 may be a component of processor 110, or data store 115 may be operably connected to processor 110 for use by processor 110. As used throughout this specification, the term "operably connected" includes direct or indirect connections, and can include connections without direct physical contact.

[0050] In one or more arrangements, one or more data stores 115 may include map data 116. The map data 116 may include maps of one or more geographic areas. In some examples, the map data 116 may include information or data about roads, traffic control devices, road markings, structures, features, and / or landmarks within one or more geographic areas. The map data 116 may be in any suitable form. In some examples, the map data 116 may include aerial photographs of an area. In some examples, the map data 116 may include ground photographs of an area, which may include 360-degree ground photographs. The map data 116 may include measurements, dimensions, distances, and / or information about one or more features included in the map data 116 and / or for other features included in the map data 116. The map data 116 may include digital maps with information about road geometry.

[0051] In one or more arrangements, map data 116 may include one or more terrain maps 117. The terrain maps 117 may include information about the terrain, roads, surfaces, and / or other features of one or more geographic areas. The terrain maps 117 may include elevation data for one or more geographic areas. The terrain maps 117 may define one or more ground surfaces, which may include paved roads, unpaved roads, land, and other surfaces that define a ground surface.

[0052] In one or more arrangements, the map data 116 may include one or more stationary obstacle maps 118. The stationary obstacle map 118 may include information about one or more stationary obstacles located within one or more geographic areas. A "stationary obstacle" is a physical object whose position does not change or does not substantially change over a period of time and / or whose size does not change or does not substantially change over a period of time. Examples of stationary obstacles may include trees, buildings, curbs, fences, rails, centerlines, utility poles, statues, monuments, signs, benches, furniture, mailboxes, large rocks, or hills. A stationary obstacle may be an object that extends above ground level. One or more stationary obstacles included in the stationary obstacle map 118 may have location data, size data, dimension data, material data, and / or other data associated therewith. The stationary obstacle map 118 may include measurements, dimensions, distances, and / or information about one or more stationary obstacles. The stationary obstacle map 118 may be of high quality and / or high detail. The static obstacle map 118 may be updated to reflect changes in the mapped area.

[0053] One or more data stores 115 may include sensor data 119. In this context, "sensor data" means any information related to sensors equipped on vehicle 100, including capabilities and other information related to those sensors. As described below, vehicle 100 may include sensor system 120. Sensor data 119 may relate to one or more sensors of sensor system 120. As an example, in one or more arrangements, sensor data 119 may include information related to one or more LIDAR sensors 124 of sensor system 120.

[0054] In some examples, at least a portion of the map data 116 and / or sensor data 119 may be located in one or more data stores 115 located onboard the vehicle 100. Alternatively or in addition, at least a portion of the map data 116 and / or sensor data 119 may be located in one or more data stores 115 located remotely from the vehicle 100.

[0055] As described above, vehicle 100 may include sensor system 120. Sensor system 120 may include one or more sensors. A "sensor" refers to a device that can detect and / or sense something. In at least one embodiment, one or more sensors detect and / or sense in real time. As used herein, the term "real time" refers to a level of processing responsiveness that allows a user or system to sense quickly enough a particular process or decision to be made or a processor to keep up with some external process.

[0056] In arrangements where sensor system 120 includes multiple sensors, the sensors may function independently or two or more of the sensors may function in combination. Sensor system 120 and / or one or more sensors may be operatively connected to processor 110, data store 115, and / or another element of vehicle 100. Sensor system 120 may generate observations regarding a portion of the environment of vehicle 100 (e.g., nearby vehicles).

[0057] The sensor system 120 may include any suitable type of sensor. Various examples of different types of sensors are described herein. However, it will be understood that embodiments are not limited to the particular sensors described. The sensor system 120 may include one or more vehicle sensors 121. The vehicle sensors 121 may detect information about the vehicle 100 itself. In one or more arrangements, the vehicle sensors 121 may be configured to detect changes in the position and orientation of the vehicle 100, for example, based on inertial acceleration. In one or more arrangements, the vehicle sensors 121 may include one or more accelerometers, one or more gyroscopes, an inertial measurement unit (IMU), a dead reckoning system, a global navigation satellite system (GNSS), a GPS, a navigation system 147, and / or other suitable sensors. The vehicle sensors 121 may be configured to detect one or more characteristics of the vehicle 100 and / or the manner in which the vehicle 100 is operating. In one or more arrangements, the vehicle sensors 121 may include a speedometer to determine the current speed of the vehicle 100.

[0058] Alternatively or additionally, sensor system 120 may include one or more environmental sensors 122 configured to acquire data regarding the environment surrounding vehicle 100 in which vehicle 100 is operating. "Ambient environmental data" includes data regarding the external environment in which the vehicle is located or one or more portions thereof. For example, one or more environmental sensors 122 may be configured to detect obstacles and / or data related to such obstacles in at least a portion of vehicle 100's external environment. The obstacles may be stationary objects and / or dynamic objects. One or more environmental sensors 122 may be configured to detect other objects in vehicle 100's external environment, such as lane markers, signs, traffic lights, traffic signals, lane lines, crosswalks, curbs near vehicle 100, off-road objects, etc.

[0059] Described herein are various examples of sensors for sensor system 120. Example sensors may be part of one or more environmental sensors 122 and / or one or more vehicle sensors 121. However, it will be understood that embodiments are not limited to the particular sensors described.

[0060] By way of example, in one or more arrangements, sensor system 120 may include one or more of radar sensors 123, LIDAR sensors 124, sonar sensors 125, weather sensors, tactile sensors, position sensors, and / or one or more cameras 126. In one or more arrangements, one or more cameras 126 may be high dynamic range (HDR) cameras, stereo cameras, or infrared (IR) cameras.

[0061] Vehicle 100 may include input system 130. An "input system" includes a component or mechanism, or group thereof, that allows various entities to input data into a machine. Input system 130 may receive input from a vehicle occupant. Vehicle 100 may include output system 135. An "output system" includes one or more components that facilitate presenting data to a vehicle occupant.

[0062] Vehicle 100 may include one or more vehicle systems 140. Various examples of one or more vehicle systems 140 are shown in FIG. 1 . However, vehicle 100 may include more, fewer, or different vehicle systems. While certain vehicle systems are defined separately, it should be understood that any of the systems or portions thereof may otherwise be combined or separated via hardware and / or software within vehicle 100. Vehicle 100 may include a propulsion system 141, a braking system 142, a steering system 143, a throttle system 144, a transmission system 145, a signaling system 146, and / or a navigation system 147. Any of these systems may include one or more devices, components, and / or combinations thereof, now known or later developed.

[0063] Navigation system 147 may include one or more devices, applications, and / or combinations thereof, now known or later developed, configured to determine the geographic location of vehicle 100 and / or determine travel routes for vehicle 100. Navigation system 147 may include one or more mapping applications that determine travel routes for vehicle 100. Navigation system 147 may include a global positioning system, a local positioning system, or a geolocation system.

[0064] The processor 110, the detection system 170, and / or the autonomous driving module 160 may be operatively connected to communicate with various vehicle systems 140 and / or their individual components. For example, the processor 110 and / or the autonomous driving module 160 may be in communication with the various vehicle systems 140 to send and / or receive information from the various vehicle systems 140 to control the movement of the vehicle 100. The processor 110, the detection system 170, and / or the autonomous driving module 160 may control some or all of the vehicle systems 140 and, therefore, may be partially or fully autonomous as defined by the Society of Automotive Engineers (SAE) Levels 0-5.

[0065] Processor 110, detection system 170, and / or autonomous driving module 160 may be operably connected to communicate with various vehicle systems 140 and / or their individual components. For example, processor 110, detection system 170, and / or autonomous driving module 160 may be in communication to send and / or receive information from various vehicle systems 140 to control the movement of vehicle 100. Processor 110, detection system 170, and / or autonomous driving module 160 may control some or all of vehicle systems 140.

[0066] Processor 110, detection system 170, and / or self-driving module 160 may be operable to control the navigation and steering of vehicle 100 by controlling vehicle systems 140 and / or one or more of its components. For example, when operating in an autonomous mode, processor 110, detection system 170, and / or self-driving module 160 may control the direction and / or speed of vehicle 100. Processor 110, detection system 170, and / or self-driving module 160 may cause vehicle 100 to accelerate, decelerate, and / or change direction. As used herein, "cause" or "causing" means to make, force, compel, direct, command, order, command, and / or enable an event or action to occur, either directly or indirectly, or to make, force, compel, direct, command, order, and / or enable the event or action to come to be in a state in which such event or action can occur.

[0067] Vehicle 100 may include one or more actuators 150. Actuator 150 may be an element or combination of elements operable to modify vehicle system 140 or one or more of its components in response to receiving signals or other inputs from processor 110 and / or autonomous driving module 160. For example, one or more actuators 150 may include motors, pneumatic actuators, hydraulic pistons, relays, solenoids, and / or piezoelectric actuators, just to name a few possibilities.

[0068] Vehicle 100 may include one or more modules, at least some of which are described herein. The modules may be implemented as computer-readable program code that, when executed by processor 110, implements one or more of the various processes described herein. One or more of the modules may be components of processor 110, or one or more of the modules may execute on and / or be distributed among other processing systems to which processor 110 is operatively connected. The modules may include instructions (e.g., program logic) executable by one or more processors 110. Alternatively, or in addition, one or more data stores 115 may include such instructions.

[0069] In one or more arrangements, one or more of the modules described herein may include artificial intelligence elements, such as neural networks, fuzzy logic, or other machine learning algorithms. Further, in one or more arrangements, one or more of the modules may be distributed among multiple modules described herein. In one or more arrangements, two or more of the modules described herein may be combined into a single module.

[0070] Vehicle 100 may include one or more autonomous driving modules 160. Autonomous driving module 160 may be configured to receive data from sensor system 120 and / or from any other type of system capable of capturing information about vehicle 100 and / or the environment external to vehicle 100. In one or more mechanisms, autonomous driving module 160 may use the data to generate one or more driving scene models. Autonomous driving module 160 may determine the position and speed of vehicle 100. Autonomous driving module 160 may determine the location of obstacles, obstructions, or other environmental features, including traffic signs, trees, shrubs, nearby vehicles, pedestrians, etc.

[0071] The autonomous driving module 160 may be configured to receive and / or determine location information regarding obstacles in the external environment of the vehicle 100 for use by the processor 110 and / or one or more of the modules described herein, and to estimate the position and orientation of the vehicle 100, the vehicle's position in global coordinates, based on signals from multiple satellites or any other data and / or signals that may be used to determine the current state of the vehicle 100 or to determine the position of the vehicle 100 relative to the vehicle's 100 environment for use either in creating a map or in determining the position of the vehicle 100 relative to the map data.

[0072] Autonomous driving module 160 may be configured, independently or in combination with detection system 170, to determine a travel path, a current autonomous driving maneuver for vehicle 100, a future autonomous driving maneuver, and / or a modification to the current autonomous driving maneuver based on data from any other suitable sources, such as data acquired by sensor system 120, a driving scene model, and / or determinations from sensor data 250. A "driving maneuver" refers to one or more actions that affect the movement of the vehicle. Examples of driving maneuvers include accelerating, decelerating, braking, turning, moving vehicle 100 sideways, changing lanes of travel, merging into lanes of travel, and / or reversing, just to name a few possibilities. Autonomous driving module 160 may be configured to implement the determined driving maneuvers. Autonomous driving module 160 may implement such autonomous driving maneuvers directly or indirectly. As used herein, "cause" or "causing" means, either directly or indirectly, to make, command, command to occur, and / or enable an event or action to occur, or to make, command, command, and / or at least enable an event or action to be in a state in which such event or action can occur. Autonomous driving module 160 may be configured to perform various vehicle functions and / or send data to, receive data from, interact with, and / or control vehicle 100 or one or more of its systems (e.g., one or more of vehicle systems 140).

[0073] Detailed embodiments are disclosed herein. However, it should be understood that the disclosed embodiments are intended as examples. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to variously employ the aspects of the present specification in substantially any suitable detailed configuration. Furthermore, the terms and phrases used herein are not intended to be limiting, but rather to provide an understandable description of possible implementations. While various embodiments are shown in FIGS. 1-4, the embodiments are not limited to the structures or applications shown.

[0074] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, blocks in the flowcharts or block diagrams may represent modules, segments, or portions of code comprising one or more executable instructions that implement the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.

[0075] The systems, components, and / or processes described above may be implemented in hardware or a combination of hardware and software, and may be implemented in a centralized manner within one processing system, or in a distributed manner where different elements are spread across several interconnected processing systems. Any type of processing system or other apparatus configured to perform the methods described herein is suitable. A typical combination of hardware and software may be a processing system having computer-usable program code that, when loaded and executed, controls the processing system such that the processing system performs the methods described herein.

[0076] The systems, components, and / or processes may also be embodied in a computer-readable storage, such as a computer program product or other data program storage device, readable by a machine, tangibly embodying a program of instructions executable by a machine to perform the methods and processes described herein. These elements may also be embodied in an application product that provides features that enable implementation of the methods described herein and that can execute the methods when loaded into a processing system.

[0077] Furthermore, the mechanisms described herein may take the form of a computer program product having computer-readable program code embodied in, e.g., stored on, one or more computer-readable media. Any combination of one or more computer-readable media may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The phrase "computer-readable storage medium" refers to a non-transitory storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include the following: a portable computer diskette, a hard disk drive (HDD), a solid-state drive (SSD), a ROM, EPROM, or flash memory, a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the context of this specification, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0078] Generally, as used herein, a module includes a routine, program, object, component, data structure, etc. that performs a particular task or implements a particular data type. In a further aspect, a memory generally stores the noted modules. The memory associated with a module may be a buffer or cache integrated within a processor, RAM, ROM, flash memory, or another suitable electronic storage medium. In still further aspects, a module contemplated by the present disclosure is implemented as an ASIC, as a hardware component of a system-on-chip (SoC), as a programmable logic array (PLA), or as another suitable hardware component incorporating a defined configuration set (e.g., instructions) to perform the functions of the present disclosure.

[0079] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including, but not limited to, wireless, wired, fiber optic, cable, radio frequency (RF), etc., or any suitable combination thereof. Computer program code for carrying out operations for aspects of the present mechanism may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java®, Smalltalk™, C++, or the like, and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection to the external computer may be made (e.g., through the Internet using an Internet Service Provider).

[0080] The terms "a" and "an," as used herein, are defined as one or more than one. The term "plurality," as used herein, is defined as two or more than two. The term "another," as used herein, is defined as at least a second or more. The terms "including" and / or "having," as used herein, are defined as comprising (i.e., open language). The phrase "and at least one of," as used herein, refers to and includes any and all combinations of one or more of the associated listed items. As an example, the phrase "at least one of A, B, and C" includes A, B, C, or any combination thereof (e.g., AB, AC, BC, or ABC).

[0081] Aspects of the present specification may be embodied in other forms without departing from the spirit or essential attributes thereof, and reference should accordingly be made to the following claims, rather than the foregoing specification, as indicating the scope of the present specification.

Claims

1. 1. A detection system comprising: a memory storing instructions that, when executed by a processor, cause the processor to: Using the sensor data to detect gesture commands relating to a movement from a user outside the vehicle; Using the sensor data, a failure related to an uncompleted task of the operation is predicted from a vehicle state, and a notification is given to the user; A detection system that causes the outstanding task for the operation to be performed when a corrective response to the fault satisfies a parameter.

2. The instructions regarding the corrective response to the fault satisfying the parameters include:

2. The detection system of claim 1, further comprising instructions to automatically move by the vehicle one of a stop position, a door position, and a mirror position associated with the vehicle to avoid the obstacle, and the parameter is a safety area around the vehicle.

3. The instructions to perform the outstanding tasks related to the operation include:

3. The detection system of claim 2, further comprising instructions for navigating the vehicle within a parking area using commands from an automated driving system (ADS), the gesture commands being associated with one of parking and exiting the vehicle.

4. The instructions regarding the corrective response to the fault satisfying the parameters include: The detection system of claim 1 , further comprising instructions to delay the outstanding task until the vehicle condition changes, the vehicle condition being predicted.

5. The instructions to notify the user include:

10. The detection system of claim 1, further comprising instructions for generating an alert associated with the obstruction, the alert being one of a flashing headlight, a horn, a verbal alert, a signal to the user's wireless device, and a video to the wireless device.

6. The instructions regarding the corrective response to the fault satisfying the parameters include: The detection system of claim 5 , further comprising instructions for receiving a vehicle command from the user according to the vehicle state and the alert, the vehicle command being different from the gesture command.

7. The instructions for detecting the gesture command for the movement include: The detection system of claim 1 , further comprising instructions for using the sensor data to infer characteristics of the gesture command with a learning model, the learning model being trained with data regarding the user and the vehicle state.

8. the obstacle is one of a wall and a person within a boundary area around the vehicle; The detection system of claim 1 , wherein the obstruction is proximate to one of a door and a tailgate associated with the vehicle.

9. 2. The detection system of claim 1, wherein the vehicle state is one of an open window, an object left in the vehicle, a person occupying the vehicle, an operator walking away from the vehicle, an authorized person outside the vehicle, and weather forecast, and the parameter factors in changes from the vehicle state.

10. A non-transitory computer-readable medium comprising instructions, The instructions, when executed by a processor, cause the processor to: Using the sensor data to detect gesture commands relating to a movement from a user outside the vehicle; Using the sensor data, a failure related to an uncompleted task of the operation is predicted from a vehicle state, and a notification is given to the user; a non-transitory computer-readable medium that causes the outstanding task for the operation to be executed when the corrective response to the failure satisfies a parameter;

11. The instructions regarding the corrective response to the fault satisfying the parameters include:

11. The non-transitory computer-readable medium of claim 10, further comprising instructions to automatically move by the vehicle one of a stop position, a door position, and a mirror position associated with the vehicle to avoid the obstacle, and the parameter is a safety area around the vehicle.

12. using the sensor data to detect gesture commands relating to a movement from a user outside the vehicle; Using the sensor data, predicting a fault related to an uncompleted task of the operation from a vehicle state and notifying the user; executing the outstanding task for the operation when the corrective response to the fault satisfies a parameter; A method comprising:

13. The corrective response to the fault that satisfies the parameters comprises:

13. The method of claim 12, further comprising automatically moving, by the vehicle, one of a stop position, a door position, and a mirror position associated with the vehicle to avoid the obstacle, wherein the parameter is a safety area around the vehicle.

14. Executing the outstanding task related to the operation includes:

14. The method of claim 13, further comprising navigating the vehicle within a parking area using commands from an automated driving system (ADS), the gesture commands being associated with one of parking and exiting the vehicle.

15. The corrective response to the fault that satisfies the parameters comprises: The method of claim 12 , further comprising delaying the uncompleted task until the vehicle condition changes, the vehicle condition being predicted.

16. Notifying the user includes:

13. The method of claim 12, further comprising generating an alert associated with the obstruction, the alert being one of a flashing headlight, a horn, a verbal alert, a signal to the user's wireless device, and a video to the wireless device.

17. The corrective response to the fault that satisfies the parameters comprises: The method of claim 16 , further comprising receiving a vehicle command from the user according to the vehicle state and the alert, the vehicle command being different from the gesture command.

18. Detecting the gesture command related to the movement includes: The method of claim 12 , further comprising using the sensor data to infer characteristics of the gesture command with a learning model, the learning model being trained with data regarding the user and the vehicle state.

19. the obstacle is one of a wall and a person within a boundary area around the vehicle; The method of claim 12 , wherein the obstruction is proximate to one of a door and a tailgate associated with the vehicle.

20. 13. The method of claim 12, wherein the vehicle state is one of an open window, an object left in the vehicle, a person occupying the vehicle, an operator walking away from the vehicle, and a weather forecast, and the parameter factors in changes from the vehicle state.

Images