METHOD AND DEVICE FOR DEMONSTRATION OF EXPECTED VEHICLE MANEUVERS FOR A PASSENGER

Abstract

Methods and a device for displaying expected vehicle maneuvers to a passenger are disclosed. An exemplary method includes determining an expected autonomous maneuver of a vehicle based on collected information corresponding to autonomous operation of the vehicle, determining a notification type for display on a passenger device based on a device type, and displaying the expected autonomous maneuver to a user via the passenger device based on the notification type.

Description

AREA OF REVELATION

[0001] This disclosure relates generally to autonomous vehicles, and more particularly to methods and apparatus for presenting expected vehicle maneuvers to a passenger. GENERAL STATE OF THE ART

[0002] In recent years, advances in autonomous vehicle control have made it possible for a vehicle to operate entirely without a human driver. Accordingly, all people in an autonomous vehicle, including the person in the driver's seat, are considered passengers. Vehicle passengers often do not know or expect vehicle maneuvers, as would be the case with a person driving and controlling the vehicle. Thus, passengers react to vehicle maneuvers rather than anticipate them.

[0003] DE 10 2014 223 275 A1 discloses an at least partially autonomously operated vehicle. A predicted path of the vehicle is monitored with the aim of identifying an object with which the vehicle is likely to collide. Furthermore, a path is suggested for the vehicle to avoid a collision. A choice is made between following either the predicted path or the suggested path.

[0004] US 2015 / 0100179 A1 discloses a system and method for dynamic virtual reality in a vehicle. This involves determining a spatial environment around a vehicle and one or more maneuver paths for the vehicle within the spatial environment. SUMMARY

[0005] This document discloses example methods and an example device for presenting expected vehicle maneuvers to a passenger. An example device disclosed herein includes a memory containing machine-readable instructions and at least one processor to execute instructions for determining an expected autonomous maneuver of a vehicle based on collected information corresponding to autonomous operation of the vehicle. Determining whether a passenger device is an augmented reality (AR) device. Determining whether the passenger device is oriented in a direction of travel of the vehicle in response to determining that the passenger device is not an AR device or that the passenger device is not oriented in a direction of travel of the vehicle.Instructing the passenger device to display at least one of haptic feedback and a notification message about the expected autonomous maneuver, and in response to determining that the passenger device is an AR device oriented in the direction of travel of the vehicle, generating an AR-based visual representation of the expected autonomous maneuver and instructing the passenger device to display the AR-based visual representation of the expected autonomous maneuver.

[0006] An exemplary method disclosed herein includes determining an expected autonomous maneuver of a vehicle based on collected information corresponding to autonomous operation of the vehicle by executing at least one instruction with the at least one processor. Determining whether a passenger device is an augmented reality (AR) device by executing at least one instruction with the at least one processor. Determining whether the passenger device is oriented in a direction of travel of the vehicle by executing at least one instruction with the at least one processor in response to determining that the passenger device is not an AR device or that the passenger device is not oriented in the direction of travel of the vehicle.Instructing the passenger device, by executing at least one instruction with the at least one processor, to present at least one of haptic feedback and a notification message about the expected autonomous maneuver, and in response to determining that the passenger device is an AR device oriented in the direction of travel of the vehicle. Generating, by executing at least one instruction with the at least one processor, an AR-based visual representation of the expected autonomous maneuver; and instructing, by executing at least one instruction with the at least one processor, the passenger device to present the AR-based visual representation of the expected autonomous maneuver.

[0007] An exemplary tangible computer-readable storage medium disclosed herein includes instructions that, when executed, cause a machine to: determine at least one expected autonomous maneuver of a vehicle based on collected information corresponding to autonomous operation of the vehicle; determine whether a passenger device is an augmented reality (AR) device; determine whether the passenger device is oriented in a direction of travel of the vehicle in response to determining that the passenger device is not an AR device or that the passenger device is not oriented in the direction of travel of the vehicle; and instruct the passenger device to present at least one of haptic feedback and a notification message about the expected autonomous maneuver and in response to determining that the passenger device is an AR device oriented in the direction of travel of the vehicle.Generating an AR-based visual representation of the expected autonomous maneuver and instructing the passenger device to display the AR-based visual representation of the expected autonomous maneuver. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is an illustration of an autonomous vehicle with passenger devices including a vehicle maneuver manager that communicates with a vehicle control system to present expected vehicle maneuvers to passengers in accordance with the present disclosure. Fig. 2 is an exemplary illustration of a first environment as provided by a first of the passenger devices of Fig. 1 considered. Fig. Figure 3 is an exemplary illustration of a second environment as provided by the first of the passenger devices of Fig. 1 considered. Fig. Figure 4 is an exemplary illustration of a third environment as provided by the first of the passenger devices of Fig. 1 considered. Fig. Figure 5 is an exemplary illustration of a second of the passenger devices of Fig. 1. Fig. 6 is an exemplary illustration of the vehicle control system of Fig. 1. Fig. 7 is an exemplary illustration of the vehicle maneuver manager from Fig. 1. Fig. 8-9 are flowcharts representative of exemplary procedures that may be performed to activate the vehicle maneuver manager from Fig. 7 to be implemented. Fig. 10 is a block diagram of a processor platform capable of executing machine-readable instructions to perform the methods of Fig. 8-9 and the vehicle maneuver manager Fig. 7 to be implemented.

[0008] The figures are not to scale. Where possible, the same reference numerals are used throughout the drawings and the accompanying written description to identify identical or similar parts. DETAILED DESCRIPTION

[0009] Passengers of a vehicle have different driving experiences than drivers of a vehicle. Vehicle passengers react to vehicle maneuvers, whereas drivers of the vehicle are proactive toward vehicle maneuvers. Drivers of the vehicle are proactive toward vehicle maneuvers because they anticipate and control the vehicle maneuvers and are thus prepared for the maneuvers to occur. In autonomous vehicles, those who would otherwise be drivers of a vehicle are effectively passengers instead. Accordingly, none of the people riding in an autonomous vehicle can anticipate the maneuvers of the autonomous vehicle, such as rapid acceleration, rapid deceleration, stopping, turning maneuvers, crash avoidance maneuvers, etc. In such cases, passengers of the autonomous vehicle may experience inconvenience because they are unprepared for the maneuvers of the autonomous vehicle.

[0010] The exemplary methods and apparatus disclosed herein determine expected maneuvers of the autonomous vehicle based on collected information corresponding to autonomous operation of the vehicle and present the expected maneuvers to passengers via passenger devices. As used herein, the term passenger device corresponds to any portable device that a passenger (e.g., a person inside the vehicle) or a prospective passenger (e.g., a person outside the vehicle who is currently boarding the vehicle) may possess and that is capable of pairing with and / or communicating with the vehicle to present notifications of expected vehicle maneuvers to the passenger or prospective passenger. As a result of such presentation, passengers may anticipate the presented future vehicle maneuvers and / or proactively respond in anticipation of them.For example, if the expected maneuver is a rapid deceleration, passengers can lean back and counteract their forward inertia. In examples where a passenger device is coupled to and / or communicating with the vehicle, but the passenger device is external to the vehicle, a potential passenger can be notified via the passenger device of autonomous operation, including, but not limited to, the vehicle entering a vehicle parking location, leaving a vehicle parking location, changing location, or colliding with another vehicle or object.

[0011] In some examples, the passenger devices are augmented reality (“AR”) devices. As used herein, an AR device is a device capable of displaying virtual representations in spatial relation to the actual environmental characteristics. AR devices include wearable AR devices, such as Google Glass™, Microsoft Hololens™, etc., or mobile devices with cameras (e.g., smartphones, tablets, etc.). In some such examples, the expected maneuvers are presented to a passenger via a visual representation overlaid in spatial relation to an environment in front of the autonomous vehicle (e.g., augmented reality). In some examples, the passenger devices are non-AR devices, such as mobile phones, watches, smartphones / tablets that are not AR-enabled, smartphones / tablets with AR disabled, etc.In some such examples, the expected maneuver is presented to a passenger via audible notifications, text notifications, haptic feedback, etc.

[0012] In some examples, the methods and apparatus disclosed herein may determine expected maneuvers of the autonomous vehicle based on collected information corresponding to autonomous operation of the vehicle and provide information regarding the expected maneuvers to other devices coupled to and / or in communication with the vehicle. While providing expected maneuvers to passengers so that the passengers can proactively respond in anticipation of the depicted vehicle maneuvers, information regarding the expected maneuvers may be sent to other devices as disclosed herein so that the devices can proactively respond in anticipation of future vehicle maneuvers.For example, if the expected maneuver is a right turn, the methods and apparatus disclosed herein may inform a maneuver compensating cup holder of the expected right turn so that the cup holder may secure an object in the cup holder, lean in the direction of the right turn, or otherwise compensate for the centrifugal acceleration expected associated with the right turn.

[0013] Fig. 1 is an illustration of an autonomous vehicle 100 including a vehicle control system 102. The vehicle control system 102 communicates with a first passenger 104 and a second passenger 106 to represent expected vehicle maneuvers. The vehicle control system 102 communicates with the first passenger via a first passenger device 108 including a vehicle maneuver manager 110. In the illustrated example of Fig. 1, the first passenger device 108 is an AR device. The vehicle control system 102 communicates with the second passenger via a second passenger device 112, which also includes the vehicle maneuver manager 110. In some examples, the second passenger device 112 is a non-AR device. In the illustrated example of Fig. 1, the vehicle control system 102 is installed in the vehicle 100. However, in some examples, the vehicle control system 102 may be a remote server that communicates with an on-board computer of the vehicle 100 to obtain information corresponding to autonomous operation of the vehicle 100.

[0014] As disclosed herein, the majority of the processing required to determine expected vehicle maneuvers based on vehicle and environmental characteristics is performed by the vehicle control system 102, the vehicle maneuver manager 110, a remote server, or any combination thereof. For example, the vehicle control system 102 may collect information corresponding to the autonomous operation of the vehicle, determine the expected maneuvers of the autonomous vehicle based on the collected information corresponding to the autonomous operation of the vehicle, and generate notifications of the expected maneuvers for transmission to the vehicle maneuver manager 110 in a format supported by the first passenger device 108 or the second passenger device 112 for display.

[0015] In some examples, the vehicle control system 102 may collect information corresponding to the autonomous operation of the vehicle for transmission to the vehicle maneuver manager 110 in a format received from the first passenger device 108 or the second passenger device 112. In some examples, the vehicle control system 102 may be a remote server that determines the expected maneuvers of the autonomous vehicle based on the information corresponding to the autonomous operation of the vehicle collected by an on-board computer of the vehicle 100 and generates notifications of the expected maneuvers for transmission to the vehicle maneuver manager 110 in a format received from the first passenger device 108 or the second passenger device 112 for display.In some such examples, the vehicle maneuver manager 110 may determine the expected maneuvers of the autonomous vehicle based on the information corresponding to the autonomous operation of the vehicle collected by the vehicle control system 102, and present notifications of the expected maneuvers to the first passenger 104 via the first passenger device 108 or to the second passenger 106 via the second passenger device 112, respectively. Accordingly, the vehicle control system 102 and the vehicle maneuver manager 110, as illustrated herein, include at least some redundant functionality to support such processing functions.

[0016] Fig. 2 is an exemplary illustration of a first environment 200 and a second environment 201, in which the second environment 201 is as represented by the first passenger device 108 of Fig. 1 corresponds to the first environment 200. As previously disclosed, the first passenger device 108 includes the vehicle maneuver manager 110, as further described in connection with Fig. 7. The first passenger device 108 is a wearable device for augmented reality, such as a device capable of displaying the second environment 201, including visual representations in spatial relation to the first environment 200. For example, the second environment 201 includes virtualized visual representations, such as a first vehicle trajectory indicator 202, associated with actual objects, such as the interior of the vehicle 100 (e.g., the vehicle frame, the dashboard, the rearview mirror, etc.), and the environment 200 (e.g., a road, an intersection, curbs, grass, etc.). In the illustrated example of Fig. 2, the vehicle trajectory indicator 202 is displayed by the vehicle maneuver manager 110 on the first passenger device 108.

[0017] The first vehicle trajectory indicator 202 is in the illustrated example Fig. 2 as an incomplete circle to illustrate that a portion of the first vehicle trajectory indicator 202 is outside the perspective of the first passenger device 108. However, because the first vehicle trajectory indicator 202 is an augmented reality notification displayed by an AR device, a user (e.g., the first passenger 104) can change their perspective to bring the entire first vehicle trajectory indicator 202 (e.g., a complete circle) into their perspective. In such examples, the first vehicle trajectory indicator 202 remains static relative to objects in the environment 200 to create the illusion that the vehicle trajectory indicator 202 is in a physical location. While the first vehicle trajectory indicator 202 may be depicted as a (e.g.,Any other shape or visual representation may be used if the vehicle is depicted as a circle (partial or full circle). As disclosed herein, the displayed shapes or visual representations provide a vehicle passenger with visual information of which the vehicle passenger would otherwise be unaware.

[0018] The vehicle maneuver manager 110 displays a plurality of vehicle trajectory indicators on the first passenger device 108, including the first vehicle trajectory indicator 202, a second vehicle trajectory indicator 204, a third vehicle trajectory indicator 206, a fourth vehicle trajectory indicator 208, a fifth vehicle trajectory indicator 210, a sixth vehicle trajectory indicator 212, a seventh vehicle trajectory indicator 214, an eighth vehicle trajectory indicator 216, a ninth vehicle trajectory indicator 218, a tenth vehicle trajectory indicator 220, and an eleventh vehicle trajectory indicator 222.The vehicle maneuver manager 110 presents the plurality of vehicle trajectory indicators 202-222 in different sizes and / or shapes relative to objects in the first environment 200 to create the illusion that the plurality of vehicle trajectory indicators 202 are at different distances from the vehicle 100, as further described below in connection with FIG. Fig. 2-4. In some examples, a quantity indicates a distance and / or vehicle speed variation.

[0019] As in Fig. 2, the vehicle trajectory indicators 202-222 are a visual representation of an expected right turn of the vehicle 100. For example, the first vehicle trajectory indicator 202, the second vehicle trajectory indicator 204, the third vehicle trajectory indicator 206, and the fourth vehicle trajectory indicator 208 are representative of an expected path of the vehicle 100 decelerating in preparation for a right turn. To represent deceleration of the vehicle 100, the vehicle maneuver manager 110 displays the first vehicle trajectory indicator 202 as greater than the second vehicle trajectory indicator 204, the second vehicle trajectory indicator 204 as greater than the third vehicle trajectory indicator 206, and the third vehicle trajectory indicator 206 as greater than the fourth vehicle trajectory indicator 208.The vehicle maneuver manager 110 may display other visual representation information to indicate deceleration of the vehicle, such as a shape (e.g., octagons correspond to deceleration), a color (e.g., red corresponds to deceleration), converging lines, a visual phantom representation of the decelerating vehicle 100, etc.

[0020] Furthermore, the vehicle maneuver manager 110 displays the first vehicle trajectory indicator 202, the second vehicle trajectory indicator 204, the third vehicle trajectory indicator 206, and the fourth vehicle trajectory indicator 208 to create the illusion that the first vehicle trajectory indicator 202, the second vehicle trajectory indicator 204, the third vehicle trajectory indicator 206, and the fourth vehicle trajectory indicator 208 are located at increasing distances from the vehicle 100. In some examples, the first vehicle trajectory indicator 202, the second vehicle trajectory indicator 204, the third vehicle trajectory indicator 206, and the fourth vehicle trajectory indicator 208 correspond to locations where the vehicle 100 will be at a particular time.For example, the first vehicle trajectory indicator 202, the second vehicle trajectory indicator 204, the third vehicle trajectory indicator 206, and the fourth vehicle trajectory indicator 208 may represent the location where the vehicle 100 will be at one-second intervals relative to its current location and time. Accordingly, the vehicle maneuver manager 110 indicates a decrease in vehicle speed over time (e.g., a deceleration) to a passenger. The plurality of vehicle trajectory indicators may be spaced at equal or unequal intervals that are greater or less than one second.

[0021] The fourth vehicle trajectory indicator 208, the fifth vehicle trajectory indicator 210, the sixth vehicle trajectory indicator 212, and the seventh vehicle trajectory indicator 214 are representative of an expected path of the vehicle changing direction. To represent a change in direction of the vehicle 100, the vehicle maneuver manager 110 displays the fourth vehicle trajectory indicator 208, the fifth vehicle trajectory indicator 210, the sixth vehicle trajectory indicator 212, and the seventh vehicle trajectory indicator 214 as equal in size but spatially shifted to the right from the current direction of movement of the vehicle 100.

[0022] The eighth vehicle trajectory indicator 216, the ninth vehicle trajectory indicator 218, the tenth vehicle trajectory indicator 220, and the eleventh vehicle trajectory indicator 222 are representative of an expected path of the vehicle 100 accelerating from the right turn. To represent acceleration of the vehicle 100, the vehicle maneuver manager 110 displays the eighth vehicle trajectory indicator 216 smaller than the ninth vehicle trajectory indicator 218, the ninth vehicle trajectory indicator 218 smaller than the tenth vehicle trajectory indicator 220, and the tenth vehicle trajectory indicator 220 smaller than the eleventh vehicle trajectory indicator 222. The vehicle maneuver manager 110 may display other visual representation information to indicate acceleration of the vehicle, such as a color (e.g., green for acceleration), a visual phantom representation of the accelerating vehicle 100, etc.

[0023] Furthermore, the vehicle trajectory indicator 110 displays the eighth vehicle trajectory indicator 216, the ninth vehicle trajectory indicator 218, the tenth vehicle trajectory indicator 220, and the eleventh vehicle trajectory indicator 222 at increasing distances from the vehicle 100. In some examples, the eighth vehicle trajectory indicator 216, the ninth vehicle trajectory indicator 218, the tenth vehicle trajectory indicator 220, and the eleventh vehicle trajectory indicator 222 are locations where the vehicle 100 will be at a given time. For example, the eighth vehicle trajectory indicator 216, the ninth vehicle trajectory indicator 218, the tenth vehicle trajectory indicator 220, and the eleventh vehicle trajectory indicator 222 may represent the location of the vehicle 100 at one-second intervals.Accordingly, the vehicle maneuver manager 110 indicates an increase in vehicle speed over time (e.g., an acceleration) to a passenger. The plurality of vehicle trajectory indicators may be spaced at equal or unequal intervals, greater or less than one second.

[0024] Fig. 3 is an exemplary illustration of a third environment 300 and a fourth environment 301, in which the fourth environment 301 is as represented by the first passenger device 108 of Fig. 1, the third environment 300 is considered. The vehicle maneuver manager 110 displays a plurality of vehicle trajectory indicators on the first passenger device 108, including a first vehicle trajectory indicator 302, a second vehicle trajectory indicator 304, a third vehicle trajectory indicator 306, a fourth vehicle trajectory indicator 308, and a stop indicator 310. Indicators 302-310 are a visual representation of the vehicle 100 coming to a complete stop.

[0025] As in connection with Fig. 2, the vehicle maneuver manager 110 displays the first vehicle trajectory indicator 302 as greater than the second vehicle trajectory indicator 304, the second vehicle trajectory indicator 304 as greater than the third vehicle trajectory indicator 306, the third vehicle trajectory indicator 306 as greater than the fourth vehicle trajectory indicator 308, and displays the first vehicle trajectory indicator 302, the second vehicle trajectory indicator 304, the third vehicle trajectory indicator 306, and the fourth vehicle trajectory indicator 308 at increasing distances from the vehicle 100 to represent deceleration of the vehicle 100. The vehicle maneuver manager 110 displays the stop indicator 310 at a location associated with an expected stop of the vehicle 100.While the stop indicator 310 is illustrated as a stop sign, other visual representations may be used so that the passenger can visually identify the location where the vehicle 100 will stop.

[0026] Fig. 4 is an exemplary illustration of a fifth environment 400 and a sixth environment 401, in which the sixth environment 401 is as represented by the first passenger device 108 of Fig. 1 corresponds to the fifth environment 400. The fifth environment 400 includes a second vehicle 402 moving along a trajectory that could potentially result in a collision with the vehicle 100. The vehicle maneuver manager 110 displays a plurality of vehicle trajectory indicators on the first passenger device 108, including a first vehicle trajectory indicator 404, a second vehicle trajectory indicator 406, a third vehicle trajectory indicator 408, a fourth vehicle trajectory indicator 410, and a collision indicator 412. The indicators 404-412 are a visual representation of the vehicle 100 maintaining its current trajectory and potentially colliding with the second vehicle 402.

[0027] For example, the vehicle maneuver manager 110 represents expected maneuvers of the second vehicle 402 and the vehicle 100. Based on the trajectory of the second vehicle 402, the vehicle trajectory indicator 110 represents a plurality of trajectory indicators 414-420 of the second vehicle on the first passenger device 108, terminating at the collision indicator 412. Accordingly, the vehicle maneuver manager 110 represents a potential collision to the passengers of the vehicle 100.

[0028] However, the vehicle trajectory indicator 110 depicts a sixth vehicle trajectory indicator 422, a seventh vehicle trajectory indicator 424, an eighth vehicle trajectory indicator 426, and a ninth vehicle trajectory indicator 428, which are representative of the vehicle 100 avoiding the potential collision with the second vehicle 402. For example, the vehicle maneuver manager 110 depicts a representation of the vehicle 100 accelerating and avoiding the potential collision by turning left from the current trajectory. The vehicle maneuver manager 110 depicts the sixth vehicle trajectory indicator 422, the seventh vehicle trajectory indicator 424, the eighth vehicle trajectory indicator 426, and the ninth vehicle trajectory indicator 428 with solid lines as representative of the current expected maneuver of the vehicle 100 based on current information.Thus, the vehicle maneuver manager 110 represents the indicators 404-410 with dotted lines as representative of a non-selected possible maneuver of the vehicle 100.

[0029] Additionally or alternatively, the vehicle maneuver manager 110 may display other possible maneuvers, such as turning right from Fig. 2, the sudden stop from Fig. 3, driving the vehicle 100 toward the shoulder of the road, etc., to avoid the potential collision. The current expected maneuver may be selected based on any number of statistical analyses, such as which maneuver has the highest probability of safe completion. The vehicle maneuver manager 110 may change the vehicle's current expected maneuver (e.g., by changing the dotted lines to solid lines) as additional information is acquired, a statistical analysis of possible consequences is determined, etc. However, the passengers of the vehicle 100 are informed of each expected vehicle maneuver (e.g., including the current expected vehicle maneuver) of the vehicle so that the passengers can prepare accordingly.

[0030] In some examples, the vehicle maneuver manager 110 may request that a passenger confirm or select an expected maneuver from a plurality of possible maneuvers. For example, the vehicle maneuver manager 110 may not have sufficient collected information to determine a maneuver with a high probability of safe completion. In such examples, the vehicle maneuver manager 110 may notify a passenger that interaction is required for the vehicle 100 to continue continuous autonomous operation. While the description in connection with Fig. 2-4 refers to AR devices, the representation of the vehicle maneuvers associated with Fig. 2-4 may also be applicable to mobile devices.

[0031] Fig. 5 is an exemplary illustration of the second passenger device 112 of Fig. 1. In the illustrated example from Fig. 5, the second passenger device 112 is a mobile device (e.g., a smartphone) having the vehicle maneuver manager 110 and a display 500 for presenting expected maneuvers to a passenger, such as the second passenger 106. In some examples, the second passenger device 112 is not AR-enabled. In some examples, the second passenger device 112 is not often oriented in the direction of travel, so a visual representation of the expected vehicle maneuver may not be presented in spatial relation to the environment in front of the vehicle. For example, the second passenger 106 may be browsing the internet since the second passenger 106 does not need to pay attention to the driving of the autonomous vehicle 100. In such examples, the vehicle maneuver manager 110 presents notifications 502-508 associated with the expected vehicle maneuvers on the display 500 of the second passenger device 112.

[0032] For example, when the autonomous vehicle comes to a stop, the vehicle maneuver manager 110 presents a notification 502. When the autonomous vehicle has stopped, the vehicle maneuver manager 110 presents a notification 504. When the autonomous vehicle is to accelerate, the vehicle maneuver manager 110 presents a notification 506. When the autonomous vehicle is about to turn right, the vehicle maneuver manager 110 presents a notification 508. The vehicle maneuver manager 110 may couple the notifications 502-508 on the display 500 of the second passenger device 112 with haptic feedback (e.g., vibrations) and / or audio. The vehicle maneuver manager 110 may include varying levels of notification, such as increasing the volume of the audio or increasing haptic feedback, when the expected maneuver is imminent.

[0033] Fig. 6 is an exemplary illustration of the vehicle control system 102 of Fig. 1. Fig. 7 is an exemplary illustration of the vehicle maneuver manager 110 of Fig. 1-5. As previously disclosed, the majority of the processing required to determine expected vehicle maneuvers based on vehicle and environmental characteristics is performed by the vehicle control system 102, by the vehicle maneuver manager 110, or any combination thereof.

[0034] The vehicle control system 102 of Fig. 6 includes an operation collector 600, a maneuver detector 602, a warning manager 604, a communication device 606, and an image processor 608. The operation collector 600, the maneuver detector 602, the warning manager 604, the communication device 606, and the image processor 608 are communicatively coupled via a bus 610. The vehicle maneuver manager 110 of Fig. 7 includes an operation collector 700, a maneuver collector 702, an alert manager 704, environmental sensor(s) 706, an image processor 708, and a user interface 710. The operation collector 700, the maneuver collector 702, the alert manager 704, the environmental sensor(s) 706, the image processor 708, and the user interface 710 are communicatively coupled via a bus 712.

[0035] The operations collector 600 collects information corresponding to autonomous operation of the vehicle 100. For example, the operations collector 600 collects the outputs from various vehicle controllers and / or sensors, such as a speedometer, a global positioning system ("GPS"), a navigation system, or external vehicle sensors (e.g., radar, sonar, cameras, infrared, etc.). The operations collector 600 collects information such as a navigation route, acceleration, deceleration, lane changes, proximity to other vehicles, speed and / or trajectory of other vehicles, road conditions, foreign objects on the road, marker cones, traffic signals / signs, and / or other data related to the operation of the vehicle 100. In some examples, the operations collector 600 collects information from an on-board computer in control of the autonomous operation of the vehicle 100.

[0036] The maneuver collector 602 determines an expected maneuver of the vehicle 100 based on information collected by the operation collector 600. For example, if the operation collector 600 detects a pothole in the middle of the road (e.g., a road condition), the maneuver collector 602 determines that the vehicle 100 will avoid the pothole (e.g., object avoidance). Example vehicle maneuvers include, but are not limited to, turning / changing direction, changing lanes, stopping, accelerating, decelerating, honking, pulling over, passing, merging, avoiding an object, redirecting, exiting, parking, reversing, etc. In some examples, the operation collector 600 collects information related to expected vehicle maneuvers directly from the on-board computer of the vehicle 100.In such examples, the maneuver detector 602 prepares the information about the expected vehicle maneuver for use by the alert manager 604, the communication device 606, and / or the image processor 608, as further disclosed herein.

[0037] The warning manager 604 of the vehicle control system 102 communicates with the warning manager 704 ( Fig. 7) of the vehicle maneuver manager 110 to determine a type of passenger device in which the vehicle maneuver manager 110 is located. For example, the alert manager 604 receives information from the alert manager 704 of the vehicle maneuver manager 110 regarding whether the passenger device is an AR device, a non-AR device, or another display device (e.g., the windshield on the vehicle 100, an overhead display for the windshield of the vehicle 100, a navigation screen of the vehicle 100, a headrest monitor / display in the vehicle 100, etc.). Based on the device type, the alert manager 604 determines a notification type.For example, alert manager 604 determines that augmented reality notifications, text notifications, audio notifications, and / or haptic feedback notifications are applicable to AR devices. In some examples, alert manager 604 determines that text notifications, audio notifications, and / or haptic feedback notifications are applicable to non-AR devices.

[0038] In some examples, alert manager 604 determines the notification type for AR devices based on an orientation of the AR device. For example, AR notifications about vehicle maneuvers are most effective when an AR device user is paying attention to the environment outside and in front of the vehicle 100. Alert manager 604 communicates with environmental sensor(s) 706 ( Fig. 7) of the vehicle maneuver manager 110 to determine the orientation of the AR device. For example, the alert manager 604 receives an indication of the AR device's orientation from the environmental sensor(s) 706 of the vehicle maneuver manager 110. If the AR device has a forward-facing orientation, the alert manager 604 determines that AR notifications are appropriate. If the AR device does not have a forward-facing orientation, the alert manager 604 determines that text notifications, audible notifications, and / or haptic feedback notifications are appropriate. The alert manager 604 instructs the image processor 608 to generate visual representations of expected vehicle maneuvers and instructs the communication device 606 to send visual representations, text notifications, audible notifications, and / or haptic feedback notifications to AR devices.The alert manager 604 instructs the communication device to send text notifications, audible notifications, and / or haptic feedback notifications to non-AR devices.

[0039] In some examples, the communication device 606 of the vehicle control system 102 sends information corresponding to the operation of the vehicle 100 to the operation collector 700 ( Fig. 7) the vehicle maneuver manager 110. In some examples, the communication device 606 sends notifications to passenger devices based on the alert manager 604 and / or the image processor 608. The communication device 606 may utilize any number of communication protocols, including, but not limited to, Bluetooth, wireless local area network, cellular data, near field communication, etc. In some examples, the communication device 606 may determine whether a passenger device is turned on, paired, or otherwise connected / available to be connected before sending the notifications.

[0040] The image processor 608 generates visual representations of the specific vehicle maneuvers. Fig. 2, the image processor 608 generates a visual representation indicating that the vehicle 100 will soon turn right. In the illustrated example of Fig. 3, the image processor 608 generates a visual representation indicating that the vehicle 100 will soon stop. In the illustrated example of Fig. 4, the image processor 608 generates a visual representation indicating that the vehicle 100 will soon avoid a potential collision with the second vehicle 402. In some examples, the image processor 608 generates the visual representations according to various augmented reality techniques, such that the visual representation remains static during passenger movement relative to the environment.

[0041] The image processor 608 generates one or more vehicle trajectory indicators to illustrate the maneuver the vehicle will soon complete. In some examples, the image processor 608 represents the vehicle trajectory indicators as circles or spheres. However, the image processor 608 may represent the vehicle trajectory indicators as lines, squares, cubes, diamonds, phantom visual representations of the vehicle 100, or any other graphical shape(s) or image(s). The image processor 608 may generate the vehicle trajectory indicators in various sizes, shapes, and / or colors to provide a passenger with additional vehicle maneuver information, such as acceleration, deceleration, a complete stop, reversing, etc. The image processor 608 may separate the vehicle trajectory indicators by distance, time, or any other metric.Additionally or alternatively, the image processor 608 may generate visual simulations of how the vehicle 100 performs the expected maneuver before the vehicle 100 performs the maneuver.

[0042] The operation collector 700 communicates with the vehicle control system 102 to collect information corresponding to the operation of the vehicle 100. For example, the operation collector 700 of the vehicle maneuver manager 110 receives information corresponding to the operation of the vehicle 100 from the communication device 606 ( Fig. 6) of the vehicle control system 102. In some examples, the operation collector 700 communicates with an on-board computer of the vehicle 100 to collect information corresponding to the operation of the vehicle 100. In some examples, the operation collector 700 is in communication with all vehicle sensors, through which the operation collector 700 collects the information corresponding to the operation of the vehicle 100. For example, the operation collector 700 collects the outputs from various vehicle controls and / or sensors, such as a speedometer, a GPS, a navigation system, or external vehicle sensors (e.g., radar, sonar, cameras, infrared, etc.).The operation collector 700 collects information such as a navigation route, acceleration, deceleration, lane changes, proximity to other vehicles, speed and / or trajectory of other vehicles, road conditions, foreign objects on roads, marker cones, traffic signals / signs, and / or other data related to the operation of the vehicle 100.

[0043] The maneuver detector 702 determines an expected maneuver of the vehicle 100 based on information collected by the operation collector 700. For example, if the operation collector 700 detects a pothole in the middle of the road (e.g., a road condition), the maneuver detector 702 determines that the vehicle 100 will avoid the pothole. In some examples, the on-board computer of the vehicle 100 determines a vehicle maneuver (e.g., avoiding the pothole) and communicates it to the maneuver detector 702. In such examples, the maneuver detector 702 prepares the information about the expected vehicle maneuver for use by the alert manager 704, the image processor 708, and the user interface 710, as further disclosed herein.Example vehicle maneuvers include turning, changing lanes, stopping, accelerating, decelerating, honking, pulling over, overtaking, merging, avoiding an object, redirecting, exiting, parking, reversing, etc.

[0044] The alert manager 704 determines a type of passenger device in which the vehicle maneuver manager 110 is disposed. For example, the alert manager 704 determines whether the passenger device is an AR device, a non-AR device, or another computing device. Based on the device type, the alert manager 704 determines a notification type. For example, the alert manager 704 determines that augmented reality notifications, text notifications, audible notifications, and / or haptic feedback notifications are applicable to AR devices. In such examples, the alert manager 704 instructs the user interface 710 to present augmented reality notifications, text notifications, audible notifications, and / or haptic feedback notifications on AR devices.In some examples, alert manager 704 determines that text notifications, audible notifications, and / or haptic feedback notifications are applicable to non-AR devices. In such examples, alert manager 704 instructs user interface 710 to present text notifications, audible notifications, and / or haptic feedback notifications on a non-front-facing AR device and non-AR devices.

[0045] In some examples, alert manager 704 determines the notification type for AR devices based on an orientation of the AR device. For example, AR notifications about vehicle maneuvers are most effective when an AR device user is paying attention to the environment in front of vehicle 100. Alert manager 704 communicates with environmental sensor(s) 706 to determine the orientation of the AR device. If the AR device has a forward-facing orientation, alert manager 704 determines that AR notifications are appropriate. Alert manager 704 instructs image processor 708 to generate visual representations of expected vehicle maneuvers for AR devices with forward-facing perspectives.If the AR device is not in a forward-facing orientation, the alert manager 704 determines that text notifications, audible notifications, and / or haptic feedback notifications are appropriate.

[0046] The environmental sensor(s) 706 detect(s) environmental characteristics to be contrasted with the visual representation of the vehicle maneuver to generate an augmented reality notification. Example environmental sensors include, but are not limited to, cameras, accelerometers, radio frequency identification (RFID) readers, etc. In some examples, the environmental sensor(s) 706 detect(s) the environment that a passenger using the passenger device would see, so that the user interface 710 can overlay the visual representation of the vehicle maneuver generated by the image processor 708 in spatial relation to that environment, a generated augmented reality experience. In some examples, the environmental sensor(s) 706 determine(s) a perspective using the passenger device.For example, the environmental sensor(s) 706 (e.g., a camera, an RFID reader, etc.) attempts to detect markers in front of the vehicle 100, the windshield of the vehicle 100 based on size, shape, and / or position, the rearview mirror of the vehicle 100 based on size, shape, position, and reflectivity, or other identifying features relative to the front of the vehicle 100 to determine whether the device has a forward-facing perspective.

[0047] In some examples, the environmental sensor(s) 706 may detect that the environment a passenger would see has not changed for a threshold duration and determine that the passenger is not viewing the environment in front of the vehicle 100 (e.g., the environment in front of the vehicle 100 would change as the vehicle 100 moves). In some examples, the environmental sensor(s) 706 may detect g-forces at various axes of the environmental sensor(s) 706 (e.g., gravitational acceleration, vehicle acceleration / deceleration, etc.) to determine the perspective of the passenger device. In some examples, one or more of the environmental sensors 706 (e.g., cameras) may be turned off to conserve power when the passenger device is not in a forward-facing perspective.In some such examples, the one or more environmental sensors 706 are enabled when the passenger device returns to a forward-facing perspective.

[0048] The image processor 708 generates a visual representation of the vehicle maneuver. As shown in Fig. 2, the image processor 708 may generate a visual representation that the vehicle 100 will soon turn right. As shown in Fig. 3, the image processor 708 may generate a visual representation that the vehicle 100 will soon stop. As shown in Fig. 4, the image processor 708 may generate a visual representation that the vehicle 100 will soon avoid a potential collision with the second vehicle 402. In some examples, the image processor 708 generates the visual representations according to various augmented reality techniques such that the visual representation remains static during passenger movement relative to the environment.

[0049] The image processor 708 generates one or more vehicle trajectory indicators to visually illustrate the vehicle maneuver the vehicle will soon complete. In some examples, the image processor 708 represents the vehicle trajectory indicators as circles or spheres. However, the image processor 708 may represent the vehicle trajectory indicators as lines, squares, cubes, diamonds, a visual phantom representation of the vehicle 100, or any other shape(s). The image processor 708 may generate the vehicle trajectory indicators in various sizes, shapes, and / or colors to provide a passenger with additional vehicle maneuver information, such as acceleration, deceleration, a complete stop, reversing, etc. The image processor 708 may separate the vehicle trajectory indicators by a predicted distance, time, or any other metric.Additionally or alternatively, the image processor 708 may generate visual simulations of how the vehicle 100 performs the expected maneuver before the vehicle 100 performs the maneuver.

[0050] The user interface 710 represents the notification of the expected maneuver determined by the operation collector 700, the maneuver collector 702, the alert manager 704, the environmental sensor(s) 706, and / or the image processor 708. In some examples, the user interface 710 is a display, such as a display screen on a wearable AR device (e.g., Google Glass™, Microsoft Hololens™, etc.), a mobile display screen, the windshield of the vehicle 100, an overhead display for the windshield of the vehicle 100, a navigation screen of the vehicle 100, a headrest monitor / headrest display within the vehicle 100, etc.The user interface 710 presents notifications of expected maneuvers, such as turning, changing lanes, stopping, accelerating, decelerating, honking, pulling over, passing, merging, avoiding an object, redirecting, exiting, parking, reversing, etc. In some examples, additional notifications, such as vehicle status, user confirmation / selection, arrival at destination, vehicle shutdown, vehicle ignition, etc., may be presented in combination with, after, or instead of a maneuver notification, such as stopping. Such additional notifications may further alert inattentive passengers.

[0051] During operation, either the vehicle control system 102 or the vehicle maneuver manager 110 may perform the processing of the expected vehicle maneuvers. In examples for extending the battery life of mobile applications, the vehicle control system 102 performs most of the processing, such as collecting information corresponding to the operation of the autonomous vehicle 100, determining the vehicle maneuver based on the collected information, determining the notification type based on the connected passenger devices in the vehicle, and / or generating the visual representation of the vehicle maneuver. In such examples, the communication device 606 of the vehicle control system 102 transmits the output of such processing to the vehicle maneuver manager 110 for display thereof.Alternatively, the vehicle maneuver manager 110 collects information corresponding to the operation of the autonomous vehicle 100, determines the vehicle maneuver based on the collected information, determines the notification type based on the passenger device, generates the visual representation of the vehicle maneuver, and / or presents the notification to the passenger.

[0052] In Fig. 6 shows an exemplary implementation of the vehicle control system 102 from Fig. 1, but one or more of the elements, processes and / or devices shown in Fig. 6, combined, separated, rearranged, omitted, eliminated and / or implemented in any other manner. Although in Fig. 7 an exemplary implementation of the vehicle maneuver manager 110 from Fig. 1-5, one or more of the Fig. 7 may be combined, separated, rearranged, omitted, eliminated, and / or implemented in any other manner. Furthermore, the operation collector 600, the maneuver detector 602, the warning manager 604, the communication device 606, the image processor 608, and / or more generally, the vehicle control system 102 may be Fig. 1; and / or the operation collector 700, the maneuver detector 702, the warning manager 704, the environmental sensor(s) 706, the image processor 708, the user interface 710 and / or more generally the vehicle maneuver manager 110 of Fig. 1-5 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, any of the operation collector 600, the maneuver detector 602, the warning manager 604, the communication device 606, the image processor 608, and / or more generally the vehicle control system 102 Fig. 1; and / or the operation collector 700, the maneuver detector 702, the warning manager 704, the environmental sensor(s) 706, the image processor 708, the user interface 710 and / or more generally the vehicle maneuver manager 110 Fig. 1-5 may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, application specific integrated circuits (ASICs), programmable logic device(s) (PLD(s)) and / or field programmable logic device(s) (FPLD(s)).If any of the apparatus or system claims of this patent are read to cover a purely software and / or firmware implementation, at least one of the operation collector 600, the maneuver detector 602, the alert manager 604, the communication device 606, the image processor 608, the operation collector 700, the maneuver detector 702, the alert manager 704, the environmental sensor(s) 706, the image processor 708, or the user interface 710 is hereby expressly defined to include a physical computer-readable storage device or disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., upon which the software and / or firmware is / are stored. Still further, the vehicle control system 110 of FIG. Fig. 1 and / or the vehicle maneuver manager 102 from Fig. 1-5 include one or more elements, processes and / or devices in addition to or instead of those described in Fig. 6-7, and / or may include more than one of any or all of the illustrated elements, processes, and devices.

[0053] Flowchart representative of exemplary methods for implementing the vehicle control system 102 of Fig. 6 and / or the vehicle maneuver manager 110 Fig. 7 are, are in Fig. 8-9. In some examples, the methods may be implemented using machine-readable instructions including programs for execution by a processor, such as the processor 1012 shown in the example processor platform 1000 described below in connection with Fig. 10. The programs may be embodied in software stored on a tangible computer-readable storage medium, such as a CD-ROM, a floppy disk, a hard disk, a digital versatile disk (DVD), a Blu-ray disk, or memory associated with the processor 1012; however, all of the program and / or portions thereof may alternatively be executed by a device other than the processor 1012 and / or embodied in firmware or dedicated hardware. Although the example programs are described with respect to the Fig. Furthermore, many other methods for implementing the vehicle control system 102 and / or the vehicle maneuver manager 110 may alternatively be used in addition to the flowcharts illustrated in Figures 8-9. For example, the execution order of the blocks may be changed and / or some of the described blocks may be altered, omitted, or combined.

[0054] As mentioned above, the procedures from Fig. 8-9 may be implemented using encoded instructions (e.g., computer and / or machine-readable instructions) stored on a physical computer-readable storage medium, such as a hard disk drive, flash memory, read-only memory (ROM), compact disk (CD), digital versatile disk (DVD), buffer memory, random-access memory (RAM), and / or any other storage device or disk for storing information for any duration (e.g., for extended periods of time, permanently, for short periods of time, for temporarily buffering and / or caching the information).As used herein, the term "tangible computer-readable storage medium" is expressly defined to include any type of computer-readable storage device and / or storage disk and excludes propagation of signals and transmission media. As used herein, "tangible computer-readable storage medium" and "tangible machine-readable storage medium" are used interchangeably. Additionally or alternatively, the example methods of [the present invention] may be used. Fig. 8-9 may be implemented using encoded instructions (e.g., computer- and / or machine-readable instructions) stored on a non-transitory computer- and / or machine-readable medium, such as a hard disk drive, flash memory, read-only memory, a compact disk, a digital versatile disk, buffer memory, random access memory, and / or any other storage device or disk for storing information for any duration (e.g., for extended periods of time, permanently, for short periods of time, for temporarily buffering and / or caching the information). As used herein, the term "non-transitory computer-readable medium" is expressly defined to include any type of computer-readable storage device and / or disk and excludes any distribution of signals and transmission media.As used herein, the term "at least," when used as a transition phrase in the preamble of a patent claim, is just as open-ended as the term "comprising" is open-ended. Comprehensive and all variations of "comprising" are expressly defined to be open-ended terms. Including and all variations of "include" are also expressly defined to be open-ended terms. In contrast, the term consisting and / or other forms of consisting are defined to be restricted-ended terms.

[0055] Fig. 8 is a flowchart representative of a method 800 that may be performed to configure the vehicle control system 102 from Fig. 6 and / or the vehicle maneuver manager 110 Fig. 7. In a first configuration (e.g., a battery-saving configuration), blocks 802-806 are performed by the vehicle control system 102, and block 808 is performed by the vehicle maneuver manager 110. At block 802, the operation collector 600 of the vehicle control system 102 collects information corresponding to the autonomous operation of the vehicle 100. The maneuver collector 602 of the vehicle control system 102 determines expected vehicle maneuvers based on the information collected by the operation collector 600 (block 804). If the maneuver collector 602 determines an expected maneuver (block 806: YES), control transfers to block 808. However, if the maneuver collector 602 does not determine an expected maneuver (block 806: NO), control returns to block 802. At block 808, the alert manager 704 of the vehicle maneuver manager 110 presents the expected maneuver to a user via the passenger device user interface 710.After that, the procedure ends 800.

[0056] In a second configuration, blocks 802-808 are performed by the vehicle maneuver manager 110. At block 802, the operation collector 700 of the vehicle maneuver manager 110 collects information corresponding to the autonomous operation of the vehicle 100. The maneuver collector 702 of the vehicle maneuver manager 110 determines expected vehicle maneuvers based on the information collected by the operation collector 700 (block 804). If the maneuver collector 702 determines an expected maneuver (block 806: YES), control transfers to block 808. However, if the maneuver collector 702 does not determine an expected maneuver (block 806: NO), control returns to block 802. At block 808, the alert manager 704 of the vehicle maneuver manager 110 presents the expected maneuver to a user via the passenger device user interface 710. Thereafter, the method 800 ends.

[0057] Fig. 9 is a flowchart representative of a method 900 that may be performed to configure the vehicle maneuver manager 110 from Fig. 7. In some examples, method 900 is an exemplary implementation of block 808 ( Fig. 8). The method 900 begins at block 902. The alert manager 704 identifies the type of passenger device with which the vehicle maneuver manager 110 is associated. As disclosed herein, the vehicle maneuver manager 110 may include AR devices and non-AR devices. If the alert manager 704 determines that the device type is an AR device (block 904: YES), control transfers to block 906.

[0058] At block 906, the environmental sensor(s) 706 (e.g., a camera, an RFID reader, etc.) attempts to detect markers in front of the vehicle 100 (e.g., located in front of the vehicle's windshield). If the environmental sensor(s) 706 does not detect any markers in front of the vehicle 100 (block 906: NO), control transfers to block 908.

[0059] At block 908, the environmental sensor(s) 706 (e.g., a camera) attempts to detect the windshield of the vehicle 100 based on the size, shape, and / or position of the windshield. If the environmental sensor(s) 706 does not detect the windshield of the vehicle 100 (block 908: NO), control transfers to block 910.

[0060] At block 910, the environmental sensor(s) 706 (e.g., a camera) attempts to detect the rearview mirror of the vehicle 100 based on its size, shape, position, and reflectivity. If the environmental sensor(s) 706 does not detect the rearview mirror of the vehicle 100 (block 910: NO), control transfers to block 912.

[0061] At block 912, the environmental sensor(s) 706 (e.g., an accelerometer, a camera, an RFID reader, etc.) attempts to detect whether the passenger device has a forward-facing perspective. If the environmental sensor(s) 706 does not determine that the passenger device has a forward-facing perspective (block 912: NO), control transfers to block 914.

[0062] At block 914, the alert manager 704 determines that the orientation of the AR device is not in the direction of travel. Thus, control transfers to block 916 after block 914 or, if the alert manager 704 determines that the type of passenger device is not an augmented reality device (block 904: NO). At block 916, the alert manager 704 determines that the notification type is a display of haptic feedback and / or notification message. The alert manager 704 instructs the user interface 710 to display the notification type based on the information obtained at block 806 ( Fig. 8) certain expected maneuver, to display the haptic feedback and / or notification message (e.g., text and / or audio) on the passenger device. Thereafter, the method 900 ends.

[0063] However, if the environmental sensor(s) 706 detect(s) markings in front of the vehicle 100 (block 906: YES), if the environmental sensor(s) 706 detect(s) the windshield of the vehicle 100 (block 908: YES), if the environmental sensor(s) 706 detect(s) the rearview mirror of the vehicle 100 (block 910: YES), and / or if the environmental sensor(s) 706 determine(s) that the passenger device has a forward-facing perspective (block 912: YES), control transfers to block 918.

[0064] At block 918, the warning manager 704 determines that the orientation of the AR device is in the direction of travel. Fig. In the example described in Figure 9, a forward direction of travel is assumed. In some examples, method 900 may be adapted to detect markers behind vehicle 100, detect the rear window, determine a rearward perspective, etc., so that the methods and apparatus disclosed herein may also be applied to autonomous vehicle operation in reverse or while moving in a reverse direction.

[0065] The environmental sensor(s) 706 (e.g., a camera) detects environmental characteristics outside the vehicle 100 (block 920). In some examples, the image processor 708 of the vehicle maneuver manager 110 generates a visual representation of the expected maneuver, which is executed at block 806 ( Fig. 8) was determined (block 922). Alternatively, the image processor 608 of the vehicle control system 102 generates the visual representation of the expected maneuver determined at block 806. At block 924, the alert manager 704 instructs the user interface 710 to display the visual representation generated at block 922 on the passenger device in spatial relation to the environmental characteristics detected at block 920. Thereafter, the method 900 ends.

[0066] Fig. 10 is a block diagram of an exemplary processor platform 1000 capable of executing instructions for implementing the methods of Fig. 8-9 and the vehicle control system 102 Fig. 6 and / or the vehicle maneuver manager 110 Fig. 7. The processor platform 1000 may be, for example, an on-board computer of the vehicle 100, a personal computer, a portable device (e.g., a cell phone, a smartphone, a tablet such as an iPad™), a personal digital assistant (PDA), an internet device, or any other type of computing device.

[0067] The processor platform 1000 of the illustrated example includes a processor 1012. The processor 1012 of the illustrated example is hardware. For example, the processor 1012 may be implemented by one or more integrated circuits, logic circuits, microprocessors, or controllers from any desired family or manufacturer.

[0068] The processor 1012 of the illustrated example includes a local memory 1013 (e.g., a cache). The processor 1012 of the illustrated example is in communication with a main memory, which includes a volatile memory 1014 and a non-volatile memory 1016, via a bus 1018. The volatile memory 1014 may be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS dynamic random access memory (RDRAM), and / or any other type of random access memory device. The non-volatile memory 1016 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1014, 1016 is controlled by a memory controller.

[0069] The processor platform 1000 of the illustrated example also includes an interface circuit 1020. The interface circuit 1020 may be implemented using any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB), and / or a PCI Express interface.

[0070] In the illustrated example, one or more input devices 1022 are connected to the interface circuit 1020. The input device(s) 1022 enable a user to input data and / or commands into the processor 1012. The input device(s) may be implemented, for example, by an audio sensor, a microphone, a camera (still image or video), a keyboard, a button, a mouse, a touchscreen, a trackpad, a trackball, an isopoint, and / or a speech recognition system.

[0071] One or more output devices 1024 are also connected to the interface circuit 1020 of the illustrated example. The output devices 1024 may be implemented, for example, by display devices (e.g., a light-emitting diode (LED), an organic light-emitting diode (OLED), a liquid crystal display, a cathode ray tube display (CRT), a touchscreen, a tactile output device, a light-emitting diode (LED), and / or speakers). The interface circuit 1020 of the illustrated example thus typically includes a graphics driver card, a graphics driver chip, or a graphics driver processor.

[0072] The interface circuit 1020 of the illustrated example also includes a communication device, such as a transmitter, a receiver, a transceiver, a modem, and / or a network interface card, to support the exchange of data with external machines (e.g., computing devices of any type) over a network 1026 (e.g., an Ethernet connection, a Digital Subscriber Line (DSL), a telephone line, a coaxial cable, a cellular phone system, etc.).

[0073] The processor platform 1000 of the illustrated example further includes one or more mass storage devices 1028 for storing software and / or data. Examples of such mass storage devices 1028 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disk drives, RAID systems, and digital versatile disk (DVD) drives.

[0074] The coded instructions 1032 from Fig.8-9 may be stored in the mass storage device 1028, in the volatile memory 1014, in the non-volatile memory 1016, and / or a removable tangible computer-readable storage medium, such as a CD or DVD.

[0075] From the above, it is apparent that the methods, apparatus, and articles of manufacture disclosed above notify a passenger of expected maneuvers of an autonomous vehicle via one or more passenger devices. The notifications disclosed herein vary depending on the device type, including, but not limited to, an augmented reality display notification, a text message notification, an audible notification, and / or a haptic feedback notification. In some examples, the notifications vary depending on an orientation of the passenger device.

[0076] Although certain exemplary methods, devices, and products have been disclosed herein, the scope of this patent is not so limited. On the contrary, this patent covers all methods, devices, and products that properly fall within the scope of the claims of this patent.

Claims

[1] Facility comprising: a memory containing machine-readable instructions; and at least one processor to execute instructions to: Determining an expected autonomous maneuver of a vehicle based on collected information corresponding to autonomous operation of the vehicle; Determining whether a passenger device is an augmented reality (AR) device; Determining whether the passenger device is oriented in a direction of travel of the vehicle; in response to determining that the passenger device is not an AR device or that the passenger device is not oriented in the direction of travel of the vehicle, instructing the passenger device to present at least one of haptic feedback and a notification message about the expected autonomous maneuver; and in response to determining that the passenger device is an AR device oriented in the direction of travel of the vehicle: Generating an AR-based visual representation of the expected autonomous maneuver; and Instructing the passenger device to display the AR-based visual representation of the expected autonomous maneuver. [2] The device of claim 1, wherein the expected autonomous maneuver is at least one of acceleration, deceleration, stopping, object avoidance, or direction change. [3] The device of claim 1, wherein the passenger device is a wearable AR device. [4] The device of claim 3, wherein the passenger device is operable to display the AR-based visual representation of the expected autonomous maneuver via a user interface in AR. [5] The device of claim 1, wherein the at least one processor, in response to determining that the passenger device is an AR device oriented in the direction of travel of the vehicle, is further operable to execute the following instructions: Detecting environmental properties outside the vehicle; and Instructing the passenger device to display the AR-based visual representation of the expected autonomous maneuver in spatial relation to the sensed environmental properties. [6] The device of claim 5, wherein the AR device is a wearable AR device and the environmental properties correspond to an environment visible to a passenger wearing the wearable AR device. [7] The device of claim 1, wherein the at least one processor is for determining whether the passenger device is aligned in a direction of travel of the vehicle by determining whether a marker disposed in the front windshield of the vehicle has been detected by a sensor of the passenger device. [8] The device of claim 1, wherein the at least one processor is for determining whether the passenger device is aligned in a direction of travel of the vehicle by determining whether a front windshield of the vehicle has been detected by a sensor of the passenger device. [9] The device of claim 1, wherein the at least one processor is for determining whether the passenger device is aligned in a direction of travel of the vehicle by determining whether a rearview mirror marker of the vehicle has been detected by a sensor of the passenger device. [10] Method comprising: Determining an expected autonomous maneuver of a vehicle based on collected information corresponding to autonomous operation of the vehicle by executing at least one instruction with the at least one processor; Determining whether a passenger device is an augmented reality (AR) device by executing at least one instruction with the at least one processor; Determining whether the passenger device is oriented in a direction of travel of the vehicle by executing at least one instruction with the at least one processor; in response to determining that the passenger device is not an AR device or that the passenger device is not oriented in the direction of travel of the vehicle, instructing the passenger device, by executing at least one instruction with the at least one processor, to present at least one of haptic feedback and a notification message about the expected autonomous maneuver; and in response to determining that the passenger device is an AR device oriented in the direction of travel of the vehicle: Generating, by executing at least one instruction with the at least one processor, an AR-based visual representation of the expected autonomous maneuver; and Instructing, by executing at least one instruction with the at least one processor, the passenger device to display the AR-based visual representation of the expected autonomous maneuver. [11] The method of claim 10, wherein the expected autonomous maneuver is at least one of acceleration, deceleration, stopping, object avoidance, or change of direction. [12] The method of claim 10, further comprising, in response to determining that the passenger device is an AR device oriented in the direction of travel of the vehicle: Detecting environmental properties outside the vehicle; and Instructing the passenger device to display the AR-based visual representation of the expected autonomous maneuver in spatial relation to the sensed environmental properties. [13] The method of claim 12, wherein the AR device is a wearable AR device and the environmental properties correspond to an environment visible to a passenger wearing the wearable AR device. [14] The method of claim 10, wherein determining whether the passenger device is oriented in a direction of travel of the vehicle includes determining, by executing at least one instruction with the at least one processor, whether a marker disposed in the front windshield of the vehicle has been detected by a sensor of the passenger device. [15] The method of claim 10, wherein determining whether the passenger device is oriented in a direction of travel of the vehicle includes determining, by executing at least one instruction with the at least one processor, whether a front windshield of the vehicle has been detected by a sensor of the passenger device. [16] The method of claim 10, wherein determining whether the passenger device is oriented in a direction of travel of the vehicle includes determining, by executing at least one instruction with the at least one processor, whether a rearview mirror marker of the vehicle has been detected by a sensor of the passenger device. [17] A physical computer-readable storage medium comprising instructions which, when executed, cause a machine to at least: Determining an expected autonomous maneuver of a vehicle based on collected information corresponding to autonomous operation of the vehicle; Determining whether a passenger device is an augmented reality (AR) device; Determining whether the passenger device is oriented in a direction of travel of the vehicle; in response to determining that the passenger device is not an AR device or that the passenger device is not oriented in the direction of travel of the vehicle, instructing the passenger device to present at least one of haptic feedback and a notification message about the expected autonomous maneuver; and in response to determining that the passenger device is an AR device oriented in the direction of travel of the vehicle: Generating an AR-based visual representation of the expected autonomous maneuver; and Instructing the passenger device to display the AR-based visual representation of the expected autonomous maneuver. [18] The storage medium of claim 17, wherein the expected autonomous maneuver is at least one of an acceleration, a deceleration, a stop, an object avoidance, or a change of direction. [19] The storage medium of claim 17, wherein the instructions, when executed, further cause the machine, in response to determining that the passenger device is an AR device oriented in the direction of travel of the vehicle: Detecting environmental properties outside the vehicle; and Instructing the passenger device to display the AR-based visual representation of the expected autonomous maneuver in spatial relation to the sensed environmental properties. [20] The storage medium of claim 19, wherein the AR device is a wearable AR device and the environmental properties correspond to an environment visible to a passenger wearing the wearable AR device.

Images