Systems and methods for providing platooning information using an augmented reality display

Abstract

A method designed to provide platoon information using a multi-focal plane augmented reality display of a host vehicle. The method includes receiving platoon data from at least one of a plurality of remote vehicles. Each of the plurality of remote vehicles is part of a vehicle platoon. The platoon data includes a position, a trajectory, and an inter-vehicle time distance for each of the plurality of remote vehicles. The method also includes determining, using the platoon data, whether the vehicle platoon is within a predetermined distance from the host vehicle. Further, the method includes sending a command signal to the multi-focal plane augmented reality display of the host vehicle to display a virtual image on the multi-focal plane augmented reality display. The virtual image indicates a platoon action related to the vehicle platoon that is within the predetermined distance from the host vehicle.

Description

Technical Field

[0001] This disclosure relates to a system and method for providing formation information using a multi-focal plane augmented reality display. Background Technology

[0002] This introduction provides an overview of the background of this disclosure. Within the scope described in this introduction, the work of the currently named inventors and aspects of the description that may not constitute prior art at the time of filing are neither explicitly nor implicitly considered prior art to this disclosure.

[0003] Some vehicles include displays to provide information to vehicle users. However, these displays do not necessarily present information related to platooning. In this disclosure, the term "platooning" refers to a method of autonomously driving a group of vehicles together to a common destination. In this disclosure, the term "convoy" refers to multiple remotely located vehicles autonomously driving together to a common destination. Providing platooning information can help vehicle users join or leave a convoy (i.e., a series of vehicles) heading to a specific destination. In this disclosure, the term "platooning information" refers to information about platooning. There is a need to develop a system and method for providing platooning information using multi-focal plane augmented reality displays of vehicles based on real-time vehicle localization. Summary of the Invention

[0004] This disclosure describes a system and method for providing platooning information using a multifocal plane augmented reality display of a master vehicle. In one aspect of this disclosure, the method includes receiving platooning data from at least one of a plurality of remote vehicles. Each of the plurality of remote vehicles is part of a convoy. The platooning data includes the position, trajectory, and time-to-departure distance of each of the plurality of remote vehicles. The method also includes using the platooning data to determine whether the convoy is within a predetermined distance from the master vehicle. The method further includes, in response to determining using the platooning data that the convoy is within a predetermined distance from the master vehicle, sending a command signal to the multifocal plane augmented reality display of the master vehicle to display a virtual image on the multifocal plane augmented reality display. The virtual image indicates platooning actions associated with the convoy within the predetermined distance from the master vehicle. The method described in this paragraph improves vehicle technology by providing the vehicle user of the master vehicle with advance notification of possible platooning actions associated with nearby convoys, while still allowing the vehicle user to have a view of the external world.

[0005] In one aspect of this disclosure, the virtual image indicates that the main vehicle is leading the convoy.

[0006] In one aspect of this disclosure, a virtual image indicates that the main vehicle is joining the convoy.

[0007] In one aspect of this disclosure, the virtual image indicates the final destination of the convoy.

[0008] In one aspect of this disclosure, a virtual image indicates the formation maneuver to be performed by each of a plurality of remote vehicles.

[0009] In one aspect of this disclosure, formation maneuvering is when each of the multiple remote vehicles in a convoy changes lanes.

[0010] In one aspect of this disclosure, formation maneuvering involves the master vehicle reducing the gap between itself and one of the multiple remote vehicles that is directly in front of it.

[0011] In one aspect of this disclosure, formation maneuvers involve the master vehicle increasing the gap between itself and one of the multiple remote vehicles that is directly in front of it.

[0012] In one aspect of this disclosure, the virtual image indicates that the convoy is for private use and that the main vehicle is not allowed to join the convoy.

[0013] In one aspect of this disclosure, the virtual image includes an inquiry asking the vehicle user of the master vehicle whether they wish to join the fleet.

[0014] In one aspect of this disclosure, the virtual image includes an inquiry asking the vehicle user of the main vehicle whether they wish to leave the fleet.

[0015] In one aspect of this disclosure, the virtual image includes text indicating time savings associated with joining the fleet.

[0016] In one aspect of this disclosure, the virtual image includes text indicating cost savings associated with joining a fleet.

[0017] In one aspect of this disclosure, the method further includes determining the position of the eyes of a vehicle user in the host vehicle, and determining the position of the virtual image based on the position of the vehicle user's eyes. Sending a command signal to the multifocal plane augmented reality display includes commanding the multifocal plane augmented reality display of the host vehicle to display the virtual image at the position determined based on the position of the vehicle user's eyes.

[0018] In one aspect of this disclosure, the method further includes receiving vehicle user target data from a vehicle user of a master vehicle. The vehicle user target data includes information about the target of the master vehicle's vehicle user. The method further includes determining whether performing a formation maneuver is consistent with the target of the master vehicle's vehicle user in response to determining, using formation data, that the convoy is within a predetermined distance from the master vehicle. A virtual image indicates the formation maneuver. The method further includes, in response to determining that the formation maneuver is consistent with the vehicle user's target, commanding a multi-focal plane augmented reality display of the master vehicle to display a virtual image. The virtual image includes a query asking the vehicle user whether to perform the formation maneuver.

[0019] In one aspect of this disclosure, the method further includes determining whether the master vehicle has completed formation maneuvers after sending a command signal to the multi-focal plane augmented reality display of the master vehicle to display a virtual image on the multi-focal plane augmented reality display. The method also includes sending a shutdown signal to the multi-focal plane augmented reality display to stop displaying the virtual image on the multi-focal plane augmented reality display in response to determining that the master vehicle has completed formation maneuvers.

[0020] This disclosure also describes a system for providing platooning information in a master vehicle. In one aspect of this disclosure, the system includes a transceiver configured to receive platooning data from at least one of a plurality of remote vehicles. Each of the plurality of remote vehicles is part of a convoy, and the platooning data includes the position, trajectory, and time-to-departure distance of each of the plurality of remote vehicles. The system also includes a multifocal plane augmented reality display configured to display a virtual image. The system also includes a controller communicating with the transceiver and the multifocal plane augmented reality display. The controller is configured to: receive the platooning data; use the platooning data to determine whether the convoy is within a predetermined distance from the master vehicle; and, in response to determining that the convoy is within a predetermined distance from the master vehicle using the platooning data, send a command signal to the multifocal plane augmented reality display of the master vehicle to display a virtual image on the multifocal plane augmented reality display, wherein the virtual image indicates the convoy. The system described in this paragraph improves vehicle technology by providing the vehicle user of the master vehicle with advance notification of possible platooning maneuvers related to nearby convoys, while still allowing the vehicle user to have a view of the outside world.

[0021] In one aspect of this disclosure, the system also includes a user tracker configured to track the position of the eyes of a vehicle user of a master vehicle. The controller is configured to: determine the position of the eyes of the vehicle user of the master vehicle; determine the position of a virtual image based on the position of the vehicle user's eyes; and command a multi-focal-plane augmented reality display of the master vehicle to display the virtual image at the position determined based on the position of the vehicle user's eyes.

[0022] In one aspect of this disclosure, the system also includes a user interface. The controller is configured to receive vehicle user target data from a vehicle user of the master vehicle via the user interface. The vehicle user target data includes information about the target of the vehicle user of the master vehicle. The controller is further configured to determine whether performing a formation maneuver is consistent with the vehicle user's target, in response to determining using formation data that the convoy is within a predetermined distance from the master vehicle, wherein a virtual image indicates the formation maneuver. The controller is further configured to command the master vehicle's multi-focal plane augmented reality display to display a virtual image in response to determining that the formation maneuver is consistent with the target of the vehicle user of the master vehicle. The virtual image includes a query asking the vehicle user whether to perform the formation maneuver.

[0023] In one aspect of this disclosure, the controller is configured to: after sending a command signal to the multi-focal plane augmented reality display of the master vehicle to display a virtual image on the multi-focal plane augmented reality display, determine whether the master vehicle has completed a formation maneuver. The controller is also configured to: in response to determining that the master vehicle has completed a formation maneuver, send a shutdown signal to the multi-focal plane augmented reality display to stop displaying the virtual image on the multi-focal plane augmented reality display.

[0024] Further applicability of this disclosure will become apparent from the detailed description provided below. It should be understood that the detailed description and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0025] The above features and advantages, as well as other features and advantages, of the systems and methods disclosed herein will be apparent from the detailed description including the claims and exemplary embodiments, taken in conjunction with the accompanying drawings. Attached Figure Description

[0026] This disclosure will be more fully understood from the detailed description and accompanying drawings, in which:

[0027] Figure 1 This is a block diagram depicting an embodiment of a vehicle including a system that uses a multi-focal plane augmented reality display to provide formation information;

[0028] Figure 2 yes Figure 1 A schematic front view of a multifocal plane augmented reality display, in which the multifocal plane augmented reality display presents text asking the vehicle user whether to join the convoy;

[0029] Figure 3 yes Figure 1 A schematic front view of a multifocal plane augmented reality display, in which the multifocal plane augmented reality display presents text asking the vehicle user whether to leave the convoy;

[0030] Figure 4 yes Figure 1 A schematic front view of a multi-focal plane augmented reality display, in which the multi-focal plane augmented reality display presents information indicating that the main vehicle is part of a convoy heading to a specific destination;

[0031] Figure 5 yes Figure 1 A schematic front view of a multi-focal plane augmented reality display, in which the multi-focal plane augmented reality display presents information indicating that the main vehicle is leading a convoy toward a specific destination;

[0032] Figure 6 yes Figure 1A schematic front view of a multi-focal plane augmented reality display, in which the multi-focal plane augmented reality display presents information indicating that the main vehicle can save time if it joins the convoy;

[0033] Figure 7 yes Figure 1 A schematic front view of a multi-focal plane augmented reality display, in which the multi-focal plane augmented reality display presents information indicating that the main vehicle will benefit from cost savings if it joins the convoy;

[0034] Figure 8 yes Figure 1 A schematic front view of a multifocal plane augmented reality display, in which the multifocal plane augmented reality display presents information indicating that the convoy is for private use and that the main vehicle is not allowed to join the convoy;

[0035] Figure 9 yes Figure 1 A schematic front view of a multi-focal plane augmented reality display, in which the multi-focal plane augmented reality display presents information instructing the master vehicle to reduce the gap between the platoon vehicles and the master vehicle;

[0036] Figure 10 yes Figure 1 A schematic front view of a multi-focal plane augmented reality display, in which the multi-focal plane augmented reality display presents information indicating that the master vehicle will increase the gap between the platoon vehicles and the master vehicle;

[0037] Figure 11 yes Figure 1 A schematic front view of a multi-focal plane augmented reality display, in which the multi-focal plane augmented reality display presents information indicating that a convoy of vehicles, including the main vehicle, will change lanes;

[0038] Figure 12 yes Figure 1 A schematic front view of a multi-focal plane augmented reality display, in which the multi-focal plane augmented reality display presents information indicating that a convoy excluding the main vehicle will change lanes;

[0039] Figure 13 yes Figure 1 A schematic front view of a dual-focal-plane augmented reality display, highlighting Figure 1 The second image plane of a dual-focal-plane augmented reality display;

[0040] Figure 14 yes Figure 1 A schematic diagram of the second image plane of a dual-focal-plane augmented reality display;

[0041] Figure 15 It is used in Figure 1A schematic diagram of a part of a system that displays lane information on a dual-focal-plane augmented reality display; and

[0042] Figure 16 This is a flowchart of a method for providing formation information using a multi-focal plane augmented reality display. Detailed Implementation

[0043] Reference will now be made in detail to several examples of this disclosure shown in the accompanying drawings. Wherever possible, the same or similar reference numerals are used in the drawings and description to denote the same or similar parts or steps.

[0044] Reference Figure 1 The main vehicle 10 generally includes a chassis 12, a body 14, front wheels, and rear wheels 17, and may be referred to as a vehicle system. In the depicted embodiment, the main vehicle 10 includes two front wheels 17a and two rear wheels 17b. The body 14 is mounted on the chassis 12 and substantially surrounds the components of the main vehicle 10. The body 14 and the chassis 12 may together form a frame. Each wheel 17 is rotatably coupled to the chassis 12 near a corresponding corner of the body 14. The main vehicle 10 includes a front axle 19 coupled to the front wheels 17a and a rear axle 25 coupled to the rear wheels 17b.

[0045] In several different embodiments, the master vehicle 10 may be an autonomous (or driverless) vehicle, and a control system 98 is integrated into the master vehicle 10. The control system 98 may be referred to as a system or a system using one or more displays 29 (e.g., multi-focal plane augmented reality displays) to provide platooning information. The master vehicle 10 is, for example, a vehicle automatically controlled to transport passengers from one location to another. In the illustrated embodiment, the master vehicle 10 is depicted as a pickup truck, but it should be understood that other vehicles may also be used, including trucks, sedans, two-door cars, sports utility vehicles (SUVs), recreational vehicles (RVs), etc. In embodiments, the master vehicle 10 may be a so-called Level 2, Level 3, Level 4, or Level 5 automation system. A Level 4 system signifies “high automation,” referring to the autonomous driving system performing aspects of dynamic driving tasks in a specific driving mode, even if the driver does not respond appropriately to intervention requests. A Level 5 system signifies “full automation,” referring to the autonomous driving system performing aspects of dynamic driving tasks continuously under a variety of road and environmental conditions that can be managed by a driver. In Level 3 vehicles, the vehicle system performs the entire Dynamic Driving Task (DDT) within its designed area. The driver is only expected to disengage from DDT if a problem arises or the vehicle is about to leave its operational area, thus substantially "requesting" driver intervention from the primary vehicle. In Level 2 vehicles, the system provides steering, braking / acceleration support, lane centering, and adaptive cruise control. However, even with these systems enabled, the driver in the driver's seat must remain in the driver's seat and continuously monitor the automated features.

[0046] As shown, the main vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34, and a communication system 36. In several different embodiments, the propulsion system 20 may include an electric motor (e.g., a traction motor) and / or a fuel cell propulsion system. The main vehicle 10 may also include a battery (or battery pack) 21 electrically connected to the propulsion system 20. Thus, the battery 21 is configured to store electrical energy and supply electrical energy to the propulsion system 20. In some embodiments, the propulsion system 20 may include an internal combustion engine. The transmission system 22 is configured to transmit power from the propulsion system 20 to the vehicle wheels 17 according to a selectable speed ratio. According to several different embodiments, the transmission system 22 may include a stepped automatic transmission, a continuously variable transmission (CVT), or other suitable transmission. The braking system 26 is configured to provide braking torque to the vehicle wheels 17. In several different embodiments, the braking system 26 may include friction brakes, brake-by-wire brakes, regenerative braking systems (e.g., electric motors), and / or other suitable braking systems. The steering system 24 is used to intervene in the attitude of the vehicle wheels 17 and may include a steering wheel 33. Although the steering system 24 is depicted as including a steering wheel 33 for illustrative purposes, in some embodiments contemplated within the scope of this disclosure, the steering system 24 may not include a steering wheel 33.

[0047] Sensor system 28 includes one or more sensors 40 (i.e., sensing devices) for sensing observable conditions of the external and / or internal environment of the host vehicle 10. Sensors 40 communicate with controller 34, and sensors may include, but are not limited to, one or more radars, one or more lidar sensors, one or more proximity sensors, one or more odometers, one or more ground-penetrating radar (GPR) sensors, one or more steering angle sensors, one or more Global Positioning System (GPS) transceivers 45, one or more tire pressure sensors, one or more cameras 41 (e.g., optical cameras and / or infrared cameras), one or more gyroscopes, one or more accelerometers, one or more inclinometers, one or more speed sensors, one or more ultrasonic sensors, one or more inertial measurement units (IMUs), and / or other sensors. Each sensor 40 is configured to generate a signal indicative of the sensed observable conditions of the external and / or internal environment of the host vehicle 10. Because sensor system 28 provides data to controller 34, sensor system 28 and its sensors 40 are considered a source of information (or simply a source).

[0048] Sensor system 28 includes one or more Global Navigation Satellite System (GNSS) transceivers 45 (e.g., Global Positioning System (GPS) transceivers) configured to detect and monitor route data (i.e., route information). GNSS transceivers 45 are configured to communicate with GNSS to locate the position of the main vehicle 10 globally. GNSS transceivers 45 communicate electronically with controller 34.

[0049] The actuator system 30 includes one or more actuator devices 42 that control one or more vehicle features, such as, but not limited to, the propulsion system 20, the transmission system 22, the steering system 24, and the braking system 26. In several different embodiments, the vehicle features may also include interior and / or exterior vehicle features, such as, but not limited to, doors, trunk, and cabin features (e.g., air, music, lighting, etc.).

[0050] Data storage device 32 stores data for automatically controlling the master vehicle 10. In several different embodiments, data storage device 32 stores a defined map of the navigable environment. In several different embodiments, the defined map may be predefined by and obtained from a remote system. For example, the defined map may be assembled by a remote system and transmitted to the master vehicle 10 (wirelessly and / or via wire) and stored in data storage device 32. Data storage device 32 may be part of controller 34, separate from controller 34, or part of controller 34 and a separate system.

[0051] The main vehicle 10 may also include one or more airbags 35 communicating with the controller 34 or another controller of the main vehicle 10. The airbags 35 include inflatable airbags and are configured to switch between a retracted configuration and a deployed configuration to cushion the effects of external forces applied to the main vehicle 10. Sensors 40 may include airbag sensors, such as an IMU, configured to detect external forces and generate signals indicating the magnitude of those forces. The controller 34 is configured to command the airbags 35 to deploy based on signals from one or more sensors 40 (e.g., airbag sensors). Therefore, the controller 34 is configured to determine when the airbags 35 deploy.

[0052] The controller 34 includes at least one processor 44 and a non-transitory computer-readable storage device or medium 46. The processor 44 may be a custom processor or a commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among multiple processors associated with the controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a means for executing instructions. The computer-readable storage device or medium 46 may include volatile and non-volatile storage, such as read-only memory (ROM), random access memory (RAM), and persistent memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operational variables when the processor 44 is powered off. The computer-readable storage device or medium 46 may be implemented using multiple memory devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or other electrical, magnetic, optical, or combined memory devices capable of storing data, some of which represents executable instructions used by the controller 34 in controlling the master vehicle 10. The controller 34 of the main vehicle 10 can be referred to as the vehicle controller and can be programmed to execute method 100 as described in detail below. Figure 16 ).

[0053] The instructions may include one or more separate programs, each including an ordered list of executable instructions for implementing logical functions. When executed by processor 44, the instructions cause processor 44 to receive and process signals from sensor system 28; execute logic, calculations, methods, and / or algorithms for automatically controlling components of master vehicle 10; and generate control signals to actuator system 30 to automatically control components of master vehicle 10 based on logic, calculations, methods, and / or algorithms. Although Figure 1 A single controller 34 is shown, but embodiments of the master vehicle 10 may include multiple controllers 34 that communicate via a suitable communication medium or a combination of communication media and cooperate to process sensor signals, perform logic, calculations, methods and / or algorithms, and generate control signals to automatically control the features of the master vehicle 10.

[0054] In several different embodiments, one or more instructions of controller 34 are included in control system 98. The host vehicle 10 includes a user interface 23, which may be a touchscreen in the dashboard. User interface 23 may include, but is not limited to, alarms, such as one or more speakers 27 for providing audible sound, tactile feedback from vehicle seats or other objects; one or more displays 29; one or more microphones 31; and / or other means adapted to provide notifications to vehicle users of the host vehicle 10. User interface 23 communicates electronically with controller 34, and the user interface is configured to receive input from a user (e.g., a vehicle operator or vehicle passenger). For example, user interface 23 may include configuration to receive input from vehicle user 11 (… Figure 6 The input can be made via a touchscreen and / or buttons. Therefore, the controller 34 is configured to receive input from the user via the user interface 23.

[0055] The main vehicle 10 may include one or more displays 29 configured to display information to a user (e.g., a vehicle operator or passenger) and may be an augmented reality (AR) display. In this disclosure, the term "AR display" refers to a display that presents information to a user while also allowing them to see the outside world. In some embodiments, the display 29 may be configured as a head-up display (HUD) and / or an information cluster display. Thus, the display 29 may be an AR HUD or an AR information cluster display. In an AR HUD, an image is projected onto the windshield 39 of the main vehicle 10. Figure 15 On the AR information cluster display, the information cluster panel of the main vehicle 10 includes a screen that displays an image (i.e., a forward-facing image) captured by one or more cameras 41 in front of the main vehicle 10 and a virtual image presented on the forward-facing image. As discussed below, the display 29 may be a multi-focal plane AR display to facilitate manipulation of the virtual image (e.g., size, position, and type).

[0056] The communication system 36 communicates with the controller 34 and is configured to wirelessly transmit information to and from other remote vehicles 48, including but not limited to other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems at remote call centers (e.g., on-star of general motors), and / or personal electronic devices (e.g., mobile phones). In this disclosure, the term “remote vehicle” refers to a vehicle configured to send one or more signals to the host vehicle 10 without being physically connected to the host vehicle 10, such as an automobile. In some embodiments, the communication system 36 is a wireless communication system configured to communicate via a wireless local area network (WLAN) employing the IEEE 802.11 standard or by using cellular data communication. However, additional or alternative communication methods, such as Dedicated Short Range Communication (DSRC) channels, are also considered within the scope of this disclosure. A DSRC channel refers to a one-way or two-way short-to-medium range wireless communication channel specifically designed for automotive use and a corresponding set of protocols and standards. Therefore, the communication system 36 may include one or more antennas and / or communication transceivers 37 for receiving and / or transmitting signals (e.g., cooperative sensing messages (CSM)). The communication transceiver 37 may be considered as sensor 40. The communication system 36 is configured to wirelessly transmit information between the host vehicle 10 and another vehicle. Further, the communication system 36 is configured to wirelessly transmit information between the host vehicle 10 and infrastructure or other vehicles.

[0057] Reference Figure 1 and Figure 2 System 98 is configured to command display 29 (e.g., AR HUD, AR information cluster display, and / or information display) to present platooning information based on data received from other remote vehicles 48 via communication system 36 (e.g., V2V communication) and / or eye-tracking data. In this process, display 29 contextually displays lane information by utilizing conformal graphics to enhance the road scene. In this disclosure, the term "conformal graphics" refers to synthetically generated content (i.e., virtual image 50) presented as part of the external world. Therefore, display 29 is a conformal display. In this disclosure, the term "conformal display" refers to a display capable of representing synthetically generated content (i.e., one or more virtual images 50) as part of the external world.

[0058] Reference Figure 2The display 29 can be configured as a head-up display (HUD) and present a virtual image 50 with information about the convoy 49 of remote vehicles 48. In the depicted example, the display 29 shows a first virtual image 50a on the road surface 62, which spans (and thus highlights) the entire length of the convoy 49. The first virtual image 50a thus highlights the remote vehicles 48 that are part of the convoy 49 and may include text 51 indicating the final destination of the convoy 49 (e.g., Chicago). The display 29 may additionally present a second virtual image 50b including an arrow 53 and additional text 51 adjacent to the arrow 53. The arrow 53 points to the convoy 49 (i.e., to the first virtual image 50a), and the text 51 adjacent to the arrow 53 is an inquiry asking the vehicle user 11 of the main vehicle 10 whether they wish to join the convoy 49 (e.g., “Join the convoy?”).

[0059] exist Figure 3 In the depicted embodiment, a first virtual image 50a generated by display 29 on road surface 62 highlights remote vehicles 48 that are part of convoy 49 and includes text 51 indicating the final destination of convoy 49 (e.g., Chicago). A second virtual image 50b includes arrows 53 pointing to the lane directly adjacent to convoy 49 (i.e., adjacent lane 55). Additionally, the second virtual image 50b includes text 51, which is an inquiry asking the vehicle user 11 of the main vehicle 10 whether they wish to exit convoy 49 (e.g., “Exit convoy?”) and move to adjacent lane 55.

[0060] exist Figure 4 In the depicted embodiment, display 29 presents one or more virtual images 50, such as a first virtual image 50a. The first virtual image 50a spans at least a portion of the length of the convoy 49. In this case, the master vehicle 10 is part of the convoy 49, and the first virtual image 50a includes text 51 indicating the final destination of the convoy 49 (e.g., Chicago). In this case, the master vehicle 10 is part of the convoy 49 but does not lead it. For this purpose, the first virtual image 50a has a rectangular shape. In other words, the first virtual image 50a can have a rectangular shape to indicate that the master vehicle 10 is part of the convoy 49 but does not lead it.

[0061] exist Figure 5In the depicted embodiment, display 29 presents one or more virtual images 50, such as a first virtual image 50a. The first virtual image 50a is displayed in front of the main vehicle 10 and in front of the convoy 49. In this case, the main vehicle 10 is part of the convoy 49, and the first virtual image 50a includes text 51 indicating the final destination of the convoy 49 (e.g., Kansas City). In this case, the main vehicle 10 is part of and leads the convoy 49. For this purpose, the first virtual image 50a has a triangular shape. In other words, the first virtual image 50a may have a triangular shape to indicate that the main vehicle 10 is part of and leads the convoy 49.

[0062] exist Figure 6 In the depicted embodiment, display 29 shows a first virtual image 50a on road surface 62, which spans (and thus highlights) the entire length of convoy 49. The first virtual image 50a thus highlights remote vehicles 48 that are part of convoy 49 and may include text 51 indicating the final destination of convoy 49 (e.g., Chicago). Display 29 may additionally present a second virtual image 50b including an arrow 53 and additional text 51 adjacent to the arrow 53. Arrow 53 points to convoy 49 (i.e., to the first virtual image 50a), and the text 51 adjacent to the arrow 53 includes: (1) a time saving (e.g., 00:17 minutes) to the final destination if the main vehicle 10 joins convoy 49; and (2) an inquiry to the vehicle user 11 of the main vehicle 10 whether they wish to join convoy 49 (e.g., “Join the convoy?”).

[0063] exist Figure 7 In the depicted embodiment, display 29 shows a first virtual image 50a on road surface 62, which spans (and thus highlights) the entire length of convoy 49. The first virtual image 50a thus highlights remote vehicles 48 that are part of convoy 49 and may include text 51 indicating the final destination of convoy 49 (e.g., Chicago). Display 29 may additionally present a second virtual image 50b including an arrow 53 and additional text 51 adjacent to the arrow 53. Arrow 53 points to convoy 49 (i.e., to the first virtual image 50a), and the text 51 adjacent to the arrow 53 includes: (1) a cost saving (e.g., $6.48) to the final destination if the primary vehicle 10 joins convoy 49; and (2) an inquiry asking the vehicle user of primary vehicle 10 whether they wish to join convoy 49 (e.g., “Join the convoy?”).

[0064] exist Figure 8In the depicted embodiment, the display 29 shows a first virtual image 50a on the road surface 62, which spans (and thus highlights) the entire length of the convoy 49. Therefore, the first virtual image 50a highlights the remote vehicles 48 that are part of the convoy 49 and may have a bright color (e.g., red) to indicate that the convoy 49 is private and that the main vehicle 10 is not allowed to join the convoy 49.

[0065] exist Figure 9 In the depicted embodiment, display 29 shows a third virtual image 50c on road surface 62, positioned between the main vehicle 10 and a remote vehicle 48 directly in front of the main vehicle. The third virtual image 50c may have a rectangular block highlighting the area between the main vehicle 10 and the remote vehicle 48 directly in front of the main vehicle 10, and the third virtual image 50c includes an arrow 53 within the highlighted area pointing to the remote vehicle 48 directly in front of the main vehicle 10. The third virtual image 50c also includes text 51 (e.g., "Narrowing the gap") indicating that the main vehicle 10 will narrow the gap between itself and the remote vehicle 48 directly in front of it due to another remote vehicle 48 exiting the convoy 49.

[0066] exist Figure 10 In the depicted embodiment, display 29 shows a third virtual image 50c on road surface 62, positioned between the main vehicle 10 and a remote vehicle 48 directly in front of the main vehicle. The third virtual image 50c may have a rectangular block highlighting the area between the main vehicle 10 and the remote vehicle 48 directly in front of the main vehicle 10, and the third virtual image 50c includes an arrow 53 pointing to the main vehicle 10 within the highlighted area. The third virtual image 50c also includes text 51 (e.g., "Expand Gap") instructing the main vehicle 10 to widen the gap between itself and the remote vehicle 48 directly in front of it to allow another remote vehicle 48 to enter the convoy 49.

[0067] exist Figure 11 In the depicted embodiment, the display 29 shows a virtual image 50 on the road surface 62. The second virtual image 50b includes an arrow 53 pointing to the lane directly adjacent to the convoy 49 (i.e., adjacent lane 55). Additionally, the virtual image 50 includes text 51 (e.g., “Pulley Lane Change”) indicating that the convoy 49, including the main vehicle 10, will move to the adjacent lane 55.

[0068] exist Figure 12In the depicted embodiment, the display 29 shows a virtual image 50 on the road surface 62. The second virtual image 50b includes an arrow 53 pointing to the lane directly adjacent to the convoy 49 (i.e., adjacent lane 55). Additionally, the virtual image 50 includes text 51 (e.g., “Pulley Lane Change”) indicating that the convoy 49, excluding the main vehicle 10, will move to the adjacent lane 55.

[0069] Reference Figure 13 and Figure 14 The display 29 can be a multifocal plane AR display as mentioned above. In this case, the display 29 has a first image plane 58 and a second image plane 60. The first image plane 58 shows a view of the external world, and the second image plane 60 is reserved for displaying the virtual image 50. Figure 2 The second image plane 60 spans multiple lanes, and the virtual image 50 appears at a greater distance on the road surface 62 relative to the first image plane 58. For example, as... Figure 14 As shown, the second image plane 60 covers the left lane 52, the center lane 54, and the right lane 56. As a non-limiting example, in the center lane 54, the second image plane 60 begins at a first predetermined distance D1 (e.g., 25 meters) from the main vehicle 10 and terminates at a second predetermined distance D2 (e.g., 90 meters) from the main vehicle 10. Regardless of the specific distance, the second predetermined distance D2 is greater than the first predetermined distance D1 to assist vehicle user 11 ( Figure 15 )See virtual image 50 ( Figure 3 In left lane 52 and right lane 56, the second image plane 60 is defined by an inclined boundary that begins at a first predetermined distance D1 from the main vehicle 10 and ends at a third predetermined distance D3 (e.g., fifty meters) from the main vehicle 10. The third predetermined distance D3 is greater than the first predetermined distance D1 and less than the second predetermined distance D2 to assist vehicle user 11. Figure 6 )See virtual image 50 ( Figure 3 As used herein, the term "multi-focal-plane AR display" refers to an AR display that presents images in more than one image plane, wherein the image planes are located at different positions. Ideally, a multi-focal-plane AR display is used in the system 98 of this disclosure to easily change the size, type, and / or position of the virtual image 50 relative to a view of the external world.

[0070] Reference Figure 15System 98 includes a user tracker 43 (e.g., an eye tracker and / or a head tracker) configured to track the position and movement of the eyes 66 or head 69 of the vehicle user 11. In the depicted embodiment, the user tracker may be configured with one or more cameras 41 of the main vehicle 10. As discussed above, the camera 41 is considered a sensor 40 of the main vehicle 10. As a sensor 40, the user tracker 43 communicates with a controller 34 including a system manager 68. During operation of system 98, system manager 68 receives at least a first input 70 and a second input 72. The first input 70 indicates the vehicle's position in space (i.e., the vehicle's spatial position), and the second input 72 indicates the vehicle user's position within the main vehicle 10 (e.g., the position of the user's eyes and / or head within the main vehicle 10). The first input 70 may include data such as GNSS data (e.g., GPS data), vehicle speed, road curvature, and vehicle steering, and may be collected from the sensors 40 of the main vehicle 10 and / or other remote vehicles 48 via the communication system 36 of the main vehicle 10. A second input 72 may be received from a user tracker (e.g., an eye tracker and / or a head tracker). The system manager 68 is configured to determine (e.g., calculate) the type, size, shape, and color of the conformal graphic (i.e., the virtual image 50) based on the first input 70 (i.e., the vehicle's spatial location), the second input 72 (e.g., the position of the user's eyes and / or head within the main vehicle 10), and the sensed vehicle driving environment (which may be obtained through sensor 40). The type, size, shape, and color of the conformal graphic of the virtual image 50 can be collectively referred to as virtual image characteristics.

[0071] Continue to refer to Figure 15 The system 98 also includes an image engine 74, which is part of the display 29 and may be an integrated circuit configured to generate virtual images 50. These generated virtual images 50 are then projected onto the windshield 39 (if the display 29 is a HUD) to be displayed on a second image plane 60 along the road surface 62.

[0072] Figure 16 This is a flowchart of a method 100 for providing formation information using a display 29 (such as a multi-focal plane augmented reality display and / or an information display). Method 100 begins at block 102, where controller 34 uses signals generated, for example, by sensors 40, to determine that the master vehicle 10 is being driven. For example, controller 34 may receive data from one of the sensors 40 (e.g., a speed sensor) to determine that the master vehicle 10 is moving. Method 100 then proceeds to block 104.

[0073] In block 104, controller 34 receives platooning data from remote vehicles 48, such as those part of platoon 49. As discussed above, the term "platooning" refers to a method of autonomously driving a group of vehicles together to a common destination. The term "platooning data" refers to data about the remote vehicles 48 that are part of platoon 49 and about platoon 49 itself. In some embodiments, platooning data includes the movement, speed, trajectory, inter-vehicle time difference, acceleration, position relative to the master vehicle 10, and destination of one or more remote vehicles 48 that are part of platoon 49. To obtain platooning data, controller 34 may use, for example, V2V communication via communication transceiver 37 to receive at least one signal from one or more sensors 40 (e.g., camera 41, GNSS transceiver 45 (e.g., Global Positioning System (GPS) transceiver)), data from the Internet, road databases, and / or data from other remote vehicles 48 (e.g., remote vehicle 48). Therefore, controller 34 can use platooning data received, for example, from GNSS transceiver 45, other sensors 40, or other suitable information sources (e.g., remote vehicle 48) to determine the position, movement, time interval, and trajectory of remote vehicles 48 that are part of platoon 49. At block 104, controller 34 can also use at least one signal from one or more sensors 40 (e.g., camera 41, GNSS transceiver 45 (e.g., a Global Positioning System (GPS) transceiver)), data from the Internet, road databases, and / or data from other remote vehicles 48 (e.g., remote vehicle 48) via communications transceiver 37 to determine the position of master vehicle 10 relative to remote vehicles 48 that are part of platoon 49. Method 100 then proceeds to block 106.

[0074] At box 106, controller 34 uses platooning data to determine whether platoon 49 is within a predetermined distance of the host vehicle 10. This predetermined distance can be determined by testing the host vehicle 10. In some embodiments, the predetermined distance may be twenty feet to allow a vehicle user 11 with 20 / 20 vision to perform an action (e.g., join platoon 49) when the host vehicle 10 is within twenty feet of platoon 49. If the distance from the host vehicle 10 to platoon 49 is greater than the predetermined distance, method 100 proceeds to box 108. At box 108, controller 34 does not perform an action, and method 100 returns to box 104. If the distance from the host vehicle 10 to platoon 49 is equal to or less than the predetermined distance, method 100 continues to box 110.

[0075] At box 110, controller 34 determines whether the objective of vehicle user 11 of master vehicle 10 is consistent with (or aligned with) formation actions associated with one or more convoys 49 within a predetermined distance from master vehicle 10. For this purpose, controller 34 may receive user preferences or objectives input by vehicle user 11 via user interface 23. In other words, at box 110, controller 34 receives vehicle user objective data from vehicle user 11 of master vehicle 10. In this disclosure, the term "vehicle user objective data" refers to data concerning the objective of vehicle user 11 of master vehicle 10, where the objective relates to one or more convoys 49 within a predetermined distance from master vehicle 10. As a non-limiting example, the objective of vehicle user 11 may include: reaching a specific final destination (e.g., Chicago); reaching a specific final destination in the most time-efficient manner; and reaching a specific final destination while minimizing costs (e.g., saving fuel). At box 110, controller 34 determines whether the objective of vehicle user 11 of master vehicle 10 is consistent with (or aligned with) formation actions associated with one or more convoys 49 within a predetermined distance from master vehicle 10. As a non-limiting example, the final destination of convoy 49 within a predetermined distance from the main vehicle 10 could be city A (e.g., Chicago), and the final destination of vehicle user 11 could be city A or another city along the route taken by convoy 49 to city A. In this case, controller 34 determines that joining convoy 49 heading to city A (i.e., platooning) aligns with the vehicle user's objective. In another example, vehicle user 11 could input their objective as arriving in city A in the shortest time or at the lowest cost. If controller 34 determines that joining convoy 49 heading to city A (i.e., platooning) is the most time-efficient or the least costly (depending on vehicle user 11's preference), then controller 34 determines that joining convoy 49 (i.e., platooning) aligns with the objective of vehicle user 11 of main vehicle 10. If controller 34 determines that one or more platooning actions associated with one or more convoys 49 within a predetermined distance from main vehicle 10 do not align with the objective of vehicle user 11 of main vehicle 10, then method 100 returns to box 108. If the controller 34 determines that the target of the vehicle user 11 of the master vehicle 10 is consistent with (or matches) the formation action associated with one or more convoys 49 within a predetermined distance from the master vehicle 10, then method 100 returns to box 112.

[0076] In block 112, controller 34 determines the location (or position) of master vehicle 10 relative to a convoy 49 aligned with the target of vehicle user 11. For this purpose, controller 34 may use data received from GNSS transceiver 45 and / or road data, such as a road database. In other words, at block 112, controller 34 determines the position of master vehicle 10 relative to one or more remote vehicles 48 that are part of convoy 49, which are within a predetermined distance of master vehicle 10 and aligned with the target of vehicle user 11 (e.g., convoy 49 heading to the same final destination as master vehicle 10, saving costs, saving time, etc.). Method 100 then continues to block 114.

[0077] At box 114, controller 34 determines one or more virtual images 50 to be displayed based on the movement, position, time interval and trajectory of convoy 49, the target of vehicle user 11 of master vehicle 10, and the position of master vehicle 10 relative to convoy 49. As a non-limiting example, virtual image 50 may indicate: master vehicle 10 is leading convoy 49; master vehicle 10 is joining convoy 49; the final destination of convoy 49; convoy 49 is private and therefore master vehicle 10 is not allowed to join convoy 49; each of the remote vehicles 48 that are part of convoy 49 will change lanes; master vehicle 10 that is part of convoy 49 will widen the gap between master vehicle 10 and the remote vehicle 48 that is part of convoy 49 and is directly in front of master vehicle 10 to allow another remote vehicle 48 that is not part of convoy 49 to join convoy 49; master vehicle 10 that is part of convoy 49 will narrow the gap between master vehicle 10 and the remote vehicle 48 that is part of convoy 49 and is directly in front of master vehicle 10 after another remote vehicle 48 leaves convoy 49; and / or an inquiry to master vehicle 10's vehicle user 11 asking if vehicle user 11 wants to join convoy 49; time savings achieved by joining convoy 49; cost savings achieved by joining convoy 49; and / or other convoy actions. In this disclosure, the term "platooning action" refers to the movement (or position) of one or more remote vehicles 48 that are part of platoon 49; the current or future movement (or position) of master vehicle 10 when master vehicle 10 is part of platoon 49; the current or future movement of master vehicle 10 relative to platoon 49 when master vehicle 10 is within a predetermined distance from platoon 49 and master vehicle 10 is not part of platoon 49; the final destination of platoon 49; cost or time savings achieved if master vehicle 10 joins platoon 49; and / or limitations regarding platoon 49, such as whether platoon 49 is for private use.

[0078] At block 116, controller 34 uses at least one input from user tracker 43 to determine the position of the user's eyes 66 and / or head 69 in the main vehicle 10. As discussed above, user tracker 43 may be a camera 41 configured to track the movement of the vehicle user's head 69 and / or eyes 66. Controller 34 then uses the input from user tracker 43 to continuously determine the position of the user's eyes 66 and / or head 69 in real time. Method 100 then proceeds to block 118.

[0079] At box 118, controller 34 determines in real time the virtual image 50 to be displayed on display 29 (e.g., a multi-focal plane augmented reality display) based on the position of the user's eyes 66 and / or head 69 in the master vehicle 10, platooning data, and / or the position of the master vehicle 10 relative to the platoon 49. Figure 2 The position, type, size, shape, and color of the virtual image 50. As a non-limiting example, the position of the virtual image 50 in the display 29 may change as the vehicle user 11 moves their head 69 and / or eyes 66. Furthermore, the size of the virtual image 50 may increase as the main vehicle 10 moves closer to the convoy 49. Next, method 100 proceeds to box 120.

[0080] At frame 120, controller 34 sends a command signal to instruct display 29 to present at least one virtual image 50 at a location previously determined at frame 118. As discussed above, the virtual image 50 indicates one or more formation maneuvers. The position of the virtual image 50 may change in real time based on the position of the vehicle user's eyes 66 and / or head 69. In response to receiving the command signal from controller 34, display 29 shows the virtual image 50 at a previously determined location with a previously determined size.

[0081] At box 122, controller 34 determines whether the master vehicle 10 has completed a formation maneuver (e.g., entering platoon 49) as presented as a virtual image 50 on display 29. To do this, controller 34 determines that sensor data can be received from sensor 40 (e.g., camera 41) and uses the sensor data to determine whether the master vehicle 10 has completed a formation maneuver (e.g., entering platoon 49, exiting platoon 49, etc.). If the formation maneuver has not been completed, method 100 returns to box 112. If the formation maneuver has been completed, method 100 proceeds to box 124. Controller 34 also sends a shutdown signal to display 29 to stop displaying the virtual image 50 indicating a completed formation maneuver.

[0082] At box 124, controller 34 uses, for example, V2V communication via communication transceiver 37 to receive updates from sensor 40, user tracker 43, GNSS transceiver 45 (e.g., Global Positioning System (GPS) transceiver), data from the Internet, road database, and / or data from other remote vehicles 48 (e.g., remote vehicle 48). These updates may relate to the movement of the master vehicle 10 and / or convoy 49. Method 100 then continues to box 126.

[0083] At box 126, controller 34 determines whether vehicle user 11 has requested to leave fleet 49. To do this, controller 34 may rely on input from vehicle user 11. For example, vehicle user 11 may request controller 34 to leave fleet 49 via user interface 23. Method 100 then continues to box 128.

[0084] At box 128, controller 34 presents a virtual image 50 including a query. This query asks vehicle user 11 of master vehicle 10 whether the vehicle user wants to leave convoy 49. If vehicle user 11 answers affirmatively (via user interface 23), master vehicle 10 can autonomously leave convoy 49.

[0085] While exemplary embodiments have been described above, it is not intended that these embodiments describe all possible forms included in the claims. The language used in this specification is descriptive rather than restrictive, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of multiple different embodiments may be combined to form further embodiments of the systems and methods of this disclosure that may not be explicitly described or shown. While multiple different embodiments may have been described as providing advantages or being preferred in terms of one or more desired characteristics relative to other embodiments or prior art implementations, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, etc. Therefore, embodiments described as less desirable in terms of one or more characteristics compared to other embodiments or prior art implementations do not exceed the scope of this disclosure and may be ideal for a particular application.

[0086] The accompanying drawings are simplified and not to exact scale. For convenience and clarity only, directional terms such as top, bottom, left, right, upper, above, above, below, behind, and front may be used relative to the drawings. These and similar directional terms should not be construed as limiting the scope of this disclosure in any way.

[0087] This document describes embodiments of the present disclosure. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take many different and alternative forms. The drawings are not necessarily to scale; some features may be enlarged or reduced to show details of specific components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but rather as a representative basis for teaching those skilled in the art to employ the systems and methods of this disclosure in a variety of different ways. As will be understood by those skilled in the art, the various features shown and described with reference to any of the drawings may be combined with features shown in one or more other drawings to produce embodiments not explicitly shown or described. The combinations of features shown provide representative embodiments of typical applications. However, various combinations and modifications of features consistent with the teachings of this disclosure may be desirable for a particular application or implementation.

[0088] The embodiments of this disclosure may be described herein in terms of functional and / or logical block components and various processing steps. It should be understood that such block components can be implemented by multiple hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of this disclosure may employ various integrated circuit components capable of performing various functions under the control of one or more microprocessors or other control devices, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc. Furthermore, those skilled in the art will understand that embodiments of this disclosure can be practiced in conjunction with multiple systems, and the systems described herein are merely exemplary embodiments of this disclosure.

[0089] For the sake of brevity, techniques related to signal processing, data fusion, signal transmission, control, and other functional aspects of the system (as well as the various operating components of the system) are not described in detail herein. Furthermore, the connecting lines shown in the figures included herein are intended to represent exemplary functional relationships and / or physical connections between elements. It should be noted that alternative or additional functional relationships or physical connections may exist in the embodiments of this disclosure.

[0090] This description is illustrative in nature and is in no way intended to limit the disclosure, its application, or its use. The broad teachings of this disclosure can be implemented in various forms. Therefore, while this disclosure includes specific examples, its true scope should not be so limited, as other variations will become apparent upon examination of the drawings, description, and appended claims.

Claims

1. A method for providing formation information using a multi-focal-plane augmented reality display of a master vehicle, comprising: Receive platooning data from at least one of a plurality of remote vehicles, each of which is part of a platoon, the platooning data including the position, trajectory and time interval of each of the plurality of remote vehicles; The formation data is used to determine whether the convoy is within a predetermined distance from the master vehicle; In response to determining, using the formation data, that the convoy is within the predetermined distance from the master vehicle, a command signal is sent to the multifocal plane augmented reality display of the master vehicle to display a virtual image on the multifocal plane augmented reality display, wherein the virtual image indicates formation actions related to the convoy within the predetermined distance from the master vehicle; Receive vehicle user target data from the vehicle user of the master vehicle, wherein the vehicle user target data includes information about the target of the vehicle user of the master vehicle; In response to determining, using the platooning data, that the convoy is within the predetermined distance from the master vehicle, it is determined whether performing the platooning action is consistent with the objective of the master vehicle's user; In response to determining that the formation maneuver is consistent with the vehicle user's objective, the multi-focal plane augmented reality display of the master vehicle is commanded to display a virtual image, wherein the virtual image includes a query asking the vehicle user whether to perform the formation maneuver; as well as The multi-focal plane augmented reality display has a first image plane and a second image plane. The first image plane shows a view of the external world, and the second image plane is reserved for displaying virtual images. The second image plane spans multiple lanes of a road with a road surface. The virtual images appear at a position farther away from the road surface relative to the first image plane. The multiple lanes include a left lane, a center lane, and a right lane. The second image plane covers the left lane, the center lane, and the right lane. In the central lane, the second image plane begins at a first predetermined distance from the main vehicle and ends at a second predetermined distance from the main vehicle, wherein the second predetermined distance is greater than the first predetermined distance; as well as In the left lane and the right lane, the second image plane is defined by an inclined boundary that begins at a first predetermined distance from the main vehicle and ends at a third predetermined distance from the main vehicle, wherein the third predetermined distance is greater than the first predetermined distance and less than the second predetermined distance. The formation maneuver is performed by the master vehicle.

2. The method of claim 1, wherein the virtual image indicates that the master vehicle is leading the convoy.

3. The method of claim 1, wherein the virtual image indicates that the master vehicle is joining the convoy.

4. The method of claim 1, wherein the virtual image indicates the final destination of the convoy.

5. The method of claim 1, wherein the virtual image indicates a formation maneuver to be performed by each of the plurality of remote vehicles.

6. The method of claim 5, wherein the platooning action is that each of the plurality of remote vehicles in the platoon changes lanes.

7. The method of claim 5, wherein the formation action is that the master vehicle reduces the gap between the master vehicle and one of the plurality of remote vehicles directly in front of the master vehicle.

8. The method of claim 5, wherein the formation action is that the master vehicle increases the gap between the master vehicle and one of the plurality of remote vehicles directly in front of the master vehicle.

9. The method of claim 1, wherein the virtual image indicates that the convoy is private and that the master vehicle is not allowed to join the convoy.

10. The method of claim 1, wherein the virtual image includes an inquiry asking the vehicle user of the master vehicle whether the vehicle user wants to join the fleet.

Images