System and method for head-up display

Abstract

Methods are provided for systems and methods for head-up display that may include obtaining sensor data, determining a plurality of trajectories, and evaluating the plurality of trajectories. The method may also include generating trajectory data, which may then be provided to a user associated with the vehicle. The method may be used for autonomous and non-autonomous vehicles. A system and a computer program product are also provided.

Description

Background Art

[0001] Traffic laws can be complex and may vary between jurisdictions. Human operators of vehicles often violate these laws for a variety of reasons, such as inadequate memory of traffic laws. Currently, many drivers and passengers rely heavily on road signs and their memory learned for their driving tests to inform them of the legality of their driving, when the vehicle instruments only provide them with very basic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 is an example environment in which a vehicle including one or more components of an autonomous system may be implemented;

[0003] Figure 2 is a diagram of one or more example systems of a vehicle including an autonomous system;

[0004] Figure 3 yes Figure 1 and Figure 2 diagrams of components of one or more example devices and / or one or more example systems;

[0005] Figure 4 is a diagram of certain components of an example autonomous system;

[0006] Figure 5A-5D is a diagram of an example implementation of a system for a heads-up display;

[0007] Figure 6A-6B is a diagram of an example implementation of a system for a heads-up display; and

[0008] Figure 7 is a flow chart of an example process for a system and method for a heads-up display. DETAILED DESCRIPTION

[0009] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent that the embodiments described in the present disclosure can be implemented without these specific details. In some instances, well-known configurations and devices are illustrated in block diagram form to avoid unnecessarily obscuring aspects of the present disclosure.

[0010] In the accompanying drawings, for ease of description, the specific arrangement or order of schematic elements (such as those representing systems, devices, modules, instruction blocks and / or data elements, etc.) is illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the accompanying drawings is not intended to mean that a specific processing order or sequence, or separation of processing is required. In addition, unless explicitly described, the inclusion of schematic elements in the accompanying drawings is not intended to mean that such elements are required in all embodiments, nor is it intended to mean that the features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0011] In addition, in the accompanying drawings, connecting elements (such as solid or dotted lines or arrows, etc.) are used to illustrate the connection, relationship or association between or among two or more other schematic elements, and the absence of any such connecting elements is not intended to mean that there can be no connection, relationship or association. In other words, some connections, relationships or associations between elements are not illustrated in the accompanying drawings so as not to obscure the present disclosure. In addition, for ease of illustration, a single connecting element can be used to represent multiple connections, relationships or associations between elements. For example, if the connecting element represents the communication of a signal, data or instruction (e.g., "software instruction"), it will be understood by those skilled in the art that such an element can represent one or more than one signal path (e.g., bus) that may be needed to affect the communication.

[0012] Although the terms "first", "second" and / or "third", etc. are used to describe various elements, these elements should not be limited by these terms. The terms "first", "second" and / or "third" are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact without departing from the scope of the described embodiments. Both the first contact and the second contact are contacts, but they are not the same contacts.

[0013] The terms used in the specification of the various embodiments described herein are included only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the specification of the various embodiments described and the appended claims, the singular forms "a", "an" and "the" are also intended to include plural forms and can be used interchangeably with "one or more than one" or "at least one", unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more than one of the associated listed items. It will also be understood that when the terms "include", "comprise", "have" and / or "have" are used in this specification, the stated features, integers, steps, operations, elements and / or components are specifically stated, but the presence or addition of one or more than one other features, integers, steps, operations, elements, components and / or groups thereof are not excluded.

[0014] As used herein, the terms "communication" and "communicating" refer to at least one of receiving, receiving, transmitting, transmitting and / or providing information (or information represented by, for example, data, signals, messages, instructions and / or commands, etc.). For a unit (e.g., a device, a system, a component of a device or system, and / or a combination thereof) to communicate with another unit, this means that the unit is able to directly or indirectly receive information from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is wired and / or wireless in nature. In addition, even if the transmitted information can be modified, processed, relayed and / or routed between the first unit and the second unit, the two units can communicate with each other. For example, even if the first unit passively receives information and does not actively transmit information to the second unit, the first unit can communicate with the second unit. As another example, if at least one intermediary unit (e.g., a third unit located between the first unit and the second unit) processes the information received from the first unit and transmits the processed information to the second unit, the first unit can communicate with the second unit. In some embodiments, a message may refer to a network packet (eg, a data packet, etc.) that includes data.

[0015] As used herein, the term "if" is optionally interpreted to mean "when," "at," "in response to being determined to be," and / or "in response to being detected," etc., depending on the context. Similarly, the phrases "if it is determined" or "if [the stated condition or event] is detected" are optionally interpreted to mean "when determining," "in response to being determined to be" or "when [the stated condition or event] is detected," and / or "in response to being detected," etc., depending on the context. In addition, as used herein, the terms "have," "have," or "possess," etc. are intended to be open-ended terms. Furthermore, unless expressly stated otherwise, the phrase "based on" is intended to mean "based at least in part on."

[0016] At least one and one or more than one include a function performed by one element, a function performed by more than one element (e.g., in a distributed manner), several functions performed by one element, several functions performed by several elements, or any combination of the above.

[0017] Some embodiments of the present disclosure are described herein in conjunction with threshold values. As described herein, satisfying (such as meeting, etc.) a threshold value may refer to a value being greater than a threshold value, more than a threshold value, above a threshold value, greater than or equal to a threshold value, less than a threshold value, less than a threshold value, below a threshold value, less than or equal to a threshold value, and / or equal to a threshold value, etc.

[0018] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to one of ordinary skill in the art that the various embodiments described may be implemented without these specific details. In other cases, well-known methods, processes, components, circuits, and networks have not yet been described in detail in order not to unnecessarily obscure aspects of the embodiments.

[0019] General Overview

[0020] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement a method for a heads-up display. The method includes obtaining sensor data associated with an environment in which a vehicle is operating using at least one processor. The method includes determining, using at least one processor, a plurality of trajectories for the vehicle based on the sensor data. The method includes evaluating, using at least one processor, scores of trajectories in a plurality of trajectories based on rule parameters, wherein the rule parameters indicate one or more rules for operating the vehicle. The method includes generating, using at least one processor, trajectory data associated with candidate trajectories in a plurality of trajectories based on the scores. The method includes causing, using at least one processor, a device to provide an output to a user associated with the vehicle based on the trajectory data associated with the candidate trajectories.

[0021] By means of the implementation of the systems, methods, and computer program products described herein, techniques for providing additional information related to the trajectory of a vehicle to a user, which additional information can be used, for example, for the interaction of a user with the vehicle via a continuous and / or guided human-machine interaction process. Some advantages of these techniques include, for example, improving the support of the driver in operating the vehicle by performing the technical task of driving the vehicle. Some advantages of these techniques include providing additional information related to the trajectory of a vehicle (e.g., a vehicle equipped with an advanced driver assistance system and / or a vehicle equipped with an autonomous system, etc.) to a user, which additional information can be used, for example, for the interaction of a user with the vehicle via a continuous and / or guided human-machine interaction process. For example, basic information leading to the trajectory of the vehicle, such as the dynamic internal state of the vehicle, is provided to the user. By means of the implementation of certain techniques described herein, information indicating the internal operation of an autonomous vehicle (AV) (e.g., about why the AV behaves this way) is conveyed to the user, which can improve the interaction of the user with the AV.

[0022] Reference now Figure 1 , illustrates an example environment 100 in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, the environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, areas 108, vehicle-to-infrastructure (V2I) devices 110, a network 112, a remote autonomous vehicle (AV) system 114, a fleet management system 116, and a V2I system 118. The vehicles 102a-102n, the vehicle-to-infrastructure (V2I) devices 110, the network 112, the autonomous vehicle (AV) system 114, the fleet management system 116, and the V2I system 118 are interconnected (e.g., establish connections for communication, etc.) via wired connections, wireless connections, or a combination of wired or wireless connections. In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, networks 112, autonomous vehicle (AV) systems 114, fleet management systems 116, and V2I systems 118 via wired connections, wireless connections, or a combination of wired or wireless connections.

[0023] Vehicles 102a-102n (individually referred to as vehicles 102 and collectively referred to as vehicles 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicles 102 are configured to communicate with V2I devices 110, remote AV systems 114, fleet management systems 116, and / or V2I systems 118 via network 112. In some embodiments, vehicles 102 include cars, buses, trucks, and / or trains, etc. In some embodiments, vehicles 102 are similar to vehicles 200 described herein (see Figure 2 ). In some embodiments, vehicles 200 in the set of vehicles 200 are associated with an autonomous queue manager. In some embodiments, vehicles 102 travel along respective routes 106a-106n (individually referred to as routes 106 and collectively referred to as routes 106) as described herein. In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0024] Objects 104a-104n (individually referred to as objects 104 and collectively referred to as objects 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.), etc. Each object 104 is stationary (e.g., located at a fixed location and over a period of time) or moves (e.g., has a speed and is associated with at least one trajectory). In some embodiments, objects 104 are associated with corresponding locations in area 108.

[0025] Routes 106a-106n (individually referred to as routes 106 and collectively referred to as routes 106) are each associated with (e.g., specify a series of actions (also referred to as trajectories) connecting states along which the AV can navigate. Each route 106 begins at an initial state (e.g., a state corresponding to a first spatiotemporal location and / or speed, etc.) and ends at a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target zone (e.g., a subspace of acceptable states (e.g., terminal states)). In some embodiments, the first state includes a location where one or more individuals will board the AV, and the second state or zone includes one or more locations where one or more individuals boarding the AV will disembark. In some embodiments, routes 106 include multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) that are associated with (e.g., define multiple trajectories). In an example, routes 106 include only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection, etc. Additionally or alternatively, the route 106 may include more precise actions or states, such as, for example, specific target lanes or precise locations within lane regions and target speeds at those locations, etc. In an example, the route 106 includes a plurality of precise state sequences along at least one high-level action with a limited look-ahead horizon to an intermediate target, wherein a combination of consecutive iterations of the limited horizon state sequences cumulatively correspond to a plurality of trajectories that collectively form a high-level route terminating at a final target state or region.

[0026] The area 108 includes a physical area (e.g., a geographic region) in which the vehicle 102 can navigate. In an example, the area 108 includes at least one state (e.g., a country, a province, a separate state of a plurality of states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, the area 108 includes at least one named thoroughfare (referred to herein as a "road"), such as a highway, an interstate highway, a parkway, a city street, etc. Additionally or alternatively, in some examples, the area 108 includes at least one unnamed road, such as a driveway, a section of a parking lot, a section of an open space and / or undeveloped area, a dirt road, etc. In some embodiments, the road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In an example, the road includes at least one lane associated with (e.g., identified based on) at least one lane marking line.

[0027] The vehicle-to-infrastructure (V2I) device 110 (sometimes referred to as a vehicle-to-infrastructure or vehicle-to-everything (V2X) device) includes at least one device configured to communicate with the vehicle 102 and / or the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, the fleet management system 116, and / or the V2I system 118 via the network 112. In some embodiments, the V2I device 110 includes a radio frequency identification (RFID) device, a sign, a camera (e.g., a two-dimensional (2D) and / or three-dimensional (3D) camera), lane markings, street lights, parking meters, etc. In some embodiments, the V2I device 110 is configured to communicate directly with the vehicle 102. Additionally or alternatively, in some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, and / or the fleet management system 116 via the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the V2I system 118 via the network 112.

[0028] The network 112 includes one or more wired and / or wireless networks. In an example, the network 112 includes a cellular network (e.g., a long-term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, a cloud computing network, etc., and / or a combination of some or all of these networks, etc.

[0029] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the network 112, the fleet management system 116, and / or the V2I system 118 via the network 112. In an example, the remote AV system 114 includes a server, a server group, and / or other similar devices. In some embodiments, the remote AV system 114 is co-located with the fleet management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle (including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing, etc.). In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the life of the vehicle.

[0030] The queue management system 116 includes at least one device configured to communicate with the vehicles 102, the V2I devices 110, the remote AV system 114, and / or the V2I system 118. In an example, the queue management system 116 includes a server, a server group, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ridesharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including autonomous systems and / or vehicles not including autonomous systems)).

[0031] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the fleet management system 116 via the network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection other than the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or a private agency (e.g., a private agency for maintaining the V2I device 110, etc.).

[0032] In some embodiments, Figure 7 As shown, the apparatus 300 is configured as software instructions to execute one or more steps of the disclosed method.

[0033] supply Figure 1 The number and arrangement of elements illustrated are examples. Figure 1 There may be additional elements, fewer elements, different elements, and / or differently arranged elements than those illustrated. Additionally or alternatively, at least one element of environment 100 may be described as being Figure 1 Additionally or alternatively, at least one set of elements of environment 100 may perform one or more functions described as being performed by at least one different set of elements of environment 100.

[0034] Reference now Figure 2 , vehicle 200 (which can be Figure 1 102) includes, or is associated with, autonomous system 202, powertrain control system 204, steering control system 206, and braking system 208. In some embodiments, vehicle 200 is similar to vehicle 102 (see Figure 1) is the same or similar. In some embodiments, the autonomous system 202 is configured to give the vehicle 200 autonomous driving capabilities (e.g., implementing at least one driving automation or maneuver-based function, feature and / or device, etc., which enables the vehicle 200 to operate partially or completely without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon reliance on human intervention, such as Level 5 ADS-operated vehicles, etc.), highly autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in certain situations, such as Level 4 ADS-operated vehicles, etc.), and / or conditionally autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in limited situations, such as Level 3 ADS-operated vehicles, etc.), etc.). In one embodiment, the autonomous system 202 includes the operational or tactical functions required to enable the vehicle 200 to operate in traffic on the road and continuously perform part or all of the dynamic driving task (DDT). In another embodiment, the autonomous system 202 includes an advanced driver assistance system (ADAS) including driver support features. The autonomous system 202 supports various levels of driving automation ranging from no driving automation (e.g., level 0) to full driving automation (e.g., level 5). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, reference may be made to SAE International's standard J3016: Taxonomy and Definitions for TermsRelated to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ridesharing company.

[0035] Autonomous system 202 includes a sensor suite that includes one or more devices such as camera 202a, LiDAR sensor 202b, Radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance that vehicle 200 has traveled, etc.). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 described herein. Data generated by one or more devices of autonomous system 202 can be used by one or more systems described herein to observe the environment (e.g., environment 100) in which vehicle 200 is located. In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, drive-by-wire (DBW) system 202h, and safety controller 202g.

[0036] The camera 202a includes a communication device 202e, an autonomous vehicle computer 202f, and / or a safety controller 202g configured to communicate with the communication device 202e via a bus (e.g., Figure 3 The camera 202a includes at least one device for communicating with the autonomous vehicle computing 202f (e.g., an image processing unit 202a, a bus ... Figure 1In some embodiments, the autonomous vehicle computing 202f determines a depth to one or more objects in a field of view of at least two of the plurality of cameras based on image data from the at least two cameras. In some embodiments, the camera 202a is configured to capture images of objects within a distance relative to the camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Thus, the camera 202a includes features such as sensors and lenses that are optimized for sensing objects at one or more distances relative to the camera 202a.

[0037] In an embodiment, the camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects that provide visual navigation information. In some embodiments, the camera 202a generates traffic light data associated with the one or more images. In some examples, the camera 202a generates TLD (traffic light detection) data associated with one or more images including a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a that generates TLD data differs from other systems incorporating cameras described herein in that the camera 202a may include one or more cameras with a wide field of view (e.g., a wide-angle lens, a fisheye lens, and / or a lens with a viewing angle of approximately 120 degrees or greater, etc.) to generate images related to as many physical objects as possible.

[0038] The light detection and ranging (LiDAR) sensor 202b includes a communication device 202e, an autonomous vehicle computing device 202f, and / or a safety controller 202g via a bus (e.g., Figure 3The LiDAR sensor 202b includes at least one device that communicates with a bus (the same or similar bus as the bus 302 of the embodiment of the present invention). The LiDAR sensor 202b includes a system configured to emit light from a light emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object encountered by the light. The LiDAR sensor 202b also includes at least one light detector that detects the light emitted from the light emitter after encountering the physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., a point cloud and / or a combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of the physical object and / or the surface of the physical object (e.g., the topology of the surface), etc. In such examples, the image is used to determine the boundaries of the physical object in the field of view of the LiDAR sensor 202b.

[0039] The radio detection and ranging (Radar) sensor 202c includes a sensor configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., Figure 3 At least one device for communicating with a bus (same or similar bus as bus 302 of the embodiment of the present invention). Radar sensor 202c includes a system configured to transmit (pulsed or continuous) radio waves. The radio waves transmitted by Radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves transmitted by Radar sensor 202c encounter physical objects and are reflected back to Radar sensor 202c. In some embodiments, the radio waves transmitted by Radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with Radar sensor 202c generates a signal representing an object included in the field of view of Radar sensor 202c. For example, at least one data processing system associated with Radar sensor 202c generates an image representing the boundary of a physical object and / or the surface of a physical object (e.g., the topology of the surface), etc. In some examples, the image is used to determine the boundary of a physical object in the field of view of Radar sensor 202c.

[0040] The microphone 202d includes a microphone configured to communicate with the communication device 202e, the autonomous vehicle computing device 202f, and / or the safety controller 202g via a bus (e.g., Figure 3 At least one device for communicating with the vehicle 200 (the same or similar bus as bus 302 of FIG. 1 ). Microphone 202d includes one or more microphones (e.g., an array microphone and / or an external microphone, etc.) that capture an audio signal and generate data associated with (e.g., representing) the audio signal. In some examples, microphone 202d includes a transducer device and / or the like. In some embodiments, one or more systems described herein can receive the data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on an audio signal associated with the data.

[0041] The communication device 202e includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the autonomous vehicle computing 202f, the safety controller 202g, and / or the DBW (drive-by-wire) system 202h. For example, the communication device 202e may include at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the autonomous vehicle computing 202f, the safety controller 202g, and / or the DBW (drive-by-wire) system 202h. Figure 3 The communication device 202e may be a device that is the same as or similar to the communication interface 314 of the vehicle. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (eg, a device for enabling wireless communication of data between vehicles).

[0042] Autonomous vehicle computing 202f includes at least one device configured to communicate with camera 202a, LiDAR sensor 202b, Radar sensor 202c, microphone 202d, communication device 202e, safety controller 202g, and / or DBW system 202h. In some examples, autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablet computers, etc.), and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units, etc.). In some embodiments, autonomous vehicle computing 202f is the same or similar to autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with Figure 1 remote AV system 114 of the same or similar autonomous vehicle system), a fleet management system (e.g., Figure 1 of the same or similar queue management system as the queue management system 116), V2I devices (e.g., Figure 1 V2I device 110 that is the same as or similar to V2I device 110) and / or V2I system (e.g., Figure 1The V2I system 118 may communicate with the same or similar V2I system.

[0043] Safety controller 202g includes at least one device configured to communicate with camera 202a, LiDAR sensor 202b, Radar sensor 202c, microphone 202d, communication device 202e, autonomous vehicle computing 202f, and / or DBW system 202h. In some examples, safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate one or more devices of vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, safety controller 202g is configured to generate control signals that take precedence over (e.g., override) control signals generated and / or transmitted by autonomous vehicle computing 202f.

[0044] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device of the vehicle 200 (e.g., turn signal lights, headlights, door locks, and / or windshield wipers, etc.).

[0045] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to make longitudinal vehicle movements such as starting forward movement, stopping forward movement, starting backward movement, stopping backward movement, accelerating in a certain direction, decelerating in a certain direction, etc., or to make lateral vehicle movements such as turning left and / or turning right. In an example, the powertrain control system 204 increases, maintains the same, or decreases the energy (e.g., fuel and / or electricity, etc.) provided to the motor of the vehicle, thereby rotating or not rotating at least one wheel of the vehicle 200. In other words, the steering control system 206 causes the activities necessary for the adjustment of the y-axis component of the vehicle motion.

[0046] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 rotates the two front wheels and / or the two rear wheels of the vehicle 200 to the left or right to turn the vehicle 200 left or right.

[0047] Braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or hold vehicle 200 stationary. In some examples, braking system 208 includes at least one controller and / or actuator configured to cause one or more calipers associated with one or more wheels of vehicle 200 to close on respective rotors of vehicle 200. Additionally or alternatively, in some examples, braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0048] In some embodiments, vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring a property of a state or condition of vehicle 200. In some examples, vehicle 200 includes platform sensors such as a global positioning system (GPS) receiver, an inertial measurement unit (IMU), wheel rate sensors, wheel brake pressure sensors, wheel torque sensors, engine torque sensors, and / or steering angle sensors. Although braking system 208 is Figure 2 Although illustrated as being located on the proximal side of the vehicle 200 , the braking system 208 may be located anywhere in the vehicle 200 .

[0049] Reference now Figure 3, a schematic diagram illustrating a device 300. As illustrated, the device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, the device 300 corresponds to: at least one device of the vehicle 102 (e.g., at least one device of the system of the vehicle 102), at least one device of the remote AV system 114, the fleet management system 116, the V2I system 118; and / or one or more devices of the network 112 (e.g., one or more devices of the system of the network 112). In some embodiments, one or more devices of the vehicle 102 (e.g., one or more devices of the system of the vehicle 102, such as the remote AV system 114, the fleet management system 116, at least one device of the V2I system 118), and / or one or more devices of the network 112 (e.g., one or more devices of the system of the network 112) include at least one device 300 and / or at least one component of the device 300. like Figure 3 As shown, apparatus 300 includes a bus 302 , a processor 304 , a memory 306 , a storage component 308 , an input interface 310 , an output interface 312 , and a communication interface 314 .

[0050] The bus 302 includes components that permit communication between components of the device 300. In some cases, the processor 304 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or an accelerated processing unit (APU), etc.), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), etc.). The memory 306 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory, etc.) that stores data and / or instructions for use by the processor 304.

[0051] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk, etc.), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a tape, a CD-ROM, a RAM, a PROM, an EPROM, a FLASH-EPROM, an NV-RAM, and / or another type of computer-readable medium, and a corresponding drive.

[0052] The input interface 310 includes components that permit the device 300 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, and / or a camera, etc.). Additionally or alternatively, in some embodiments, the input interface 310 includes a sensor for sensing information (e.g., a global positioning system (GPS) receiver, an accelerometer, a gyroscope, and / or an actuator, etc.). The output interface 312 includes components for providing output information from the device 300 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs), etc.).

[0053] In some embodiments, communication interface 314 includes a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter, etc.) that permits device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. In some examples, communication interface 314 permits device 300 to receive information from another device and / or provide information to another device. In some examples, communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, interface and / or cellular network interface, etc.

[0054] In some embodiments, the device 300 performs one or more processes described herein. The device 300 performs these processes based on the processor 304 executing software instructions stored by a computer-readable medium such as a memory 306 and / or a storage component 308. Computer-readable media (e.g., non-transitory computer-readable media) are defined herein as non-transitory memory devices. Non-transitory memory devices include storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0055] In some embodiments, the software instructions are read into the memory 306 and / or storage component 308 from another computer-readable medium or from another device via the communication interface 314. The software instructions stored in the memory 306 and / or storage component 308, when executed, cause the processor 304 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry is used in place of or in combination with the software instructions to perform one or more processes described herein. Therefore, unless expressly stated otherwise, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0056] The memory 306 and / or the storage component 308 include a data storage unit or at least one data structure (e.g., a database, etc.). The device 300 can receive information from the data storage unit or at least one data structure in the memory 306 or the storage component 308, store information in the data storage unit or at least one data structure, communicate information to the data storage unit or at least one data structure, or search for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0057] In some embodiments, the device 300 is configured to execute software instructions stored in the memory 306 and / or the memory of another device (e.g., another device that is the same as or similar to the device 300). As used herein, the term "module" refers to at least one instruction stored in the memory 306 and / or the memory of another device, which, when executed by the processor 304 and / or the processor of another device (e.g., another device that is the same as or similar to the device 300), causes the device 300 (e.g., at least one component of the device 300) to perform one or more processes described herein. In some embodiments, the module is implemented in software, firmware, and / or hardware, etc.

[0058] supply Figure 3 The number and arrangement of components illustrated are examples. Figure 3 The device 300 may include additional components, fewer components, different components, or differently arranged components than those illustrated. Additionally or alternatively, a set of components (e.g., one or more components) of the device 300 may perform one or more functions described as being performed by another component or set of components of the device 300.

[0059] Reference now Figure 4, illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack"). As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, the planning system 404, the positioning system 406, the control system 408, and the database 410 are included in and / or implemented in an automatic navigation system of a vehicle (e.g., the autonomous vehicle computing 202f of the vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, the planning system 404, the positioning system 406, the control system 408, and the database 410 are included in one or more independent systems (e.g., one or more systems that are the same or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more independent systems located in the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in a memory), computer hardware (e.g., by a microprocessor, microcontroller, application specific integrated circuit (ASIC) and / or field programmable gate array (FPGA), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with a remote system (e.g., an autonomous vehicle system that is the same or similar to the remote AV system 114, a fleet management system 116 that is the same or similar to the fleet management system 116, and / or a V2I system that is the same or similar to the V2I system 118, etc.).

[0060] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a), the image being associated with (e.g., representing) one or more physical objects within the field of view of the at least one camera. In such examples, the perception system 402 classifies the at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of the physical object by the perception system 402, the perception system 402 transmits data associated with the classification of the physical object to the planning system 404.

[0061] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data (e.g., data associated with the classification of physical objects described above) from the perception system 402, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the perception system 402. In other words, the planning system 404 can perform tasks related to the tactical functions required to operate the vehicle 101 in traffic on the road. Tactical efforts involve maneuvering the vehicle in traffic during the journey, which includes but is not limited to deciding whether and when to overtake another vehicle, change lanes, or select an appropriate rate, acceleration, deceleration, etc. In some embodiments, the planning system 404 receives data associated with an updated position of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0062] In some embodiments, the positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, the positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, the positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and the positioning system 406 generates a combined point cloud based on each point cloud. In these examples, the positioning system 406 compares the at least one point cloud or the combined point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in the database 410. Then, based on the positioning system 406 comparing the at least one point cloud or the combined point cloud with the map, the positioning system 406 determines the position of the vehicle in the area. In some embodiments, the map includes a combined point cloud of the area generated before the navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of roadway geometry, a map describing the road network connectivity, a map describing the physical properties of the roadway (such as traffic speed, traffic volume, the number of vehicle and bicycle traffic lanes, lane width, lane traffic direction or the type and location of lane markings, or a combination thereof, etc.), and a map describing the spatial location of road features (such as crosswalks, traffic signs, or various types of other driving lights, etc.). In some embodiments, the map is generated in real time based on data received by the perception system.

[0063] In another example, positioning system 406 receives global navigation satellite system (GNSS) data generated by a global positioning system (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with a location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such an example, positioning system 406 determines the position of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, positioning system 406 generates data associated with the position of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the position of the vehicle. In such an example, the data associated with the position of the vehicle include data associated with one or more semantic properties corresponding to the position of the vehicle.

[0064] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate the powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), the steering control system (e.g., steering control system 206) and / or the braking system (e.g., braking system 208). For example, the control system 408 is configured to perform operating functions such as lateral vehicle motion control or longitudinal vehicle motion control. Lateral vehicle motion control causes the necessary activities for the adjustment of the y-axis component of the vehicle motion. Longitudinal vehicle motion control causes the necessary activities for the adjustment of the x-axis component of the vehicle motion. In the example, in the case where the trajectory includes a left turn, the control system 408 transmits a control signal to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby turning the vehicle 200 left. Additionally or alternatively, the control system 408 generates and transmits control signals to cause other devices of the vehicle 200 (eg, headlights, turn signal lights, door locks, and / or windshield wipers, etc.) to change states.

[0065] In some embodiments, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multi-layer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model alone or in combination with one or more of the above systems. In some examples, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in an environment, etc.).

[0066] Database 410 stores data transmitted to, received from, and / or updated by perception system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes a storage component (e.g., a storage component) for storing data and / or software related to operations and using at least one system of autonomous vehicle computing 400. Figure 3In some embodiments, database 410 stores data associated with a 2D and / or 3D map of at least one area. In some examples, database 410 stores data associated with a 2D and / or 3D map of a portion of a city, portions of multiple cities, multiple cities, counties, states, and / or countries (State) (e.g., a country), etc. In such an example, a vehicle (e.g., a vehicle that is the same or similar to vehicle 102 and / or vehicle 200) can be driven along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and cause at least one LiDAR sensor (e.g., a LiDAR sensor that is the same or similar to LiDAR sensor 202b) to generate data associated with an image representing an object included in the field of view of the at least one LiDAR sensor.

[0067] In some embodiments, database 410 can be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle that is the same or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system that is the same or similar to remote AV system 114), a fleet management system (e.g., a vehicle that is the same or similar to remote AV system 114), and a fleet management system (e.g., a vehicle that is the same or similar to remote AV system 114). Figure 1 The same or similar queue management system as the queue management system 116 of FIG. 1 and / or the V2I system (e.g., Figure 1 The V2I system 118 is the same as or similar to the V2I system) and the like.

[0068] The present disclosure relates to systems, methods, and computer program products for providing information and / or guidance to a user during vehicle operation, such as displaying via a heads-up display and / or providing auditory notifications, etc. Currently, most drivers and passengers rely almost entirely on road signs and their memories of studying for a driving test to understand legal behavior, with vehicle instruments providing them with only very basic information (e.g., speed limits on a navigation screen). Advantageously, the disclosed systems, methods, and computer program products can obtain and display more information than a user would normally know, such as complex and changing traffic regulations, etc.

[0069] For example, at a given point in time, the vehicle is configured to put together a detailed understanding of the world and plan systems such as Figure 4The planning system 404 of the vehicle, etc., can generate candidate trajectories that represent reasonable choices to be taken by the vehicle. The candidate trajectories can be evaluated by a set of rules for specific attributes, such as legality (e.g., binary legal or illegal, etc.), overall score, and one or more of safety. In some examples, these rankings are communicated to the user of the vehicle, such as by color coding different trajectories according to their scores.

[0070] Reference now Figure 5A-5D , which illustrates a diagram of an implementation of a system 500 for a heads-up display of a vehicle 550. In some embodiments, the system 500 includes a vehicle computing 540 (such as a vehicle computing device similar to Figure 2 AV Computing 202F and / or Figure 4 AV computing 400, etc.) and vehicle 550 (similar to Figure 1 The vehicle 102 and / or Figure 2 In one or more examples or embodiments, the system 500 is an autonomous vehicle (e.g., Figure 1 and Figure 2 exemplified as vehicles 102 and 200), autonomous systems (similar to Figure 2 autonomous system 202 and / or one or more components of autonomous system 202), a device (similar to Figure 3 The system 500 may be used to operate an autonomous vehicle. The system 500 may not be used to operate an autonomous vehicle, such as for use in a non-autonomous vehicle, etc.

[0071] In one or more embodiments or examples, system 500 is configured to utilize one or more sensors (such as sensor 508) of vehicle 550, such as Figure 2 One or more cameras 202a, one or more LiDAR sensors 202b, and / or one or more Radar sensors 202c of the autonomous vehicle 200. In addition, the system 500 can be configured to utilize one or more of the output interfaces and communication interfaces of the vehicle 550 (e.g., using Figure 3 Output interface 312 and communication interface 314) provide different kinds of displays. Some data (e.g., sensor data 509, trajectory data 514, rulebook 510, information) can be stored in a database (such as Figure 4 Database 410 and / or Figure 3 in the storage component 308).

[0072] In one or more embodiments or examples, the system 500 includes Figure 4 One or more planning systems 504 and perception systems 502 that are the same or similar to the planning system 404 and the perception system 402.

[0073] A system 500 is disclosed herein. The system 500 includes at least one processor. The system 500 includes at least one memory storing instructions thereon, the instructions causing the at least one processor to perform operations when executed by the at least one processor, the operations including obtaining sensor data 509 associated with an environment in which a vehicle 550 is operating. The operations include determining a plurality of trajectories 512 for the vehicle 550 based on the sensor data 509. The operations include evaluating scores of trajectories in the plurality of trajectories 512 based on rule parameters 511, wherein the rule parameters 511 indicate one or more rules for operating the vehicle 550. The operations include generating trajectory data 514 associated with candidate trajectories 513 in the plurality of trajectories 512 based on the scores. The operations include causing a device to provide an output to a user associated with the vehicle 550 based on the trajectory data 514 associated with the candidate trajectories 513.

[0074] In one or more examples, the system 500 is incorporated into a non-autonomous vehicle. For example, the system 500 is used to provide guidance to an operator and / or passengers of the vehicle. In one or more examples, the system 500 is incorporated into an autonomous vehicle. For example, the system 500 is used for operation of an autonomous vehicle. The system 500 of an autonomous vehicle can also be used to provide guidance to an operator and / or passengers of an autonomous vehicle.

[0075] In one or more examples or embodiments, the system 500 is configured to determine information about the environment in which the vehicle is operating and provide relevant information to the user of the vehicle. The relevant information may range from trajectory information derived from traffic regulations to information generated by regional customs and / or parking information. As an example, the system 500 determines multiple trajectories 512 for the vehicle 550, such as multiple potential trajectories, etc. Then, the system 500 evaluates the trajectory based on the rule parameters 511, and gives a score to the trajectory in one or more examples. The rule parameters 511 can be based on a rule book 510, which represents the rules for operating the vehicle in a hierarchical structure (such as including "road rules", etc.) for a specific location where the system 500 is located. For example, the system 500 then generates trajectory data 514 associated with at least one trajectory (such as candidate trajectory 513, etc.) in the multiple trajectories 512. In one or more embodiments or examples, the system 500 provides information related to the candidate trajectory 513 (e.g., trajectory data 514). The trajectory data 514 may be one or more of many types of data that may be provided to a user, such as visual data or auditory data, etc. In some examples, the trajectory data 514 provides guidance to a user of the vehicle 550. The trajectory data 514, for example, provides the user with relevant information for the operation of the vehicle 550. As an example, the trajectory data 514 indicates what areas should or should not be driven on, delineation of areas where it is illegal to drive, rules of the road, yielding, etc., as will be discussed in detail below.

[0076] In one or more embodiments or examples, the system 500 polls the trajectories generated by the planning system 504 (periodically or continuously), scores at least a subset of these trajectories, and then displays the scores of the various trajectories (directly or through a representation) to the user operating the vehicle 550. Examples may include scores for lane switching, turning, deceleration, acceleration, etc. Thus, at any given moment, the user can obtain information about the operation of the vehicle (such as whether an operation such as switching lanes, overtaking, etc. is performed due to traffic regulations).

[0077] In one or more examples or embodiments, the system 500 is configured to communicate with the user via a communication channel such as a wireless communication channel. Figure 5BThe perception system 502 shown, etc., obtains sensor data 509 from the sensor 508. The system 500 can use the sensor data 509 to determine a plurality of trajectories 514. The sensor data 509 can be one or more of the following: Radar sensor data, image sensor data (e.g., camera sensor data), audio sensor data, and LiDAR sensor data. The specific type of sensor data 509 is not limiting. The sensor data 509 can indicate the environment around the vehicle 550. For example, the sensor data 509 can indicate an object and / or multiple objects in the environment in which the vehicle 550 operates.

[0078] Sensors, such as sensor 508, may be one or more sensors, such as onboard sensors. Sensor 508 may be associated with vehicle 550. Vehicle 550 may include one or more sensors that may be configured to monitor the environment in which the vehicle is operating, such as via sensor 508, through sensor data 509. For example, monitoring may provide sensor data 509 indicating what is happening in the environment around vehicle 550, such as for determining a trajectory of vehicle 550. Sensor 508 may be one or more of the following: a Radar sensor, a camera sensor, a microphone, an infrared sensor, an image sensor, and a LiDAR sensor. Sensor 508 may include Figure 2 One or more of the sensors shown in , such as camera 202a, LiDAR sensor 202b, Radar sensor 202c, and microphone 202d.

[0079] In one or more embodiments or examples, the sensor data 509 indicates an environment, which may be a road, area, surface, town, city, country, such as a vehicle 550 is operating (e.g., driving) and / or located in. The environment may include any number of aspects, including signs, infrastructure, etc. In one or more embodiments or examples, the environment includes non-transitory objects.

[0080] In one or more examples and embodiments, the system 500 is configured such as by Figure 4500 . This may allow the system 500 to obtain different rule parameters 511 (e.g., from the rule book 510) for a particular area where the vehicle 550 is located. For example, the rule parameters in the first location may indicate different rules from the rule parameters in the second location, such as due to changed road regulations. The system 500 may be configured to obtain different rule parameters 511 from the rule book 510 according to the location of the vehicle 550. In some embodiments, the planning system 504 may access data including rules (e.g., rule parameters 511) for planning. For example, the rules (as indicated by the rule parameters 511) are specified using a formal language (e.g., using Boolean logic). In some examples, the rules are road rules, passenger comfort rules, and / or expression rules. In some examples, the road rules define whether a specific maneuver is permitted in the driving lane of the vehicle and / or the environment of the vehicle. For example, the rule book 510 may include a rule parameter 511 indicating that lane changes are prohibited in a construction zone. Then, based on rule parameters 511, system 500 will not score trajectories with lane changes highly, and thus will not perform maneuvers that require lane changes. In some examples, passenger comfort rules define whether a particular passenger within a vehicle is prone to motion sickness and is sensitive to high g-equivalent ("g" forces). In a situation encountered by the vehicle, at least some of the rules may apply to that situation. Rules may have priorities (e.g., may be associated with a priority level and / or a score). For example, a rule of "if the road is a highway, move to the leftmost lane" may have a lower priority than "if approaching an exit within one mile, move to the rightmost lane."

[0081] In one or more embodiments or examples, the system 500 determines a plurality of trajectories 512 for a vehicle 550, such as via a planning system 504. The system 500 obtains sensor data 509, and in one or more embodiments or examples, the planning system 504 uses the sensor data 509 to determine any agents in the environment. In addition, the system 500 may use a positioning system such as Figure 4406, etc.) to determine where the vehicle is located. Optionally, based on the positioning of the vehicle determined by the positioning system and the sensor data 509, the system 500 can determine one or more trajectories (such as multiple trajectories 512, etc.) so that the vehicle does not adversely encounter one or more agents in the environment. In one or more examples or embodiments, the system 500 only determines potential trajectories and does not make any assessment of the legality or possibility of the trajectory until the evaluation step. In addition, the system 500, for example, determines trajectories that will follow the normal rules of the road (such as staying in the lane, etc.). In one or more examples or embodiments, the system 500 predicts the trajectory of the agent in the environment, and the system determines multiple trajectories 512 that do not intersect with the trajectory of the agent. In other words, the system 500 can be configured to obtain sensor data 509, and optionally obtain positioning data, for determining multiple trajectories 512.

[0082] In some examples, multiple trajectories 512 are candidate trajectories that the driver can select at the current point in time. Multiple trajectories 512 may include any number of trajectories, and the specific number is not limiting. In some instances, such as when there is only a single potential legal and / or safe trajectory, a single trajectory can be determined for vehicle 550. In one or more embodiments or examples, system 500 determines the trajectory of vehicle 550.

[0083] A trajectory can be considered as one or more maneuvers that can be performed by the vehicle 550. In one or more examples or embodiments, a trajectory includes a series of states associated with parameters (e.g., time and speed), such as a current position, an intermediate position, and / or a target position. For example, this can be a lane-level trajectory. In one or more examples or embodiments, a trajectory is different from a route. Multiple trajectories, typically a very large number of trajectories, can form a route. In other words, a trajectory can be time-limited, moment-to-moment action, as opposed to indicating a long-term route.

[0084] In one or more embodiments or examples, determining multiple trajectories 512 includes determining multiple trajectories every 30 seconds or less. The system 500 determines multiple trajectories 512, for example, every 30 seconds, 25 seconds, 20 seconds, 15 seconds, 10 seconds, or 5 seconds or less. This can be compared to route determination, which occurs much less frequently, such as only when an action significantly changes a route. The system 500 can continuously determine multiple trajectories 512, which may be advantageous for providing continuously updated information to the user.

[0085] In one or more embodiments or examples, the system 500 is configured to evaluate each of the plurality of trajectories 512. The system 500, for example, obtains the plurality of trajectories 512 and evaluates each of the plurality of trajectories 512 to output a candidate trajectory 513 and / or a score. In one or more examples or embodiments, the candidate trajectory 513 is considered the "best" trajectory of the plurality of trajectories based on a rule as indicated by the rule parameter 511.

[0086] For example, the system 500 evaluates the scores of the various trajectories based on the rule parameters 511 (discussed below). The rule parameters 511 may be obtained by the system 500, such as from a memory and / or a database such as Figure 4 The rule parameters 511 may be obtained from a database 410 of a vehicle 550, etc. The rule parameters 511 may be stored in the system 500. In one or more embodiments or examples, the system 500 periodically obtains the rule parameters 511. This may allow the system 500 to have appropriate rule parameters 511 for a particular location where the vehicle 550 is located. The system 500 may be configured to obtain the rule parameters 511 at certain known delineations, such as crossing state and / or country borders, entering or leaving a town and / or city, etc.

[0087] In one or more examples or embodiments, the system 500 is configured to evaluate each of the plurality of trajectories 512 to provide a score. In one or more examples or embodiments, the score is a representative numerical value of a particular trajectory based on the rule parameters 511. The score can be an overall score, which in some examples is used to evaluate the legality and / or safety of the plurality of trajectories. The score can indicate the value of the particular trajectory, such as based on compliance with certain rules indicated by the rule parameters 511, etc. In one or more examples or embodiments, the score is also based on user preferences, such as fuel efficiency, environmental impact, speed, etc. In some examples, the system 500 selects the candidate trajectory 513 having the lowest score in the plurality of trajectories 512.

[0088] For example, the system 500 is configured to evaluate each of the plurality of trajectories 512 based on the rule parameters 511. In some examples, the system 500 is configured to provide a numerical evaluation (e.g., a score) of each of the plurality of trajectories 512 based on any violation of the rule parameters 511. For example, the system 500 can be configured to give a "point" (e.g., as a score) to the trajectory for each rule that the trajectory violates. In one or more examples or embodiments, after evaluating each trajectory, the system 500 selects the trajectory with the lowest total number of points (e.g., the lowest score) in the plurality of trajectories 512, which indicates the lowest violation of the rule. The trajectory with the lowest score can be a candidate trajectory 513.

[0089] In some examples, the system 500 obtains the rule parameter 511 from a rule book 510 (e.g., the rule parameter 511 is based on the rule book), which may include a set of rule parameters associated with a list of corresponding rules and corresponding data indicating such rules. The rule book 510 may contain multiple rule parameters, and the system 500 may obtain the rule parameter 511 associated with a specific location of the vehicle 550.

[0090] The rules indicated by rule parameters 511 include aspects of operating vehicle 550, such as staying within a specific distance of other agents in the environment, speed limits, what areas can be driven on, whether lane changes are allowed, whether overtaking is allowed, where stopping, standing or parking is allowed, whether the driver needs to give way to cross traffic, parking areas, etc. Rule parameters 511 can vary depending on the jurisdiction (e.g., country, city, location) in which vehicle 550 is operating. In one or more examples or embodiments, rule parameters 511 indicate one or more of road rules, passenger comfort rules, and expression rules. For example, rule parameters 511 indicate regulations and / or customary rules. Rule parameters 511 can be set by the manufacturer of vehicle 550.

[0091] In one or more examples or embodiments, the system 500 is configured to generate trajectory data 514 (such as Figure 5C , etc.). The system 500 may output the trajectory data 514, such as to a Figure 5D The system 500 may include an interface 516 as shown. For example, the system 500 obtains the scores and the candidate trajectory 513 with the lowest score. The system 500 may then obtain relevant data about the candidate trajectory 513, such as one or more of the sensor data 509, the rulebook 510, and the rule parameters 511. In one or more examples or embodiments, based on all of the obtained data, the system generates trajectory data 514 that may be relevant to the user of the vehicle. The system 500 may obtain user input regarding the type of trajectory data 514 that the user may want to receive. The system 500 may generate trajectory data 514 based on the user input. The system 500 may generate different types of trajectory data 514.

[0092] In some examples, the system 500 uses the positioning data and / or sensor data 509 to determine what type of trajectory data 514 to generate. For example, if the system 500 obtains sensor data 509 indicating a nearby parking space, the system 500 generates trajectory data 514 related to parking. As another example, when the positioning data and / or sensor data 509 indicates that the vehicle is traveling at a high speed on a highway, the system 500 is configured to generate trajectory data 514 related to that action rather than parking. In one or more examples or embodiments, the system 500 is configured to use the scores and candidate trajectories 513 to determine what trajectory data 514 to generate, and may not use the positioning data and / or sensor data 509.

[0093] In some embodiments, the system 500 determines whether the sensor data 509 and / or the positioning data meet one or more trajectory criteria. The trajectory criteria may indicate an action taken by the vehicle. As an example, the trajectory criteria may include parking criteria, highway criteria, local traffic criteria, etc. Each trajectory criterion may have associated trajectory data 514. When it is determined that the sensor data 509, positioning data, score, and / or candidate trajectory 513 meet one or more of the trajectory criteria, the system 500 may be configured to generate trajectory data 514 associated with the met trajectory criteria. When it is determined that the sensor data 509, positioning data, score, and / or candidate trajectory 513 do not meet one or more of the trajectory criteria, the system 500 may be configured not to generate trajectory data 514 associated with the met trajectory criteria.

[0094] In one or more examples or embodiments, trajectory data 514 is associated with candidate trajectory 513 (e.g., potential trajectory) in plurality of trajectories 512. In some examples, candidate trajectory 513 is an allowable potential trajectory that has been evaluated for the rule(s) indicated by rule parameters 511. Trajectory data 514 may indicate one or more actions that vehicle 550 may take. As described, trajectory data 514 may include an analysis of different environmental (such as positioning, etc.) rules based on rule parameters 511.

[0095] In one or more embodiments or examples, generating trajectory data 514 includes generating trajectory data 514 based on scores and sensor data 509. Thus, trajectory data 514 can be based on scores and sensor data 509. This can include real-time sensor data such as obtained by vehicle 550 during operation of vehicle 550. Advantageously, using sensor data 509 to generate trajectory data 514 can allow for recording and evaluating transient agents in the environment. In some examples, system 500 uses sensor data 509 to provide a more complete view of the environment for a user of vehicle 550.

[0096] In one or more examples or embodiments, the system 500 is configured to cause the device to provide an output based on the trajectory data 514. The system 500 can perform such actions via the interface 516. In other words, the system 500 can be configured to provide guidance and / or information to the user of the vehicle 550. As an example, the output provides information such as whether certain potential trajectories are illegal or poorly rated, such as for safety reasons (e.g., fast-moving cars approaching in different lanes). The type of device may include a device for providing visual images and / or auditory sounds, and the output may indicate a specific visual image and / or sound, such as via the interface 516. The device (such as a smart phone of an operator and / or passenger of the vehicle 550, etc.) can be incorporated into the vehicle 550, or can be separated from the vehicle 550.

[0097] In one or more embodiments or examples, causing the device to provide an output to a user based on the trajectory data 514 includes causing the device to display a user interface object 520 representing the trajectory data 514 to the user. The user interface object 520 can indicate any number of display information. The display information can include one or more of route information, driving information, traffic information, speed limits, intersections, legal parking areas, legal turning areas, etc. Examples of user interface objects 520 include icons, notifications to be presented, shading, colors, legality information, percentages, etc. The user interface object 520 can be a visual symbol that provides some information and guidance to the user of the vehicle 550. Figure 6A-6B Various examples of user interfaces including the disclosed user interface objects are provided.

[0098] In one or more embodiments or examples, the user interface object 520 represents an action that the vehicle 550 may or may not take. In some examples, the system 500 is configured to overlay the user interface object 520 on the road in the user's field of view so that the user can see both the road and the user interface object 520.

[0099] For example, if the system 500 determines that a car is approaching in the left lane, the user interface object 520 can show the user a lane shaded in red. This can indicate that the vehicle 550 cannot enter the left lane. Once the car has passed, the user interface object 520 can turn green, indicating that it is safe to change lanes. As another example, the user interface object 520 is a lane shaded in red for traffic moving in the opposite direction. Since it would be illegal for the vehicle 550 to enter this lane as indicated by the rule parameter 511, it may always be shaded in red.

[0100] The system 500 can use different user interface objects to provide any useful information to the user. In one or more embodiments or examples, the user interface object 520 shows the speed of the current lane where the vehicle 550 is located. In this way, the user will know the speed without having to rely on road signs that may not be easy to understand. The system 500 can be configured to obtain a rule parameter 511 that indicates the speed limit of the current location where the vehicle 550 is operating.

[0101] As another example, user interface object 520 may be used to show the availability of parking spots for vehicle 550. Open spots may be shaded as one color, while non-open spots may be shaded as a different color. In addition, system 500 may determine whether parking is allowed at a certain location based on rule parameters 511. Even if there are no other vehicles at the location, it may be illegal for a vehicle to park there, and therefore system 500 may provide a user interface object 520 indicating that parking is prohibited.

[0102] In one or more than one embodiment or example, user interface object 520 comprises user interface object and / or pictogram of color coding.User interface object 520 can represent multiple tracks 512 that can be taken by vehicle 550.Color coding can be used to distinguish tracks based on the score of track.Color coding can be used for regional description or semantic purpose.In one or more than one embodiment or example, pictogram illustrates vehicle, pedestrian, bicycle etc.In some examples, pictogram is the representation of the environment where vehicle 550 is located.Although color coding is discussed here, other types of visual prompts can be used for user interface objects.Different patterns, hatches etc. can distinguish different user interface objects, which may be particularly advantageous for color blind users.

[0103] Figure 6A-6B An example is given of a vehicle (such as Figure 5B-5D The vehicle 550 and / or Figure 2 An example of a user interface object displayed on a windshield 600 of a vehicle 200, etc. Fig. 6AAs shown, the user will be able to see the road 602 where the vehicle is operating. Based on the trajectory data, the system 500 can be configured to display a first user interface object 604, which is displayed as a red-shaded lane. The system 500 can be configured to display a second user interface object 606 as a vehicle. The second user interface object 606 can be overlaid on the actual vehicle in the environment, or can be shown as indicating to the user that the vehicle will come in the red-shaded lane indicated by the first user interface object 604. Alternatively, the vehicle shown by the user interface object 606 can be the vehicle itself that the user can see, rather than a user interface object. In another example, the first user interface object 604 is displayed as red, because the rule parameter indicates the "no-go zone" of the place where the vehicle is located.

[0104] Figure 6B illustrates additional information that can be provided to the user. Fig. 6A , the user will be able to see the road 602 through the windshield 600 of the vehicle. Based on the trajectory data, the system 500 can be configured to display an interface object indicating an available parking space on the roadside. For example, the system 500 is configured to display a first user interface object 652 indicating an available parking space, such as via color shading of the area. The first user interface object 652 can be green to indicate an available spot, or red, such as shown in the second user interface object 654 and the third user interface object 656, indicating an unavailable spot. The system 500 can be configured to use rule parameters to determine whether the available parking spot shown as the first user interface object 652 is a legal place to park. The trajectory data can indicate whether the available spot is a legal parking spot.

[0105] In one or more embodiments or examples, the device is one or more of the following: a display device, an augmented reality device, a user device, and a projection device. The specific device is not restrictive, and any type of device that can provide visual images to the user can be used. The device can be part of the vehicle 550. The display device can include a display on the vehicle, such as a dashboard display, etc. The device can be separated from the vehicle. The user device can be a personal device of a user that communicates data with the vehicle, such as a smart phone or tablet computer, etc.

[0106] In one or more embodiments or examples, causing the device to provide an output to the user based on the trajectory data includes projecting a user interface object onto a surface. For example, the system 500 projects the trajectory data 514 onto a surface of the vehicle 550. The system 500 may include, for example, a projector on a dashboard of the vehicle 550. In one or more embodiments or examples, the surface is one or more of a windshield of the vehicle 550 and a dashboard of the vehicle 550. If desired, the surface can be any surface that is visually accessible to the user while not hindering the user's ability to operate the vehicle 550. As an example, the surface is a surface in the user's field of view, such as a windshield, etc.

[0107] In one or more embodiments or examples, causing the device to provide an output to a user based on the trajectory data 514 includes causing the device to display to the user a second user interface object representing one of the plurality of trajectories 512. For example, the system 500 displays potential trajectories for the vehicle 550 using another user interface object 520. The second interface object may indicate a trajectory that the vehicle 550 may or may not take. Fig. 6A In the example, another user interface object may show a red track switching to the left lane with an approaching vehicle (as an example only, other indicators of a "bad" track may be used). Another user interface object may show a green track going straight (as an example only, other indicators of a "green" track may be used).

[0108] In one or more embodiments or examples, causing the device to provide output 514 to the user based on the trajectory data 514 includes causing the device to generate an audible signal based on an audio message associated with the trajectory data. The audio message can be an audio sound including one or more of a bell, a whistle, an alarm. The audio message can be a message including text, such as a voice-assisted message, etc. In some embodiments or examples, the system 500 utilizes a combination of audio notifications and visual notifications for the user.

[0109] In one or more examples or embodiments, such as when the vehicle 550 is an autonomous vehicle, the system 500 is configured to control the operation of the vehicle 550 based on the trajectory data 514 and / or the candidate trajectory 513. In some examples, the control operation includes generating control data for the control system of the autonomous vehicle. In some examples, the control operation includes providing the control data to the control system of the autonomous vehicle. The control operation may include transmitting the control data to, for example, the control system of the autonomous vehicle and / or an external system. The control operation may include controlling the control system and / or the external system of the autonomous vehicle based on the control data.

[0110] The system 500 can be particularly advantageous for L0-L5 autonomous vehicles. For example, the system 500 provides guidance to the driver and / or passengers without automation in an L0 vehicle. In some examples, for driver assistance (L1), partially autonomous (L2) vehicles, conditionally automated (L3) vehicles, highly automated (L4) vehicles, and fully autonomous (L5) vehicles, the system 500 provides guidance to the driver and / or passengers and also fully or partially operates the autonomous vehicle.

[0111] Reference now Figure 7 , which illustrates a flow chart of a method or process 800 for operating and / or controlling an autonomous vehicle and / or a non-autonomous vehicle, etc., for a system and method for a heads-up display. The method may be implemented by a system disclosed herein (such as Figure 4 AV Computing 400 or Figure 2 AV Computing 202F and Figure 1 and Figure 2 The method can be performed by a system disclosed herein (such as a vehicle 102, 200, etc.). Figure 5A-5D Calculation 540 and Figure 6A-6B The method may be performed by a system (such as a computer, etc.) of a vehicle that is not an autonomous vehicle. The disclosed system may include at least one processor that may be configured to perform one or more of the operations of method 800. Method 800 may be performed by another device or group of devices (e.g., completely and / or partially, etc.) separate from the system disclosed herein or including the system disclosed herein.

[0112] Method 800 is disclosed. Method 800 includes obtaining sensor data associated with an environment in which a vehicle is operating using at least one processor at step S802. Method 800 includes determining multiple trajectories for the vehicle based on the sensor data using at least one processor at step S804. Method 800 includes evaluating scores of trajectories in the multiple trajectories based on rule parameters using at least one processor at step S806, wherein the rule parameters indicate one or more rules for operating the vehicle. Method 800 includes generating trajectory data associated with candidate trajectories in the multiple trajectories based on the scores using at least one processor at step S808. Method 800 includes causing a device to provide an output to a user associated with the vehicle based on the trajectory data associated with the candidate trajectories using at least one processor at step S810. In one or more embodiments or examples, the multiple trajectories are candidate trajectories that a driver may select at a current point in time.

[0113] The one or more road rules may be passenger comfort rules and / or expression rules. For example, this includes legal and / or customary rules. Examples include areas where driving should not occur, whether lane changes are allowed, whether overtaking is allowed, where stopping, standing or parking is allowed, whether the driver needs to yield to cross traffic, and speed limits. Scores may be used to assess one or more of the safety and legality of the plurality of trajectories, such as for each of the plurality of trajectories.

[0114] Candidate trajectories may be considered as allowable potential trajectories that have been evaluated against the rule(s) indicated by the rule parameters. Trajectory data may include any data associated with the candidate trajectory, such as a depiction of areas where the trajectory is illegal. As for causing the device to provide an output, this may be directed to the driver and / or passengers of the vehicle. The output may inform the user whether some potential paths are illegal or poorly rated for specific reasons such as safety.

[0115] In one or more embodiments or examples, causing the device to provide output to the user based on the trajectory data at step S810 includes causing the device to display a user interface object representing the trajectory data to the user. The user interface object is, for example, one or more than one of an icon, a notification, a shadow, a color, a legitimacy, and a percentage. In one or more embodiments or examples, the device is one or more than one of the following: a display device, an augmented reality device, a user device, and a projection device. In one or more embodiments or examples, causing the device to provide output to the user based on the trajectory data at step S810 includes projecting the user interface object onto a surface. The device may be a mobile phone of a user of a vehicle or the vehicle itself. In one or more embodiments or examples, the surface is one or more than one of a windshield of a vehicle and a dashboard of a vehicle. The surface may be a surface in the user's field of view, such as a windshield. In one or more examples, the surface extends out of the dashboard (e.g., projected on an element (e.g., glass) attached to the dashboard, etc.). In one or more embodiments or examples, causing the device to provide an output to a user based on the trajectory data at step S810 includes causing the device to display to the user a second user interface object representing one of the plurality of trajectories. In one or more embodiments or examples, causing the device to provide an output to a user based on the trajectory data at step S810 includes causing the device to generate an audible signal based on an audio message associated with the trajectory data.

[0116] In one or more embodiments or examples, generating the trajectory data at step S808 includes generating the trajectory data based on the score and the sensor data. In one or more embodiments or examples, the user interface object includes a color-coded user interface object and / or a pictogram. The color-coded user interface object can be a semantic and / or regional depiction. The pictogram can be a representation of the environment. In one or more embodiments or examples, determining the plurality of trajectories at step S804 includes determining the plurality of trajectories every 30 seconds or less.

[0117] In the previous description, aspects and embodiments of the present disclosure have been described with reference to many specific details, which can vary between implementations. Therefore, the description and the accompanying drawings should be regarded as illustrative, rather than restrictive. The only and exclusive indication of the scope of the invention, and the applicant's expectation that the content of the scope of the invention is the literal and equivalent scope of such claims issued from this application in the specific form of issuing the claims, including any subsequent amendments. Any definition of the terms clearly set forth herein for being included in such claims should be based on the meaning of such terms as used in the claims. In addition, when the term "also includes" is used in the previous description or the appended claims, the following of the phrase can be an additional step or entity, or a sub-step / sub-entity of the previously described step or entity.

[0118] A non-transitory computer-readable medium is disclosed that includes instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform operations according to one or more of the methods disclosed herein.

[0119] Also disclosed are methods, non-transitory computer readable media, and systems according to any of the following:

[0120] Item 1. A method comprising:

[0121] obtaining, using at least one processor, sensor data associated with an environment in which the vehicle is operating;

[0122] determining, using the at least one processor, a plurality of trajectories for the vehicle based on the sensor data;

[0123] evaluating, using the at least one processor, a score for a trajectory in the plurality of trajectories based on a rule parameter, wherein the rule parameter indicates one or more rules for operating the vehicle;

[0124] generating, using the at least one processor, track data associated with a candidate track in the plurality of tracks based on the score; and

[0125] Using the at least one processor, an apparatus is caused to provide an output to a user associated with the vehicle based on the trajectory data associated with the candidate trajectory.

[0126] Item 2. The method of item 1, wherein causing the device to provide the output to the user based on the trajectory data comprises:

[0127] The device is caused to display a user interface object representing the trajectory data to a user.

[0128] Item 3. A method according to Item 2, wherein the device is one or more of the following: a display device, an augmented reality device, a user device, and a projection device.

[0129] Item 4. The method of any one of Items 2 to 3, wherein causing the apparatus to provide the output to a user based on the trajectory data comprises:

[0130] The user interface object is projected onto a surface.

[0131] Item 5. The method of Item 4, wherein the surface is one or more of a windshield of the vehicle and a dashboard of the vehicle.

[0132] Item 6. The method of any one of Items 2 to 5, wherein causing the apparatus to provide the output to a user based on the trajectory data comprises:

[0133] The apparatus is caused to display to a user a second user interface object representing one of the plurality of trajectories.

[0134] Item 7. A method according to any of the preceding items, wherein causing the device to provide the output to the user based on the trajectory data comprises:

[0135] The device is caused to generate an audible signal based on an audio message associated with the trajectory data.

[0136] Item 8. The method of any of the preceding items, wherein generating the trajectory data comprises generating the trajectory data based on the score and the sensor data.

[0137] Item 9. A method according to any one of items 2 to 8, wherein the user interface objects include color-coded user interface objects and / or pictograms.

[0138] Item 10. A method according to any of the preceding items, wherein determining the plurality of trajectories comprises determining the plurality of trajectories every 30 seconds or less.

[0139] Item 11. A non-transitory computer-readable medium comprising instructions, the instructions being stored on the non-transitory computer-readable medium, the instructions, when executed by at least one processor, causing the at least one processor to perform operations comprising:

[0140] obtaining sensor data associated with an environment in which the vehicle is operating;

[0141] determining a plurality of trajectories for the vehicle based on the sensor data;

[0142] evaluating scores of trajectories in the plurality of trajectories based on rule parameters, wherein the rule parameters indicate one or more rules for operating the vehicle;

[0143] generating trajectory data associated with a candidate trajectory in the plurality of trajectories based on the score; and

[0144] The device is caused to provide an output to a user associated with the vehicle based on the trajectory data associated with the candidate trajectory.

[0145] Item 12. The non-transitory computer-readable medium of Item 11, wherein causing the device to provide the output to a user based on the trajectory data comprises:

[0146] The device is caused to display a user interface object representing the trajectory data to a user.

[0147] Item 13. The non-transitory computer-readable medium of Item 12, wherein the device is one or more of: a display device, an augmented reality device, a user device, and a projection device.

[0148] Item 14. The non-transitory computer-readable medium of any one of Items 12 to 13, wherein causing the apparatus to provide the output to a user based on the trajectory data comprises:

[0149] The user interface object is projected onto a surface.

[0150] Item 15. The non-transitory computer-readable medium of Item 14, wherein the surface is one or more of a windshield of the vehicle and a dashboard of the vehicle.

[0151] Item 16. The non-transitory computer-readable medium of any one of Items 12 to 15, wherein causing the apparatus to provide the output to a user based on the trajectory data comprises:

[0152] The apparatus is caused to display to a user a second user interface object representing one of the plurality of trajectories.

[0153] Item 17. The non-transitory computer-readable medium of any one of Items 11 to 16, wherein causing the apparatus to provide the output to a user based on the trajectory data comprises:

[0154] The device is caused to generate an audible signal based on an audio message associated with the trajectory data.

[0155] Item 18. The non-transitory computer-readable medium of any one of Items 11 to 17, wherein generating the trajectory data comprises generating the trajectory data based on the score and the sensor data.

[0156] Item 19. The non-transitory computer-readable medium of any one of Items 12 to 18, wherein the user interface object comprises a color-coded user interface object and / or a pictogram.

[0157] Item 20. The non-transitory computer-readable medium of any one of Items 11 to 19, wherein determining the plurality of trajectories comprises determining the plurality of trajectories every 30 seconds or less.

[0158] Item 21. A system comprising at least one processor and at least one memory, wherein the at least one memory has stored thereon instructions, wherein when the instructions are executed by the at least one processor, the at least one processor performs an operation, wherein the operation comprises:

[0159] obtaining sensor data associated with an environment in which the vehicle is operating;

[0160] determining a plurality of trajectories for the vehicle based on the sensor data;

[0161] evaluating scores of trajectories in the plurality of trajectories based on rule parameters, wherein the rule parameters indicate one or more rules for operating the vehicle;

[0162] generating trajectory data associated with a candidate trajectory in the plurality of trajectories based on the score; and

[0163] The device is caused to provide an output to a user associated with the vehicle based on the trajectory data associated with the candidate trajectory.

[0164] Item 22. The system of Item 21, wherein causing the device to provide the output to the user based on the trajectory data comprises:

[0165] The device is caused to display a user interface object representing the trajectory data to a user.

[0166] Item 23. A system according to Item 22, wherein the device is one or more of the following: a display device, an augmented reality device, a user device, and a projection device.

[0167] Clause 24. The system of any one of clauses 22 to 23, wherein causing the device to provide the output to the user based on the trajectory data comprises:

[0168] The user interface object is projected onto a surface.

[0169] Item 25. The system of Item 24, wherein the surface is one or more of a windshield of the vehicle and a dashboard of the vehicle.

[0170] Clause 26. The system of any one of clauses 22 to 25, wherein causing the device to provide the output to the user based on the trajectory data comprises:

[0171] The apparatus is caused to display to a user a second user interface object representing one of the plurality of trajectories.

[0172] Clause 27. The system of any one of clauses 21 to 26, wherein causing the device to provide the output to the user based on the trajectory data comprises:

[0173] The device is caused to generate an audible signal based on an audio message associated with the trajectory data.

[0174] Item 28. The system of any one of Items 21 to 27, wherein generating the trajectory data comprises generating the trajectory data based on the score and the sensor data.

[0175] Item 29. A system according to any one of Items 22 to 28, wherein the user interface objects include color-coded user interface objects and / or pictograms.

[0176] Item 30. A system according to any one of Items 21 to 29, wherein determining the plurality of trajectories comprises determining the plurality of trajectories every 30 seconds or less.

Claims

1. A method comprising: obtaining, using at least one processor, sensor data associated with an environment in which the vehicle is operating; determining, using the at least one processor, a plurality of trajectories for the vehicle based on the sensor data; evaluating, using the at least one processor, a score for a trajectory in the plurality of trajectories based on a rule parameter, wherein the rule parameter indicates one or more rules for operating the vehicle; generating, using the at least one processor, track data associated with a candidate track in the plurality of tracks based on the score; and Using the at least one processor, an apparatus is caused to provide an output to a user associated with the vehicle based on the trajectory data associated with the candidate trajectory.

2. The method according to claim 1, wherein: Causing the apparatus to provide the output to the user based on the trajectory data comprises: The device is caused to display a user interface object representing the trajectory data to the user.

3. The method according to claim 2, wherein: The device is one or more of: a display device, an augmented reality device, a user device, and a projection device.

4. The method according to claim 2, wherein: Causing the apparatus to provide the output to the user based on the trajectory data comprises: The user interface object is projected onto a surface.

5. The method according to claim 4, wherein: The surface is one or more of a windshield of the vehicle and a dashboard of the vehicle.

6. The method according to claim 2, wherein: Causing the apparatus to provide the output to the user based on the trajectory data comprises: The apparatus is caused to display to the user a second user interface object representing one of the plurality of trajectories.

7. The method according to claim 1, wherein: Causing the apparatus to provide the output to the user based on the trajectory data comprises: The device is caused to generate an audible signal based on an audio message associated with the trajectory data.

8. The method according to claim 1, wherein: Generating the trajectory data includes generating the trajectory data based on the score and the sensor data.

9. The method according to claim 2, wherein: The user interface objects include color-coded user interface objects and / or pictograms.

10. The method according to claim 1, wherein: Determining the plurality of trajectories includes determining the plurality of trajectories every 30 seconds or less.

11. A system comprising at least one processor and at least one memory, wherein the at least one memory stores instructions, wherein when the instructions are executed by the at least one processor, the at least one processor performs an operation, wherein the operation comprises: obtaining sensor data associated with an environment in which the vehicle is operating; determining a plurality of trajectories for the vehicle based on the sensor data; evaluating scores of trajectories in the plurality of trajectories based on rule parameters, wherein the rule parameters indicate one or more rules for operating the vehicle; generating trajectory data associated with a candidate trajectory from the plurality of trajectories based on the score; as well as The device is caused to provide an output to a user associated with the vehicle based on the trajectory data associated with the candidate trajectory.

12. The system according to claim 11, wherein: Causing the apparatus to provide the output to the user based on the trajectory data comprises: The device is caused to display a user interface object representing the trajectory data to the user.

13. The system according to claim 12, wherein: The device is one or more of: a display device, an augmented reality device, a user device, and a projection device.

14. The system according to claim 12, wherein: Causing the apparatus to provide the output to the user based on the trajectory data comprises: The user interface object is projected onto a surface.

15. The system of claim 14, wherein: The surface is one or more of a windshield of the vehicle and a dashboard of the vehicle.

16. The system of claim 12, wherein: Causing the apparatus to provide the output to a user based on the trajectory data comprises: The apparatus is caused to display to the user a second user interface object representing one of the plurality of trajectories.

17. The system of claim 11, wherein: Causing the apparatus to provide the output to the user based on the trajectory data comprises: The device is caused to generate an audible signal based on an audio message associated with the trajectory data.

18. The system of claim 11, wherein: Generating the trajectory data includes generating the trajectory data based on the score and the sensor data.

19. The system of claim 11, wherein: Determining the plurality of trajectories includes determining the plurality of trajectories every 30 seconds or less.

20. A non-transitory computer-readable medium comprising instructions stored thereon, the instructions, when executed by at least one processor, causing the at least one processor to perform operations comprising: obtaining sensor data associated with an environment in which the vehicle is operating; determining a plurality of trajectories for the vehicle based on the sensor data; evaluating scores of trajectories in the plurality of trajectories based on rule parameters, wherein the rule parameters indicate one or more rules for operating the vehicle; generating trajectory data associated with a candidate trajectory from the plurality of trajectories based on the score; as well as The device is caused to provide an output to a user associated with the vehicle based on the trajectory data associated with the candidate trajectory.

Images