Methods, systems, storage media, and computer program products for a vehicle

Abstract

The present disclosure relates to methods, systems, and storage media for a vehicle. Methods for target determination using an eye tracker device and LiDAR point cloud data are provided, which can include receiving first data characterizing three-dimensional coordinates associated with a first location. The data is obtained via a sensor fixed to a vehicle. Some of the described methods further include receiving second data characterizing LiDAR point cloud data obtained from a LiDAR device fixed to the vehicle. The LiDAR point cloud data includes three-dimensional coordinates associated with the first location. A visual indication of the first location can be provided on a user interface of the vehicle. The visual indication can be generated based on the first data and the second data. In response to a user input to select the visual indication, the vehicle can be operated to navigate to the first location. Systems and computer program products are also provided.

Description

Technical Field

[0001] This disclosure relates to target determination using an eye-tracking device and LiDAR point cloud data. Background Technology

[0002] Autonomous vehicles can sense their surroundings and navigate to a target location with minimal or no human input. To safely traverse the chosen path while avoiding potential obstacles, the vehicle can rely on various types of sensor data. For example, LiDAR (Light Detection and Ranging) sensor data can include 3D data in the form of point clouds. Sensor data can also include data associated with a user's gaze or fixation, which can be acquired using eye-tracking devices and corresponding cameras mounted on the vehicle. Determining the target location based on the user's gaze and displaying it in the user interface can be computationally intensive and requires a specialized user interface. Summary of the Invention

[0003] According to one aspect of this disclosure, a method for a vehicle is provided, comprising: using at least one processor to receive first data representing three-dimensional coordinates (e.g., coordinates of a destination location as seen by a driver) associated with a first location, the first data being obtained via at least one sensor (eye tracker device) fixed to the vehicle (in the vehicle's cockpit); using the at least one processor to receive second data representing LiDAR point cloud data obtained from at least one LiDAR device fixed to the vehicle, the LiDAR point cloud data including three-dimensional coordinates associated with the first location; using the at least one processor to provide a visual indication of the first location on a user interface of the vehicle, the visual indication being generated based on the first data and the second data; and using the at least one processor to operate the vehicle to navigate to the first location in response to user input for selecting the visual indication.

[0004] In the above method, generating the visual indication further includes: using the at least one processor to determine the third data representing the first location as a superposition of the three-dimensional coordinates and the LiDAR point cloud data (a superposition of eye tracker data and point cloud data); using the at least one processor to map the third data with fourth data representing map data, the map data including the first location and one or more second locations representing marked safety points (the map data includes marked safety points); and using the at least one processor to determine the first location based on at least one of the one or more second locations included in the fourth data (at least one safety point among the safety points included in the map data).

[0005] In the above method, the one or more second locations include at least one loading location and at least one unloading location.

[0006] In the above method, operating the vehicle to navigate to the first location further includes: generating a trajectory toward the first location from the current location of the vehicle toward the first location using a planning system based on user input for selecting the visual indication; and operating the vehicle to navigate to the first location based on the trajectory.

[0007] In the above method, the at least one sensor is included in a plurality of sensors fixed to the vehicle and configured to transmit field-of-view data to the at least one processor.

[0008] In the above method, the at least one sensor is configured to track the eye movements of the user locating the first location.

[0009] In the above method, the user input is received as a gesture of the user regarding the location of the first location, as observed by the at least one sensor.

[0010] According to another aspect of this disclosure, a system for a vehicle is provided, comprising: at least one sensor attached to the vehicle; at least one LiDAR device attached to the vehicle; at least one processor; and at least one non-transitory storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the method described above.

[0011] According to another aspect of this disclosure, at least one non-transitory storage medium is provided, which stores instructions that, when executed by at least one processor, cause the at least one processor to perform the method described above. Attached Figure Description

[0012] Figure 1 It is an example environment that can realize a vehicle that includes one or more components of an autonomous system;

[0013] Figure 2 It is a diagram of one or more systems that include autonomous vehicles;

[0014] Figure 3 yes Figure 1 and Figure 2 A diagram of one or more devices and / or one or more system components;

[0015] Figure 4A It is a diagram of some components of an autonomous system;

[0016] Figure 4B This is a diagram illustrating the implementation of a neural network;

[0017] Figure 5 This is a diagram of the implementation of a target determination system configured to perform target determination using eye-tracking devices and LiDAR point cloud data.

[0018] Figure 6 This is a diagram showing the detailed implementation of a target determination system configured to use an eye-tracking device and LiDAR point cloud data for target identification; and

[0019] Figure 7 This is a flowchart of the process for target identification using an eye-tracking device and LiDAR point cloud data. Detailed Implementation

[0020] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent that the embodiments described herein can be practiced without these specific details. In some instances, well-known constructions and apparatuses are illustrated in block diagram form to avoid unnecessarily obscuring aspects of this disclosure.

[0021] In the accompanying drawings, for ease of description, specific arrangements or orders of schematic elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements) are illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or separation of processes. Furthermore, unless explicitly described, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0022] Furthermore, in the accompanying drawings, connecting elements (such as solid or dashed lines or arrows) are used to illustrate connections, relationships, or associations between or among two or more other schematic elements. The absence of any such connecting element does not imply that connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the content of this disclosure. Additionally, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents communication of signals, data, or instructions (e.g., "software instructions"), those skilled in the art will understand that such an element may represent one or more signal paths (e.g., a bus) that may be necessary to influence the communication.

[0023] Although the terms "first," "second," and / or "third," etc., are used to describe various elements, these elements should not be limited by these terms. The terms "first," "second," and / or "third" are used only to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0024] The terminology used in the description of the various embodiments described herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms and may be used interchangeably with “one or more” or “at least one” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms “comprising,” “including,” “possessing,” and / or “having” are used in this specification, they specifically indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0025] As used herein, the terms "communication" and "to communicate" refer to at least one of the following: receiving, receiving, transmitting, conveying, and / or providing information (or information represented by, for example, data, signals, messages, instructions, and / or commands). For a unit (e.g., an apparatus, system, component of an apparatus or system, and / or combinations thereof) that wants to communicate with another unit, this means that the unit is able to receive information directly or indirectly from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is essentially wired and / or wireless. Furthermore, two units can communicate with each other even if the transmitted information can be modified, processed, relayed, and / or routed between the first and second units. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively transmit information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediary unit (e.g., a third unit located between the first and second units) processes information received from the first unit and transmits the processed information to the second unit. In some embodiments, a message may refer to a network packet that includes data (e.g., a data packet, etc.).

[0026] As used herein, depending on the context, the term "if" may optionally be interpreted as "when," "in," "in response to being determined," and / or "in response to being detected," etc. Similarly, depending on the context, the phrases "if determined" or "if [the stated condition or event] is detected" may optionally be interpreted as "in response to being determined," "in response to being determined," "or" "in response to being detected," and / or "in response to being detected," etc. Furthermore, as used herein, the terms "have," "possess," or "own," etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least partially based on."

[0027] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0028] General Overview

[0029] An eye tracker device and corresponding camera setup can be positioned on the vehicle to capture the user's gaze. The location of the user's gaze can be identified and provided as the target location or destination point within the vehicle's user interface. The target location can be overlaid on a representation of the environment in which the vehicle is operating. The user can select the target location, and the vehicle can determine its trajectory to that location. The target location can also be determined with reference to predetermined safe locations (such as pick-up and unload (e.g., PuDo) locations in LiDAR point cloud data). Other gestures (such as pointing) can also be used to determine the target location.

[0030] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement techniques for determining a target location using an eye-tracking device and LiDAR point cloud data. The target location or destination can be determined based on eye-tracking data acquired by sensors such as an eye-tracking device positioned in a vehicle and configured to observe the driver of the vehicle. The eye-tracking data may include three-dimensional coordinate data associated with the target location viewed by the driver. The technique may also include using a LiDAR device fixed to the vehicle to obtain LiDAR point cloud data. The LiDAR point cloud data may include three-dimensional coordinate data associated with the target location. Visual indications of the target location can be provided in the user interface of the vehicle. Visual indications can be determined based on eye-tracking data and LiDAR point cloud data. A user can interact with the user interface to select a target location as the destination. The technique may also include operating the vehicle to navigate to the target location based on user selections provided to the user interface.

[0031] This paper provides a technique for target determination using eye-tracking devices and LiDAR point cloud data, utilizing the systems, methods, and computer program products described herein. Determining target locations using eye-tracking devices and LiDAR point cloud data offers users an interactive way to identify and select specific pick-up and unload (PuDo) locations in an ad-hoc manner. This technique also provides vehicle operators and / or passengers with an efficient and secure engagement mechanism to interact with vehicle navigation and route planning systems. Integration with known safe zones or destination stations (such as PuDo locations) in the LiDAR point cloud data scene or local map can improve safe navigation to selected target locations. Eye-tracking data can be stored and used to train trajectory predictions for specific locations.

[0032] Now for reference Figure 1Example environment 100 is illustrated, in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118. Vehicles 102a-102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 are interconnected via wired connections, wireless connections, or a combination of wired and wireless connections (e.g., establishing connections for communication, etc.). In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 via wired connection, wireless connection, or a combination of wired and wireless connection.

[0033] Vehicles 102a-102n (specifically referred to as vehicle 102 and collectively as vehicle 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicle 102 is configured to communicate with V2I device 110, remote AV system 114, queue management system 116 and / or V2I system 118 via network 112. In some embodiments, vehicle 102 includes cars, buses, trucks and / or trains, etc. In some embodiments, vehicle 102 is associated with vehicle 200 described herein (see Figure 2 The vehicles 102 are the same as or similar to autonomous vehicles 202. In some embodiments, vehicles 200 in a group of vehicles 200 are associated with an autonomous queue manager. In some embodiments, as described herein, vehicles 102 travel along corresponding routes 106a-106n (each individually referred to as route 106 and collectively as route 106). In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0034] Objects 104a-104n (each individually referred to as object 104 and collectively as object 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.). Each object 104 (e.g., located at a fixed location and for a period of time) is either stationary or (e.g., having a speed and associated with at least one trajectory) moving. In some embodiments, object 104 is associated with a corresponding location in area 108.

[0035] Routes 106a-106n (each individually referred to as Route 106 and collectively as Route 106) are each associated with a series of actions (also referred to as trajectories) along which the connecting AV can navigate. Each Route 106 begins with an initial state (e.g., a state corresponding to a first spatiotemporal location and / or velocity, etc.) and ends with a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target area (e.g., a subspace of an acceptable state (e.g., a termination state)). In some embodiments, a first state includes a location where one or more individuals will pick up the AV, and a second state or area includes a location where one or more individuals carrying the AV will be unloaded. In some embodiments, Route 106 includes multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) associated with multiple trajectories (e.g., defining multiple trajectories). In the example, Route 106 includes only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection. Additionally or alternatively, route 106 may include more precise actions or states, such as, for example, specific target lanes or precise locations within a lane area and target rates at those locations. In the example, route 106 includes multiple precise state sequences along at least one high-level action with a finite look-ahead horizon leading to an intermediate target, wherein the cumulative combination of successive iterations of the finite horizon state sequences corresponds to multiple trajectories that collectively form a high-level route terminating at a final target state or region.

[0036] Region 108 includes a physical area (e.g., a geographic region) that the vehicle 102 can navigate. In examples, region 108 includes at least one state (e.g., a country, a province, a single state among multiple states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, region 108 includes at least one named arterial road (referred to herein as a "road"), such as a highway, interstate highway, park road, city street, etc. Additionally or alternatively, in some examples, region 108 includes at least one unnamed road, such as a driveway, a section of a parking lot, a section of vacant land and / or undeveloped area, dirt road, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In examples, a road includes at least one lane associated with at least one lane marking (e.g., identified based on at least one lane marking). Region 108 may include pick-up and unload (PuDo) locations that serve as safe places for vehicle occupants to enter or leave the vehicle 102.

[0037] The Vehicle-to-Infrastructure (V2I) device 110 (sometimes referred to as a Vehicle-to-Everything (V2X) device) includes at least one device configured to communicate with vehicle 102 and / or V2I system 118. In some embodiments, V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, queue management system 116, and / or V2I system 118 via network 112. In some embodiments, V2I device 110 includes radio frequency identification (RFID) devices, signs, cameras (e.g., two-dimensional (2D) and / or three-dimensional (3D) cameras), lane markings, streetlights, parking meters, etc. In some embodiments, V2I device 110 is configured to communicate directly with vehicle 102. Additionally or alternatively, in some embodiments, V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, and / or queue management system 116 via V2I system 118. In some embodiments, V2I device 110 is configured to communicate with V2I system 118 via network 112.

[0038] Network 112 includes one or more wired and / or wireless networks. In the example, network 112 includes cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMNs), Local Area Networks (LANs), Wide Area Networks (WANs), Metropolitan Area Networks (MANs), telephone networks (e.g., Public Switched Telephone Networks (PSTN)), private networks, self-organizing networks, intranets, the Internet, fiber-based networks, cloud computing networks, etc., and / or combinations of some or all of these networks.

[0039] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, V2I device 110, network 112, queue management system 116, and / or V2I system 118 via network 112. In examples, the remote AV system 114 includes a server, server group, and / or other similar devices. In some embodiments, the remote AV system 114 is located in the same location as the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle, including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing. In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the lifespan of the vehicle.

[0040] The queue management system 116 includes at least one device configured to communicate with vehicle 102, V2I device 110, remote AV system 114, and / or V2I system 118. In examples, the queue management system 116 includes servers, server groups, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ride-sharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including autonomous systems and / or vehicles not including autonomous systems)).

[0041] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the queue management system 116 via a network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or private entity (e.g., a private entity maintaining the V2I device 110).

[0042] supply Figure 1The number and arrangement of the elements are shown as examples. (and) Figure 1 Compared to the illustrated elements, there may be additional elements, fewer elements, different elements, and / or elements arranged differently. Additionally or alternatively, at least one element of environment 100 may be described as being composed of… Figure 1 At least one different element of environment 100 performs one or more functions. Additionally or alternatively, at least one set of elements of environment 100 may perform one or more functions described as being performed by at least one different set of elements of environment 100. In some embodiments, target determination system 505 may be included in environment 100. Target determination system 505 may be configured within or outside vehicle 102. In some embodiments, a first portion of target determination system 505 may be configured within vehicle 102, and a second portion of target determination system 505 may be configured outside vehicle 102.

[0043] Now for reference Figure 2 The vehicle 200 includes an autonomous system 202, a powertrain control system 204, a steering control system 206, and a braking system 208. In some embodiments, the vehicle 200 and the vehicle 102 (see...) Figure 1 The vehicle 200 is similar to or the same as the vehicle in question. In some embodiments, the vehicle 200 has autonomous capabilities (e.g., implementing at least one function, feature, and / or device that enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon human intervention) and / or highly autonomous vehicles (e.g., vehicles that abandon human intervention in certain situations)). For a detailed description of fully autonomous and highly autonomous vehicles, refer to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated herein by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ride-sharing company.

[0044] Autonomous system 202 includes a sensor suite comprising one or more devices such as camera 202a, LiDAR sensor 202b, radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance traveled by vehicle 200). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 as described herein. The data generated by one or more devices of autonomous system 202 may be used by one or more systems as described herein to observe the environment in which vehicle 200 is located (e.g., environment 100). In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, and safety controller 202g.

[0045] Camera 202a includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one means of communicating with the same or similar bus as bus 302. Camera 202a includes at least one camera (e.g., a digital camera using a light sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) for capturing images of physical objects (e.g., cars, buses, curbs, and / or people, etc.). In some embodiments, camera 202a generates camera data as output. In some examples, camera 202a generates camera data including image data associated with an image. In this example, the image data may specify at least one parameter corresponding to the image (e.g., image characteristics such as exposure, brightness, etc., and / or image timestamp, etc.). In such examples, the image may be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, camera 202a includes multiple independent cameras configured (e.g., positioned on) a vehicle to capture images for stereoscopic imaging (stereoscopic vision). In some examples, camera 202a includes generating image data and transmitting the image data to an autonomous vehicle computing 202f and / or a queue management system (e.g., with...). Figure 1The queue management system 116 (same as or similar to a queue management system) has multiple cameras. In such an example, the autonomous vehicle calculation 202f determines the depth of one or more objects in the fields of view of at least two of the multiple cameras based on image data from at least two cameras. In some embodiments, camera 202a is configured to capture images of objects within a distance relative to camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Therefore, camera 202a includes features such as sensors and lenses optimized for sensing objects at one or more distances relative to camera 202a.

[0046] In embodiments, camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects providing visual navigation information. In some embodiments, camera 202a generates traffic light data associated with one or more images. In some examples, camera 202a generates TLD data associated with one or more images, including formats such as RAW, JPEG, and / or PNG. In some embodiments, camera 202a, which generates TLD data, differs from other systems containing cameras described herein in that camera 202a may include one or more cameras with a wide field of view (e.g., a wide-angle lens, a fisheye lens, and / or a lens with an angle of view of about 120 degrees or greater) to generate images associated with as many physical objects as possible.

[0047] In some embodiments, camera 202a may include an eye-tracking device 202a configured to observe the driver of a vehicle. For example, the eye-tracking device 202a may be fixed inside the vehicle's cockpit and may be configured on the driver's face, particularly over the driver's eyes. The eye-tracking device 202a can capture image data of the driver's eyes and can generate three-dimensional coordinate data associated with the location the driver is looking at and corresponding to a navigation target location. The accuracy of the eye-tracking device 202a can be calibrated, tuned, or trained.

[0048] The laser detection and ranging (LiDAR) sensor 202b includes components configured to communicate with a communication device 202e, an autonomous vehicle computing unit 202f, and / or a safety controller 202g via a bus (e.g., with...). Figure 3At least one device that communicates with the same or similar bus (bus 302). The LiDAR sensor 202b includes a system configured to emit light from a emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object it encounters. The LiDAR sensor 202b also includes at least one photosensor that detects the light after it has encountered a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., point cloud and / or combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of a physical object and / or the surface of the physical object (e.g., the topology of the surface). In such examples, the image is used to determine the boundaries of the physical object within the field of view of the LiDAR sensor 202b.

[0049] The radio detection and ranging (radar) sensor 202c includes components configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus (bus 302). The radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by the radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves emitted by the radar sensor 202c encounter a physical object and are reflected back to the radar sensor 202c. In some embodiments, the radio waves emitted by the radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with the radar sensor 202c generates a signal representing objects included in the field of view of the radar sensor 202c. For example, at least one data processing system associated with the radar sensor 202c generates an image representing the boundaries of physical objects and / or the surfaces of physical objects (e.g., surface topology). In some examples, this image is used to determine the boundaries of physical objects in the field of view of the radar sensor 202c.

[0050] Microphone 202d includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3At least one device that communicates with the same or similar bus as bus 302. Microphone 202d includes one or more microphones (e.g., array microphones and / or external microphones, etc.) that capture audio signals and generate data associated with (e.g., representing) the audio signals. In some examples, microphone 202d includes transducer devices and / or similar devices. In some embodiments, one or more systems described herein can receive data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on the audio signal associated with the data.

[0051] The communication device 202e includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, an autonomous vehicle computing system 202f, a safety controller 202g, and / or a drive-by-wire (DBW) system 202h. For example, the communication device 202e may include communication with… Figure 3 The communication device 202e is the same as or similar to the communication interface 314. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles).

[0052] The autonomous vehicle computing 202f includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, a communication device 202e, a security controller 202g, and / or a DBW system 202h. In some examples, the autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablets) and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units). In some embodiments, the autonomous vehicle computing 202f is the same as or similar to the autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, the autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with...). Figure 1 Remote AV systems 114 are the same as or similar to autonomous vehicle systems), queue management systems (e.g., with...). Figure 1 The queue management system 116 is the same as or similar to the queue management system 116), and V2I devices (e.g., with Figure 1 V2I devices (same as or similar to V2I devices 110) and / or V2I systems (e.g., with V2I devices 110) Figure 1 The V2I system 118 communicates with the same or similar V2I system.

[0053] The safety controller 202g includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, a communication device 202e, an autonomous vehicle computing system 202f, and / or a DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) the control signals generated and / or transmitted by the autonomous vehicle computing system 202f.

[0054] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200, including one or more devices (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device (e.g., turn signals, headlights, door locks, and / or windshield wipers, etc.) of the vehicle 200.

[0055] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate in a certain direction, decelerate in a certain direction, make a left turn and / or make a right turn, etc. In examples, the powertrain control system 204 increases, keeps the same, or decreases the energy (e.g., fuel and / or electricity, etc.) supplied to the motor of the vehicle, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate.

[0056] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 causes the two front wheels and / or the two rear wheels of the vehicle 200 to turn left or right, thereby causing the vehicle 200 to turn left or right.

[0057] The braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or keep the vehicle 200 stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to close one or more calipers associated with one or more wheels of the vehicle 200 on the respective rotor of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0058] In some embodiments, the vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring the nature of the state or conditions of the vehicle 200. In some examples, the vehicle 200 includes platform sensors such as a Global Positioning System (GPS) receiver, an Inertial Measurement Unit (IMU), a wheel rate sensor, a wheel brake pressure sensor, a wheel torque sensor, an engine torque sensor, and / or a steering angle sensor.

[0059] Now for reference Figure 3 A schematic diagram of device 300 is illustrated. As illustrated, device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, device 300 corresponds to: at least one device of vehicle 102 (e.g., at least one device of the system of vehicle 102); at least one device of target determination system 505; and / or one or more devices of network 112 (e.g., one or more devices of the system of network 112). In some embodiments, one or more devices of vehicle 102 (e.g., one or more devices of the system of vehicle 102), one or more devices of target determination system 505, and / or one or more devices of network 112 (e.g., one or more devices of the system of network 112) include at least one device 300 and / or at least one component of device 300. Figure 3 As shown, the device 300 includes a bus 302, a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0060] Bus 302 includes components for communication between the components of the licensed device 300. In some embodiments, processor 304 is implemented in hardware, software, or a combination of hardware and software. In some examples, processor 304 includes a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), and / or accelerated processing unit (APU), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field-programmable gate array (FPGA) and / or application-specific integrated circuit (ASIC), etc.). Memory 306 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory, etc.) that stores data and / or instructions for use by processor 304.

[0061] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, CD-ROMs, RAM, PROMs, EPROMs, FLASH-EPROMs, NV-RAMs, and / or other types of computer-readable media, and corresponding drives.

[0062] Input interface 310 includes components that enable the device 300 to receive information, such as via user input (e.g., a touchscreen display, keyboard, keypad, mouse, buttons, switches, microphone, and / or camera). Additionally or alternatively, in some embodiments, input interface 310 includes sensors for sensing information (e.g., a Global Positioning System (GPS) receiver, accelerometer, gyroscope, and / or actuator). Output interface 312 includes components for providing output information from device 300 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0063] In some embodiments, the communication interface 314 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable the licensing device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, the communication interface 314 enables the licensing device 300 to receive information from and / or provide information to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, etc. Interfaces and / or cellular network interfaces, etc.

[0064] In some embodiments, device 300 performs one or more of the processes described herein. Device 300 performs these processes based on software instructions stored in a computer-readable medium, such as memory 306 and / or storage component 308, executed by processor 304. Computer-readable medium (e.g., non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0065] In some embodiments, software instructions are read from another computer-readable medium or from another device via communication interface 314 into memory 306 and / or storage component 308. When executed, the software instructions stored in memory 306 and / or storage component 308 cause processor 304 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, unless explicitly stated otherwise, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0066] The memory 306 and / or storage component 308 include a data storage unit or at least one data structure (e.g., a database). The device 300 is capable of receiving information from the data storage unit or at least one data structure in the memory 306 or storage component 308, storing the information in the data storage unit or at least one data structure, communicating information to the data storage unit or at least one data structure, or searching for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0067] In some embodiments, device 300 is configured to execute software instructions stored in the memory of memory 306 and / or another device (e.g., another device identical or similar to device 300). As used herein, the term "module" refers to at least one instruction stored in the memory of memory 306 and / or the other device, which, when executed by the processor of processor 304 and / or the processor of another device (e.g., another device identical or similar to device 300), causes device 300 (e.g., at least one component of device 300) to perform one or more processes as described herein. In some embodiments, modules are implemented in software, firmware, and / or hardware, etc.

[0068] supply Figure 3 The number and arrangement of components are illustrated as examples. In some embodiments, with Figure 3Compared to the illustrated components, device 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a group of components of device 300 (e.g., one or more components) may perform one or more functions described as being performed by another component or another group of components of device 300.

[0069] Now for reference Figure 4A The diagram illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack"). As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in and / or implemented in the vehicle's automatic navigation system (e.g., the autonomous vehicle computing 202f of vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more separate systems (e.g., one or more systems that are the same as or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 41 are included in one or more independent systems located within the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in memory), computer hardware (e.g., via microprocessors, microcontrollers, application-specific integrated circuits (ASICs), and / or field-programmable gate arrays (FPGAs), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with remote systems (e.g., autonomous vehicle systems identical or similar to remote AV system 114, queue management systems identical or similar to queue management systems 116, and / or V2I systems identical or similar to V2I system 118, etc.).

[0070] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a) that is associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of physical objects by the perception system 402, the perception system 402 transmits data associated with the classification of the physical objects to the planning system 404.

[0071] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data from the sensing system 402 (e.g., the data associated with the classification of physical objects described above), and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the sensing system 402. In some embodiments, the planning system 404 receives data associated with the updated location of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0072] In some embodiments, positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and positioning system 406 generates a composite point cloud based on the individual point clouds. In these examples, positioning system 406 compares the at least one point cloud or composite point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in database 410. Then, based on the comparison of the at least one point cloud or composite point cloud with the map, positioning system 406 determines the location of the vehicle in the area. In some embodiments, the map includes a composite point cloud of the area generated prior to navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of the geometry of the roadway, a map describing the connectivity of the road network, a map describing the physical properties of the roadway (such as traffic speed, traffic flow, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or the type and location of lane markings, or combinations thereof), and a map describing the spatial locations of road features (such as pedestrian crossings, traffic signs, or various types of other traffic lights). In some embodiments, the map is generated in real time based on data received by the sensing system.

[0073] In another example, positioning system 406 receives Global Navigation Satellite System (GNSS) data generated by a Global Positioning System (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with the location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such examples, positioning system 406 determines the location of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, positioning system 406 generates data associated with the location of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the location of the vehicle. In such examples, the data associated with the location of the vehicle includes data associated with one or more semantic properties corresponding to the location of the vehicle.

[0074] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate the powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), the steering control system (e.g., steering control system 206), and / or the braking system (e.g., braking system 208). In an example, where the trajectory includes a left turn, the control system 408 transmits control signals to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to change the state of other devices of the vehicle 200 (e.g., headlights, turn signals, door locks, and / or windshield wipers, etc.).

[0075] In some embodiments, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and / or target determination system 505 implement at least one machine learning model individually or in combination with one or more of the aforementioned systems. In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and / or target determination system 505 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in the environment, etc.). The following is about Figure 4B Includes examples of machine learning model implementations.

[0076] Database 410 stores data transmitted to, received from, and / or updated by the sensing system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes storage components for storing operation-related data and / or software, and for computing 400 using autonomous vehicles (e.g., with...). Figure 3(The storage component 308 is the same as or similar to the storage component 308). In some embodiments, database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., countries). In such examples, a vehicle (e.g., the same as or similar to vehicle 102 and / or vehicle 200) can drive along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and causes at least one LiDAR sensor (e.g., the same as or similar to LiDAR sensor 202b) to generate data associated with images representing objects included in the field of view of the at least one LiDAR sensor.

[0077] In some embodiments, database 410 may be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle identical or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system identical or similar to remote AV system 114), and a queue management system (e.g., with...). Figure 1 Queue management system 116 (same as or similar to queue management system) and / or V2I system (e.g., with Figure 1 Among the V2I systems (118 similar to or similar V2I systems), etc.

[0078] Now for reference Figure 4B The diagram illustrates an implementation of a machine learning model. More specifically, it illustrates an implementation of a convolutional neural network (CNN) 420. For illustrative purposes, the following description of CNN 420 will concern the implementation of CNN 420 via a perception system 402. However, it will be understood that in some examples, CNN 420 (e.g., one or more components of CNN 420) is implemented by systems other than or besides perception system 402 (such as planning system 404, localization system 406, control system 408, and / or target determination system 505, etc.). Although CNN 420 includes certain features as described herein, these features are provided for illustrative purposes and are not intended to limit this disclosure.

[0079] CNN 420 includes multiple convolutional layers comprising a first convolutional layer 422, a second convolutional layer 424, and a convolutional layer 426. In some embodiments, CNN 420 includes a subsampling layer 428 (sometimes referred to as a pooling layer). In some embodiments, subsampling layer 428 and / or other subsampling layers have a dimension smaller than that of the upstream system (i.e., the number of nodes). By means of subsampling layer 428 having a dimension smaller than that of the upstream layers, CNN 420 combines the amount of data associated with the initial input and / or output of the upstream layers, thereby reducing the computational cost required for downstream convolution operations by CNN 420. Additionally or alternatively, by means of subsampling layer 428 associated with at least one subsampling function (e.g., configured to perform at least one subsampling function), CNN 420 combines the amount of data associated with the initial input.

[0080] Based on the perception system 402 providing corresponding inputs and / or outputs associated with each of the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426 to generate corresponding outputs, the perception system 402 performs convolution operations. In some examples, based on the perception system 402 providing data as input to the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426, the perception system 402 implements a CNN 420. In such examples, based on the perception system 402 receiving data from one or more different systems (e.g., one or more systems of a vehicle identical or similar to vehicle 102, a remote AV system identical or similar to remote AV system 114, a queue management system identical or similar to queue management system 116, and / or a V2I system identical or similar to V2I system 118, etc.), the perception system 402 provides data as input to the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426.

[0081] In some embodiments, the perception system 402 provides data associated with an input (referred to as initial input) to a first convolutional layer 422, and the perception system 402 uses the first convolutional layer 422 to generate data associated with an output. In some embodiments, the perception system 402 provides the output generated by the convolutional layer as input to different convolutional layers. For example, the perception system 402 provides the output of the first convolutional layer 422 as input to a subsampling layer 428, a second convolutional layer 424, and / or a convolutional layer 426. In such an example, the first convolutional layer 422 is referred to as the upstream layer, and the subsampling layer 428, the second convolutional layer 424, and / or the convolutional layer 426 are referred to as downstream layers. Similarly, in some embodiments, the perception system 402 provides the output of the subsampling layer 428 to the second convolutional layer 424 and / or the convolutional layer 426, and in this example, the subsampling layer 428 will be referred to as the upstream layer, and the second convolutional layer 424 and / or the convolutional layer 426 will be referred to as the downstream layer.

[0082] In some embodiments, before providing input to the CNN 420, the perception system 402 processes the data associated with the input provided to the CNN 420. For example, the perception system 402 processes the data associated with the input provided to the CNN 420 based on the normalization of sensor data (e.g., image data, LiDAR data, and / or radar data, etc.) by the perception system 402.

[0083] In some embodiments, the perception system 402 generates an output by performing convolution operations associated with each convolutional layer based on the CNN 420. In some examples, the CNN 420 generates an output by performing convolution operations associated with each convolutional layer and an initial input based on the perception system 402. In some embodiments, the perception system 402 generates an output and provides that output as a fully connected layer 430. In some examples, the perception system 402 provides the output of a convolutional layer 426 as a fully connected layer 430, wherein the fully connected layer 430 includes data associated with multiple feature values ​​referred to as F1, F2, ..., FN. In this example, the output of the convolutional layer 426 includes data associated with multiple output feature values ​​representing a prediction.

[0084] In some embodiments, the perception system 402 identifies a prediction from among a plurality of predictions based on a feature value identified as the highest probability of being the correct prediction among multiple predictions. For example, if the fully connected layer 430 includes feature values ​​F1, F2, ..., FN and F1 is the largest feature value, the perception system 402 identifies the prediction associated with F1 as the correct prediction among multiple predictions. In some embodiments, the perception system 402 trains the CNN 420 to generate predictions. In some examples, the perception system 402 trains the CNN 420 to generate predictions based on training data associated with predictions provided to the CNN 420.

[0085] Now for reference Figure 5A diagram illustrates an implementation 500 of target determination processing using eye trackers and LiDAR point cloud data. In some embodiments, implementation 500 includes a target determination system 505, vehicles 102a-102n and / or vehicle 200, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116, and / or V2I system 118. In some embodiments, the target determination system 505 includes vehicles 102a-102n and / or vehicles 200, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) devices 110, network 112, remote autonomous vehicle (AV) systems 114, queue management systems 116 and / or V2I systems 118. The target determination system 505 forms a network of vehicles 102a-102n and / or vehicles 200, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) devices 110, network 112, remote autonomous vehicle (AV) systems 114, queue management systems 116 and / or V2I systems 118. A vehicle-to-infrastructure (V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116 and / or V2I system 118, a target determination system 505 is coupled to and / or uses vehicles 102a-102n and / or vehicles 200, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116 and / or V2I system 118.

[0086] like Figure 5 As shown, implementation 500 may include a target determination system 505. The target determination system 505 may include a target determiner 510, configured to acquire and process eye-tracking data from an eye-tracker device fixed to the vehicle and LiDAR point cloud data from a LiDAR device fixed to the vehicle. The eye-tracker data and LiDAR point cloud data can be used to generate visual indications of a target location or destination corresponding to the location being observed or focused on by the vehicle operator. Visual indications can be provided on the vehicle's user interface for user interaction. The user can provide selections of visual indications via audible commands, gestures, or touchscreen input. A trajectory 515 can be determined by the target determiner 510 as a result of the selection. The trajectory 515 can be provided to the planning system 404, and the vehicle can be navigated toward the target location based on the trajectory 515.

[0087] Now for reference Figure 6 , showed Figure 5The goal is to define a detailed implementation diagram of the system. For example... Figure 6 As shown, the target determination system 505 may include a target determiner 510. The target determiner 510 may receive eye-tracking data 520 and LiDAR point cloud data 525. The eye-tracking device 202a may acquire data 520 corresponding to locations that the vehicle driver may wish to navigate to. The driver may observe target locations or destinations in the environment in which the vehicle is operating, and the eye-tracking device 202a may generate eye-tracking data 520 corresponding to the target locations or destinations observed by the vehicle driver. The eye-tracking data 520 may include three-dimensional coordinate data corresponding to the target locations or destinations.

[0088] LiDAR point cloud data 525 can also be acquired by LiDAR sensor 202b configured on the vehicle. LiDAR point cloud data 525 can provide a reference environment corresponding to the environment in which the vehicle is operating, and eye-tracking data 520 can be registered or processed based on the reference provided by LiDAR point cloud data 525. LiDAR point cloud data 525 may include three-dimensional coordinate data corresponding to a target location or destination.

[0089] Eye-tracking data 520 and LiDAR point cloud data 525 can be processed by a 3D locator 530 to generate an overlay 535 of the LiDAR point cloud data 525, which is associated with the target location and included in the eye-tracking data 520. The overlay 535 can be set in a user interface 540 that can be configured in the vehicle. The overlay 535 can be cross-referenced with map data 545 generated by a mapper 550. The mapper 550 can be configured to determine and / or store one or more safe locations (PuDo locations) used for the pick-up and unloading of vehicle occupants. In this way, an overlay 535 corresponding to the target location can be provided in the user interface 540 regarding one or more PuDo locations provided by the mapper 550 and included in the map data 545.

[0090] An overlay 535 corresponding to a safe target location can be displayed in the user interface 540 for the user to select the target location as a navigation destination. In response to input provided by the user to the user interface 540, the target destination identifier 555 can determine a trajectory 515 corresponding to a navigable path from the vehicle's current location to the target location. The trajectory 515 can be provided to the vehicle planning system 404, and the vehicle can be operated to navigate to the target location based on the trajectory 515.

[0091] Now for reference Figure 7A flowchart of a process 700 for target determination using an eye-tracking device and LiDAR point cloud data is shown. In some embodiments, one or more steps described with respect to process 700 are performed by target determination system 505 (e.g., wholly and / or partially, etc.). Additionally or alternatively, in some embodiments, one or more steps described with respect to process 700 are performed by other means or groups of means (such as sensing system 402, planning system 404, positioning system 406, and / or control system 408, etc.) that are separate from or include target determination system 505 (e.g., wholly and / or partially, etc.).

[0092] At point 702, first data representing three-dimensional coordinates associated with a first location can be received. The first location may be a target location or destination that the driver is attempting to navigate to. The first data may be obtained via at least one sensor fixed to the vehicle. In some embodiments, the sensor may include an eye-tracking device fixed in the cockpit of the vehicle. The eye-tracking device may be positioned to capture data corresponding to the driver's gaze or viewing direction. The sensor may be included among multiple sensors fixed to the vehicle. Each sensor may be configured to transmit field-of-view data and track eye movements of the user regarding the location of the first location. The field-of-view data may include one or more reference points corresponding to anatomical features of the driver, such as eyes or nose. In some embodiments, the first data may include three-dimensional coordinate data associated with the target location or destination that the driver is viewing or focusing on.

[0093] At point 704, second data representing LiDAR point cloud data can be received. LiDAR point cloud data can be obtained from at least one LiDAR device fixed to a vehicle.

[0094] At point 706, a visual indication of the first location can be provided on the user interface of the vehicle. The visual indication can correspond to the target location that the driver is observing or focusing on when the first data is received from a sensor (e.g., an eye tracker device) at point 702. The visual indication can be generated based on the first and second data.

[0095] In some embodiments, generating visual indications may include determining third data representing a first location as an overlay of three-dimensional coordinates and LiDAR point cloud data. Therefore, this overlay can provide indications of a target location in a local scene or map generated using the LiDAR point cloud data. Various non-limiting audible and graphical affordances can be used to provide visual indications in a user interface (such as geometry, animations, icons, sounds, or combinations thereof).

[0096] Generating visual indications may further include mapping the third data to fourth data, which represents map data indicating the local environment in which the vehicle is operating. The fourth data (e.g., map data) may include a first location (e.g., a target location) and may also include one or more second locations representing marked safety points. In some embodiments, one or more second locations may include PuDo locations (such as at least one pick-up location and at least one unload location). Mapping the third data to the fourth data may correspond to: determining common coordinate values ​​between the coordinates of the third data and the coordinates of the fourth data and generating visual indications regarding these common coordinate values.

[0097] Generating visual indications may also include determining a first location (e.g., a target location) based on at least one of one or more second locations included in the fourth data. One or more second locations may correspond to safe locations where the vehicle can pick up or unload vehicle occupants. PuDo locations may be pre-marked in the fourth data (e.g., map data) as safe stations or safe zones for vehicle occupants to board or disembark.

[0098] At point 708, in response to user input for selecting a visual indicator, the vehicle can be operated to navigate to a first location. The user can touch the user interface or make gestures to the user interface to select a visual indicator already provided to indicate the first location (e.g., the target location). Touches or gestures can be processed to select a visual indicator as the first location the vehicle can navigate to. In some embodiments, gestures can be observed by one or more sensors attached to the vehicle (such as an eye-tracking device, a microphone, or a touchscreen configured with a user interface). The gestures can be observed when the user locates the first location (e.g., the target location).

[0099] Operating a vehicle to navigate to a first location may also include using a planning system (such as planning system 404) configured in the vehicle to generate a trajectory toward the first location. The trajectory can be determined and generated from the vehicle's current location toward the first location (e.g., a target location) based on user input for selecting visual instructions provided via a user interface. As a result of the user input, the vehicle can be operated to navigate to the first location based on the trajectory. For example, positioning system 406 and control system 408 can operate the vehicle to navigate to the first location based on the trajectory determined via planning system 404.

[0100] In some embodiments, the eye-tracking device can be used to provide data about a user's viewing or gaze at a user interface, where visual indications of a target location are provided. The eye-tracking device can generate data representing three-dimensional coordinates associated with locations on the user interface, where destination points are available for destination selection. Locations on the user interface can correspond to the presentation of visual indications associated with the location the user wishes to select. The user can then provide input (such as gestures or inputs from a physical device coupled to the user interface) to select a location on the user interface as a destination or target destination. In this way, the eye-tracking device can be used to generate data representing three-dimensional coordinates of locations both inside and outside the vehicle. Therefore, the operator of the vehicle can operate the vehicle more efficiently and safely when selecting a destination presented on the vehicle's user interface.

[0101] The target determination techniques described herein, utilizing eye-tracking devices and LiDAR point cloud data, offer a technological solution that provides advantages over existing target determination systems. These advantages may include, but are not limited to, increased processing time and accuracy for target location determination in autonomous vehicle operating environments. The target determination system described herein also provides an improved user interface for target location selection and navigation. Therefore, safe zone locations (such as pick-up and unload points) can be easily selected as destination stations to allow vehicle occupants to safely enter and exit the vehicle. The target determination system described herein can also generate trajectory data that can be used to train a vehicle planning system for more efficient route generation and navigation planning regarding target locations or safe zones used for passenger boarding and disembarking.

[0102] In the preceding description, aspects and embodiments of this disclosure have been described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The sole and exclusive indication of the scope of this invention, and what the applicant expects to be the scope of this invention, is the literal and equivalent scope of the claims published from this application in the specific form of the published claims, including any subsequent amendments. Any definitions of terms expressly set forth herein for inclusion in such claims should be taken as meaning as such terms are used in the claims. Furthermore, when the term “comprising” is used in the preceding specification or appended claims, what follows that phrase may be an additional step or entity, or a sub-step / sub-entity of a previously stated step or entity.

[0103] Cross-reference to related applications

[0104] This application claims priority to U.S. Provisional Application 63 / 299,094, filed January 13, 2022, under Section 119(e) of Title 35 of the U.S. Patent and Trademark Code, the entire contents of which are expressly incorporated herein by reference.

Claims

1. A method for a vehicle, comprising: Using at least one processor, first data representing three-dimensional coordinates associated with a first location, which is a target location or destination that the user is viewing or focusing on, is received via at least one sensor fixed to the vehicle. Using the at least one processor, second data representing LiDAR point cloud data obtained from at least one LiDAR device fixed to the vehicle is received, the LiDAR point cloud data including three-dimensional coordinates associated with the first location; Using the at least one processor, a visual indication of the first location is provided on the user interface of the vehicle, the visual indication being generated based on the first data and the second data; as well as Using the at least one processor, in response to user input for selecting the visual indication, the vehicle is operated to navigate to the first location. The generation of the visual indication further includes: using the at least one processor to determine third data representing the first location as an overlay of the three-dimensional coordinates and the LiDAR point cloud data, the overlay providing an indication of the target location or destination point of a local scene or map generated using the LiDAR point cloud data; using the at least one processor to map the third data with fourth data representing map data, the map data including the first location and one or more second locations representing marked safety points; and using the at least one processor to determine the first location based on at least one of the one or more second locations included in the fourth data.

2. The method according to claim 1, wherein, The one or more second locations include at least one loading location and at least one unloading location.

3. The method according to claim 1 or 2, wherein, Operating the vehicle to navigate to the first location also includes: Based on user input for selecting the visual indication, a planning system is used to generate a trajectory toward the first location from the vehicle's current location; and The vehicle is operated based on the trajectory to navigate to the first location.

4. The method according to claim 1 or 2, wherein, The at least one sensor is included in a plurality of sensors fixed to the vehicle and configured to transmit field-of-view data to the at least one processor.

5. The method according to claim 4, wherein, The at least one sensor is configured to track the eye movements of the user locating the first location.

6. The method according to claim 1 or 2, wherein, The user input is received as a gesture by the user regarding the location of the first location, as observed by the at least one sensor.

7. A system for a vehicle, comprising: At least one sensor is attached to the vehicle; At least one LiDAR device is attached to the vehicle. At least one processor, and At least one non-transitory storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 6.

8. At least one non-transitory storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program that, when executed by at least one processor, causes the at least one processor to perform the method according to any one of claims 1 to 6.

Images