Vehicle-aware visualization

Abstract

Perceptual visualization is provided. The system may receive first sensor data from a first plurality of sensors of a first sensor type. The system may receive second sensor data from a second plurality of sensors of a second sensor type. The system may generate an environmental map based on the first sensor data and the second sensor data. The system may identify a plurality of objects and an indication of a route based on an environmental map. The system may classify a plurality of objects and routes. The system may display a plurality of first visual representations corresponding to the plurality of objects and a route-based second visual representation based on the classification.

Description

[0001] Cross-references to related applications

[0002] This application claims the interests and priorities of U.S. Provisional Application 63 / 643404, filed May 6, 2024, and U.S. Provisional Application 63 / 652551, filed May 28, 2024, each of which is incorporated herein by reference in its entirety.

[0003] introduction

[0004] Vehicles can include various sensors to perceive their environment. Vehicles can then navigate or communicate based on the sensor data. Summary of the Invention

[0005] This disclosure generally relates to systems and methods for perception visualization. For example, perception visualization can be presented on a vehicle's graphical user interface based on fused sensor data from multiple sensor groups associated with the vehicle. Each sensor group may include at least one sensor type, such as radar, ultrasonic, or optical cameras. Some or all of the sensor groups can collect 360° coverage around the vehicle. For example, a combination of front, rear, and side cameras can generate a first set of sensor data, which can be transformed to generate a first coded image. A set of radar, ultrasonic, or other sensors can generate a second coded image. Some vehicles may include multiple sets of the same sensor type, such as multiple cameras (e.g., long-range and short-range cameras or visual and IR cameras). A vehicle may include any number of sensor groups. For example, an example vehicle may include two sets of cameras and a set of radar sensor / transmitter pairs.

[0006] The mapper can ingest coded images from various sensor arrays to generate an environmental feature map. This environmental map can sometimes be referred to as a feature map or a fusion of various coded images. The decoder can ingest the environmental map and extract various datasets. For example, the decoder can extract object data related to the location, speed, or other aspects of vehicles, pedestrians, or other vulnerable road users, potholes, or other objects. The decoder can extract lane data, such as lane markings or cone information, lane centering, etc. The decoder can extract traffic control data, such as road signs, traffic lights, or other traffic control equipment, or traffic control data retrieved based on GNSS or other location sensors. The system can present any extracted information on a graphical user interface. For example, the system can display vehicles on the road along with lane markings and paths, as well as other objects of interest near the vehicles.

[0007] In at least one aspect, a system includes one or more processors coupled to a memory. The system can receive first sensor data from a first plurality of sensors of a first sensor type. The system can receive second sensor data from a second plurality of sensors of a second sensor type. The system can generate an environmental map based on the first and second sensor data. The system can identify multiple objects and route indications based on the environmental map. The system can classify the multiple objects and routes. The system can display multiple first visual representations corresponding to the multiple objects and second visual representations based on the classifications.

[0008] These and other aspects, as well as specific embodiments, are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of the aspects and embodiments, and provide an overview or framework for understanding the nature and characteristics of the protected aspects and embodiments. The accompanying drawings provide illustrations and further understanding of the aspects and embodiments, and are incorporated in and constitute a part of this specification. The foregoing information and the following detailed description, as well as the accompanying drawings, include illustrative examples and should not be considered limiting. Attached Figure Description

[0009] The accompanying drawings are not intended to be drawn to scale. In the various drawings, similar reference numerals and names indicate similar elements. For clarity, not every part may be labeled in every drawing. In the drawings:

[0010] Figure 1 A system for visualizing the environment associated with a vehicle is described based on several aspects.

[0011] Figure 2 The electric vehicle is described based on some aspects.

[0012] Figure 3 It is a top view of the vehicles arranged in the environment according to certain aspects.

[0013] Figure 4 A block diagram depicts the data flow of a data processing system based on several aspects.

[0014] Figure 5 A block diagram depicts the data flow of the output of a data processing system based on several aspects.

[0015] Figure 6 A graphical user interface is described based on several aspects.

[0016] Figure 7 A block diagram depicts the architecture of a computer system that can be deployed to implement the systems and methods described and illustrated herein.

[0017] Figure 8An example graphical user interface is depicted based on some aspects.

[0018] Figure 9 An example graphical user interface is depicted based on some aspects.

[0019] Figure 10 An example graphical user interface is depicted based on some aspects.

[0020] Figure 11 An example graphical user interface is depicted based on some aspects.

[0021] Figure 12 An example graphical user interface is depicted based on some aspects.

[0022] Figure 13 This is an example top view of a vehicle arranged in the environment according to certain aspects.

[0023] Figure 14 An example graphical user interface is depicted based on some aspects.

[0024] Figure 15 This is an example top view of a vehicle arranged in the environment according to certain aspects.

[0025] Figure 16 It is a block diagram of an example of a data processing system based on some aspects.

[0026] Figure 17 It is a block diagram of an example of a data processing system based on some aspects.

[0027] Figure 18 It is a block diagram of an example of a data processing system based on some aspects. Detailed Implementation

[0028] The following is a more detailed description of various concepts and specific implementations related to methods, apparatuses, and systems for environmental visualization. The various concepts introduced above and discussed in more detail below can be implemented in any of a variety of ways.

[0029] Figure 1A system for visualizing the environment associated with a vehicle is described, based on several aspects. The system may include, interface with, or otherwise communicate with a data processing system 100 for a vehicle (such as an electric vehicle). The data processing system 100 may include the vehicle or may be part of it (e.g., hosted by the vehicle). The data processing system 100 may include various sensors 105 (or sensor arrays 105) or interface with them to determine the conditions of the environment near the vehicle. The data processing system 100 may include at least one sensor data encoder 110 or interface with it to encode data from the sensor array 105. The data processing system 100 may include at least one mapper 115 or interface with it to generate a feature map based on multiple encoded images generated by the sensor array 105. The data processing system 100 may include at least one decoder 120 or interface with it to extract features from the feature map. The data processing system 100 may include at least one interface or interface with it, which includes a user interface 125, such as an instrument cluster display or center information display (CID) in a cockpit.

[0030] The data processing system 100 may include at least one data storage library 135. Sensor 105, sensor data encoder 110, mapper 115, decoder 120, or user interface 125 may each include at least one processing unit or other logic device, such as a programmable logic array engine, or a module configured to communicate with the data storage library 135 or a database. Sensor 105, sensor data encoder 110, mapper 115, decoder 120, or user interface 125 may be a separate component, a single component, or part of the data processing system 100. The data processing system 100 may include hardware elements such as one or more processors, logic devices, or circuitry. For example, the data processing system 100 may include… Figure 7 One or more components or functional structures of a computing device as depicted in the text.

[0031] Data storage 135 may include one or more local or distributed databases, and may include a database management system. Data storage 135 may include computer data storage or memory, and may store one or more data structures, such as data structures corresponding to sensor data 140 or visual representation 145.

[0032] Sensor data 140 may refer to or include information received from one or more sensors 105. For example, sensor data may be grouped according to sensor group 105. For example, a first group of cameras may generate first sensor data 140, a second group of cameras (e.g., including optical cameras different from the first group) may generate second sensor data 140, and a radar sensor may generate third sensor data 140. Each group of sensor data 140 may include different information. For example, sensor data 140 associated with radar may include speed information, and sensor data 140 associated with a visible spectrum camera may include color data. Other sensor data may include overlapping and non-overlapping information, such as the position, brightness, condition, speed, or other aspects of a road or objects in or otherwise associated with a road.

[0033] Visual representation 145 may refer to or include symbolic representations of objects associated with the vehicle environment. For example, visual representation 145 may include one or more visual representations 145 for road users (such as cars, trucks, or vulnerable road users (VRUs) such as pedestrians, motorcyclists, cyclists, etc.). Visual representation 145 may include depictions of roads, such as road markings, obstacles, lane widths, driving paths, or other aspects. Visual representation 145 may include traffic control equipment, such as stop signs, stop lights, or traffic cones. Visual representation 145 may differ from physical objects. A specific visual representation 145 for a vehicle may depict one or more objects of a particular object category. For example, a first visual representation 145 may depict any of a car, truck, or SUV; a second visual representation 145 may depict another vehicle, such as a box truck, semi-trailer, or concrete mixer truck. Similarly, one or more categories of road markings may correspond to one or more visual representations 145.

[0034] Some visual representations 145 may include aspects corresponding to elevated or reduced salience, such as highlighting, color, arrows, or descriptive names. Some visual representations 145 may correspond to a vehicle (e.g., a vehicle implementing the systems and features provided herein). Raised or otherwise distinguished salience relative to other objects may be provided to the visual representation 145 of the vehicle. For example, the vehicle may be presented as the center of the display, or include color, shape, or other similarity to the vehicle, or other salience. Objects in the vehicle's path (e.g., vehicles in the same lane (sometimes referred to as the vehicle's lane) as the vehicle of interest may be presented based on specified colors or other salience features.

[0035] The data processing system 100 may include or interface with one or more sensors 105 configured to sense information associated with the operation of the vehicle or its environment. The sensors 105 may be arranged as a sensor group. Each sensor 105 may include a field of view (FOV) that may overlap to generate continuous sensor data 140 associated with a portion of the vehicle. Each sensor group may capture a set of sensor data 140. Some sensor data groups 140 may surround the vehicle (360° coverage) or substantially surround the vehicle. The sensor group may include sensors 105 of similar type. For example, the sensor group may include a line-of-sight sensor, such as a camera (e.g., a visible spectrum camera). Therefore, the sensor data 140 received from each sensor 105 in the sensor group 105 can be transformed into an encoded form.

[0036] Data processing system 100 may include or interface with one or more sensor data encoders 110. Sensor data encoders 110 may generate codes from at least one set of sensors 105. Codes may include combinations of data received from various sensors. Code generation may include performing spatial or other transformations, such as perspective transformations. For example, sensor group 105 may include sensors arranged substantially along the driving surface, such as forward-facing cameras, reversing cameras, or blind-spot cameras for advanced driver assistance systems (ADAS). Codes may apply transformations to generate a top-down view (sometimes referred to as a birdseye view). Some transformations may vary depending on the various sensors 105 in the sensor group. For example, sensor group may include a first camera with a fisheye lens (e.g., a wide-angle reversing camera) and a second camera with a balanced (or telephoto) lens, such as a forward-facing camera for an adaptive cruise control (ACC) system.

[0037] The sensor data encoder 110 can encode any information embedded in sensor data 140 generated by one or more sensors 105 in the sensor group 105. For example, the encoded information may include the relative or absolute position, depiction, shape, or other aspects of an object, road, or environment. Specific encoded information can depend on the sensor type and location. For example, a radar or LiDAR sensor may generate speed, distance, or transparency information that can be omitted via one or more cameras, or an infrared camera may generate temperature data that can be omitted via a radar sensor (e.g., a transmitter-receiver paired receiver).

[0038] Data processing system 100 may include or interface with one or more mappers 115. Mappers 115 may ingest various codes and generate feature maps based on information embedded in these codes. Mappers 115 may implement perspective (e.g., spatial) or other transformations to align data between various codes. For example, mapper 115 may align features from radar, ultrasound, images, or other sensors to generate feature maps. Feature maps may include a combination of information from environmentally relevant sources and therefore may be referred to as "feature maps" or "environment maps" without limitation.

[0039] The data processing system 100 may include or interface with one or more decoders 120. Decoder 120 may decode features from a feature map. Decoder 120 may extract features corresponding to predefined categories or having predefined labels. For example, decoder 120 may decode features corresponding to objects in a road, belonging to or otherwise associated with it, road lanes, traffic control equipment, or other environmental data.

[0040] Some features may include objects such as motor vehicles or other vehicles, traffic control equipment, lane markings, or travel paths. Decoder 120 can classify the decoded features according to a set of predefined categories to display a visual representation 145 corresponding to that category. For example, decoder 120 may determine that the extracted features correspond to various vehicles and the corresponding travel lanes for each vehicle. Decoder 120 may classify objects or other features to determine their corresponding visual representations 145. For example, decoder 120 may determine that a VRU corresponds to a visual representation of a bicycle, motorcycle, scooter, or pedestrian. Decoder 120 may update the recognition of visual representation 145. For example, decoder 120 may update the recognition periodically or based on a trigger. The update of visual representation 145 may be synchronized or asynchronous with other processes, such as route recognition or route indication, or determination or indication of navigation intent.

[0041] The data processing system 100 may include or interface with one or more user interfaces 125. The user interface 125 may include a graphical user interface (GUI). The GUI may include visual representations corresponding to features decoded by the decoder 120. For example, the user interface 125 may present depictions of objects and roads. Objects may include any of various vehicle types, road markings, or other visual representations. The classification of features is not limited to what is displayed via the user interface 125. For example, a vehicle may perform navigation actions based on extracted features, even if those features are not displayed. Navigation actions may include, for example, turn signal indication, lane change, braking, acceleration, or steering input (e.g., turning within a lane, between lanes, or outside a marked lane).

[0042] Figure 2 An example cross-sectional view depicts an electric vehicle 200 equipped with at least one battery pack 210. The electric vehicle 200 may include electric trucks, electric sport utility vehicles (SUVs), electric vans, electric cars, electric motorcycles, electric scooters, electric passenger cars, electric passenger or commercial trucks, hybrid vehicles, or other vehicles such as marine or air transport vehicles, aircraft, helicopters, submarines, ships, or drones, among other possibilities. The battery pack 210 may also be used as an energy storage system to power buildings, such as residential or commercial buildings. The electric vehicle 200 may be fully electric or partially electric (e.g., plug-in hybrid), and further, the electric vehicle 200 may be fully autonomous, partially autonomous, or driverless. The electric vehicle 200 may also be manually operated or non-autonomous. An electric vehicle 200, such as an electric truck or car, may include an onboard battery pack 210, a battery 215 or battery module 215, or a battery cell 220 to power the electric vehicle.

[0043] The electric vehicle 200 may include a chassis 225 (e.g., a frame, internal frame, or support structure). The chassis 225 may support various components of the electric vehicle 200. The chassis 225 may span the front portion 230 (e.g., a hood or cover portion), body portion 235, and rear portion 240 (e.g., a luggage compartment, payload, or trunk portion) of the electric vehicle 200.

[0044] Battery pack 210 may be installed or placed within electric vehicle 200. For example, battery pack 210 may be installed in one or more of the front portion 230, body portion 235, or rear portion 240 of the chassis 225 of electric vehicle 200. Battery pack 210 may include or be connected to at least one bus, such as a current collector element. For example, the first bus and the second bus may include conductive material to connect or otherwise electrically couple the battery 215, battery module 215, or battery cell 220 to other electrical components of electric vehicle 200, thereby providing power to various systems or components of electric vehicle 200.

[0045] The electric vehicle 200 may include or interface with one or more sensors 105 configured to monitor the environment associated with the electric vehicle 200. For example, sensor 105 may include ultrasonic or other time-of-flight sensors 105 or cameras configured to detect aspects of the environment associated with the electric vehicle 200.

[0046] Each of the sensors 105 may correspond to a field of view (FOV), which can refer to the FOV of a camera or other optical sensor, a scanning area of ​​a radar transmitter / receiver pair, a detection area of ​​an ultrasonic transmitter / receiver pair, or another monitoring area associated with another sensor type. Any of the sensors 105 may be dedicated to visualizing vehicle perception information or may be used for any other purpose. For example, sensor 105 may include a camera or radar for automatic evasive braking or ACC, a backup camera, or another sensor. For example, the first sensor 245 may include a reversing camera that includes at least visible spectrum sensor data 140. The second sensor 250 may include a blind spot sensor implemented according to any sensor type. Another instance of the second sensor 250 may be present on opposite sides of the vehicle, wherein at least a portion of the sensor is substantially symmetrical about the longitudinal axis of the vehicle. The third sensor 255 may include a vehicle position sensor, such as a Wi-Fi, cellular, or Global Navigation Satellite System (GNSS) sensor, such as GLONASS or GPS. The fourth sensor 260 may include a wing sensor that monitors the FOV of the side of the electric vehicle 200. Corresponding sensors may be positioned on opposite sides of the electric vehicle 200. The fifth sensor 265 may include a front-view camera; the sixth sensor 270 may include a front blind-spot camera; and the seventh sensor 275 may include an outward-facing camera from the vehicle's cockpit, such as a camera used by an automatic evasive braking or ACC system. One or more sensors (e.g., the fifth sensor 265) may include multiple sensor orientations (e.g., forward, rearward, outward) or types (e.g., FOV or sensor type, such as a visible spectrum camera, IR camera, ultrasonic, or radar). For example, a side mirror or mirror assembly may include three sensors 105 to collect sensor data 140, which is fused by a mapper 115 or otherwise used in the systems and methods described herein. Various electric vehicles 200 may implement any of a variety of sensors. Some electric vehicles may omit any of the depicted sensors or include any additional sensors of any sensor type or location.

[0047] Figure 3 It is based on the arrangement of vehicles in the environment according to certain aspects (e.g., Figure 2A top view 300 of the electric vehicle 200. View 300 includes the fields of view of various sensors of the vehicle. For example, the frontal field of view (FOV) 302 may include objects in front of the vehicle 200, which may include a primary object 304 or various other detected objects 306. The first set of cameras may correspond to the camera that captures the first FOV 302, and the cameras that capture the second FOV 308, the third FOV 310, the fourth FOV 312, the fifth FOV 314, the sixth FOV 316, and the seventh FOV 318 (collectively referred to as the first field of view array 324).

[0048] A second set of cameras can capture data from a second field-of-view array 320. Additional sensor groups (such as radar and ultrasound) can collect additional sensor data. Each sensor or group can capture features not available to other sensors 105 or groups. For example, the second field-of-view array 320 can detect VRUs 322 not included in the FOV of the first field-of-view array 324. Additional sensor types (e.g., radar or ultrasound) can detect additional features, such as the velocity or reflectivity distribution of the VRU, which can be received by the decoder 120 to determine the classification of the VRU 322. For example, based on the captured features of a shape profile corresponding to a human torso (e.g., received from a camera) combined with location information corresponding to a pedestrian walkway (e.g., received from GNSS) and a velocity corresponding to 3 kilometers per hour (kph), the decoder 120 can associate a VRU 322 with a visual representation 145 indicating a pedestrian. Depending on different speeds or locations, the decoder can associate the same shape profile with a bicycle, scooter, or other visual representation 145.

[0049] Combining sensor data prior to feature generation or extraction / decoding (early fusion) can aid in the classification of objects or other aspects of the vehicle environment. For example, cross-domain (e.g., image / radar) information can improve the performance of object type classification or other predictions. Furthermore, the data processing system 100 can continue to operate even in the event of communication loss or data loss with one or more sensors or sensor groups. For instance, in the event that at least a portion of sensor 105 or sensor group 105 is missing, obstructed, or otherwise inoperable for the associated FOV, other sensors of the same or different types can provide information to maintain system operation.

[0050] Figure 4A block diagram 400 depicts the data flow of a data processing system 100 according to some aspects. Various sensor groups 105 can generate sensor data. For example, sensor groups 105 may include non-overlapping groups (e.g., where at least one of the first, second, or nth groups of sensors 105 does not include another sensor 105 identical to another in the first, second, or nth group of sensors). The first group of sensors 105 can generate data corresponding to, for example, having similar characteristics to... Figure 3 The first sensor data 405 of the camera with the first field of view array 324. The second sensor group 105 can generate data corresponding to, for example, having a similar... Figure 3 The second sensor data 410 of the camera in the second field of view array 320. The nth sensor group 105 can generate nth sensor data 415 of another sensor type, such as radar echo data.

[0051] Sensor encoder 110 can generate codes from each sensor group. A first code can be generated by combining various streams of first sensor data 405. For example, sensor encoder 110 can generate a code for a narrow field-of-view (NFOV) forward-facing camera (e.g., corresponding to...). Figure 3 The front FOV 302), the left wing FOV and right wing FOV with side mirror-mounted cameras (e.g., corresponding to...). Figure 3 The third FOV 310 and the sixth FOV 316 are stitched together with another region of interest (ROI) FOV. The ROI FOV can include virtual cameras generated from one or more sensors, such as... Figure 3 The combination of the fourth FOV 312 and the fifth FOV 314. In some cases, the combination of sensors can collectively form a 360° FOV around the vehicle. The data processing system 100 can combine various streams of second sensor data 410 to generate a second code. The system can continue to operate using visual data based on the absence of either the first embedding or the second embedding. For example, the first and second embeddings can include redundant 360° coverage, or the system can continue to operate in the absence of 360° coverage (e.g., the absence of a rear-facing camera when the vehicle is moving forward). Additional codes can combine various streams of their corresponding sensor data to generate additional codes for any sensor (such as an ultrasonic sensor). The nth code can combine various streams of nth sensor data 415 to generate the nth code (e.g., radar).

[0052] Although encoding can include overlapping information, encodings can vary in perspective (e.g., differ from each other or from another part of data processing system 100). For example, sensor encoder 110 can perform spatial transformations of the encodings to generate sensor data transformations based on a viewpoint (e.g., a bird's-eye view, an oblique bird's-eye view, or an isometric view from top to bottom). Such transformations can help fuse various encodings into the same feature map. For example, sensor encoder 110 can transform first sensor data 405, second sensor data 410, and so on up to nth sensor data 415 to achieve first sensor data transformation 420, second sensor data transformation 425, and so on up to nth sensor data transformation 430. Various transformations can be within the same spatial viewpoint to facilitate the fusion of their various features.

[0053] Mapper 115 can fuse features to generate an environment map 435 (which can be referred to as a feature map of the components of environment map 435 without limitation) that includes various features of sensor data transformations 420, 425, and 430. Mapper 115 can embed all the features that make up the map into environment map 435. For example, environment map 435 may include position, color, reflectivity, speed, or other information captured by any sensor 105 of data processing system 100 or interfacing with it. A subset of the features of environment map 435 can be used for various purposes. For example, features corresponding to the presence of objects in the vehicle's path can be used for evasive braking to reduce reliance on other parts of the model.

[0054] Decoder 120 can decode information embedded in environment map 435. For example, decoder 120 can decode object data 440 associated with objects such as road users, obstacles, and some traffic control devices (e.g., traffic cones). Decoder 120 can decode lane data 445. For example, decoder 120 can decode features associated with dashed or solid lane dividers, the color of lane dividers, the location of obstacles, or other features associated with the vehicle's travel path. Decoder 120 can decode traffic control data 450, such as the state of drivable space. Traffic control data 450 may also include information derived from skipped layer connections (e.g., traffic light colors or speed limit signs may be clearer in a non-transformed view before generating environment map 435, or such information may be embedded separately from spatial transformations 420, 425, 430 in environment map 435). Decoder 120 can decode any other environmental data 455 associated with vehicles, such as indications of volume occupancy of various parts of the environment (e.g., a three-dimensional spatial grid).

[0055] Figure 5A block diagram 500 depicts the data flow of the output of a data processing system 100 according to several aspects. The data flow includes an object tracker 505, a lane tracker 510, and a space tracker 515 of the data processing system 100. Similar to other aspects of the data processing system 100, the object tracker 505, lane tracker 510, and space tracker 515 can be configured according to, for example, references to... Figure 1 and Figure 7 The various circuits described are used to implement this.

[0056] Object tracker 505 can associate categories or identities with detected objects (e.g., extracting feature embeddings of vehicles, vehicle types, pedestrians, physical obstacles, buildings, or any other object features via a 3D box regressor). Object tracker 505 can implement kinematic estimation to estimate the velocity or orientation of identified objects and cause the display of a visual representation 145 of the objects. For example, the kinematic estimator can implement smoothing functions to reduce jitter and maintain object identity over time (e.g., data filtering or processing of various identified objects can be implemented). For example, such filtering can verify the positions of multiple objects and routes after identification and before displaying multiple objects and route indications. Verification can include temporal dependencies between positions and previous positions (e.g., to avoid the appearance / disappearance of false sensor readings).

[0057] Lane tracker 510 can determine the position of a lane based on received decoded lane data 445, identifying the lane itself, any adjacent lanes or other lanes on the road, or other road boundaries such as physical obstacles or road markings. Lane tracker 510 can recognize a visual representation corresponding to the lane. Lane tracker 510 can cause the display of the lane or associated features via a display of user interface 125. Lanes can further vary based on drivable space, such as based on other vehicles, obstructions, or road conditions. Lane tracker 510 can identify lanes not occupied by vehicles. For example, lane tracker 510 can identify bicycle lanes, pedestrian walkways (e.g., sidewalks), etc.

[0058] A spatial tracker 515 (such as a grid-based occupancy tracker) can identify the spatial occupancy of various objects in the environment. For example, the spatial tracker 515 can subdivide the environment into a two-dimensional or three-dimensional grid to determine driving paths that may deviate from lanes in some situations. For example, driving paths may include leaving and re-entering lanes based on the flow of other vehicles (e.g., in response to construction vehicles or obstacles such as deer impacts). Paths may also depend on the presence of objects such as traffic cones or obstacles. The spatial tracker 515 can provide driving paths that can be smoothed over time or otherwise adjusted to avoid discontinuous paths. For example, the spatial tracker 515 can be provided based on piecewise polynomial (e.g., cubic) splines. The polynomial can be configured to correspond to the turning radius of the vehicle.

[0059] Any data derived from or otherwise derived from the object tracker 505, lane tracker 510, or space tracker 515, or from the data processing system 100, may be provided to the user interface 125 for presentation (e.g., visual display via the GUI). Similarly, any data may be used to generate or disable navigation actions, whether by the data processing system 100 or another component of the vehicle (such as the navigation control unit 525). That is, the perception system may be provided for autonomous or semi-autonomous systems or for user-combined displays. Some data may be provided to the user interface 125 but not to the navigation control unit 525; conversely, some data may be provided to the navigation control unit 525 but not to the user interface 125.

[0060] Figure 6 A graphical user interface 125 is described according to several aspects. Display 600 can be presented via an instrument cluster display (ICD), a center information display (CID), or another display of the vehicle. For example, the systems and methods provided herein can enable the presentation of a GUI to occupants of an electric vehicle to visualize interactions with the vehicle (such as...). Figure 2 The environment associated with electric vehicles (200).

[0061] Display 600 may include a vehicle 602 corresponding to other aspects of the vehicle including sensor 105 or data processing system 100. Display 600 may include lane markings 608 or other indications of the driving path (e.g., off-road trails). Lane markings 608 may define lane 610 and one or more adjacent lanes 612, 614. The data processing system may determine the depicted lanes based on features decoded from sensor 105 or features received from a stored instance of a map (e.g., location information such as GPS sensor data).

[0062] Display 600 may include various additional objects, including vehicle 604. Increased salience may be provided to subsets of objects, such as a primary object ID 606 (e.g., vehicle 604, which is in the travel path of vehicle 602, or otherwise most relevant to the travel path of vehicle 602). Display 600 may include information associated with traffic control equipment, such as a display of vehicle speed relative to speed limit 616, which may be determined based on extracted traffic control data 450 or map-derived data.

[0063] Figure 7 An example block diagram of an example computer system 700 is depicted. The computer system or computing device 700 may include a data processing system 100 or components thereof, or may be used to implement a data processing system or components thereof. The computing system 700 includes: at least one bus 705 or other communication component for conveying information; and at least one processor 710 or processing circuitry coupled to the bus 705 for processing information. The computing system 700 may also include one or more processors 710 or processing circuitry coupled to the bus for processing information. The computing system 700 also includes at least one main memory 715 (such as random access memory (RAM) or other dynamic storage device) coupled to the bus 705 for storing information and instructions to be executed by the processor 710. The main memory 715 may be used to store information during the execution of instructions by the processor 710. The computing system 700 may also include at least one read-only memory (ROM) 720 or other static storage device coupled to the bus 705 for storing static information and instructions for the processor 710. A storage device 725 (such as a solid-state device, disk, or optical disk) may be coupled to the bus 705 to persistently store information and instructions.

[0064] The computing system 700 may be coupled to a display 735, such as a liquid crystal display or an active matrix display, via a bus 705 for displaying information to a user (such as a user positioned inside or outside the cockpit of the electric vehicle 200). An input device 730 (such as a button or voice interface) may be coupled to the bus 705 for transmitting information and commands to the processor 710. The input device 730 may include a touchscreen display 735. The input device 730 may also include cursor controls (such as a mouse, trackball, or arrow keys) for transmitting directional information and command selection to the processor 710 and for controlling cursor movement on the display 735.

[0065] The processes, systems, and methods described herein can be implemented by a computing system 700 in response to a processor 710 executing an instruction arrangement contained in main memory 715. Such instructions may be read into main memory 715 from another computer-readable medium, such as storage device 725. Execution of the instruction arrangement contained in main memory 715 causes the computing system 700 to perform the exemplary processes described herein. One or more processors in a multiprocessor arrangement may also be used to execute the instructions contained in main memory 715. Hardwired circuitry may be used in place of or in combination with software instructions and the systems and methods described herein. The systems and methods described herein are not limited to any particular combination of hardware circuitry and software.

[0066] Although already Figure 7 An example computing system is described herein, but the subject matter including the operations described herein may be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed herein and their structural equivalents), or in a combination of one or more of them.

[0067] Figure 8 A graphical user interface (GUI) is depicted according to several aspects. The GUI may be depicted via a display 600 that is the same as or different from other instances of the GUI 125 disclosed herein. For example, the same display 600 may depict various elements of the GUI 125 in response to an explicit selection of a display mode, input received from a user or autonomous system (e.g., a turn signal indication), or detected conditions (such as the presence of a vehicle or other object, lane, etc.). Different displays 600 may be used to selectively display a particular GUI or specific elements thereof. For example, any GUI or its various elements depicted herein may be displayed via an ICD, CID, or other vehicle display. One or more instances of a computing system, such as the data processing system 100, may generate elements of the GUI 125 provided via the display 600.

[0068] Display 600 may include the vehicle 602 and neighboring vehicles 604 in adjacent lanes 612, 614. Detection indication 802 may depict the presence of an object detected in the field of view of sensor 105. For example, detection indication 802 may be provided to indicate the detection or classification of any object (e.g., motor vehicle, VRU, or traffic control device such as a traffic cone). The vehicle's user interface 125 may include any number of indications (e.g., LEDs or audible alarms for vehicles in blind spots) such that indications can be provided via display 600 or other indicators. Navigation window 804 depicts the vehicle's planned route 805. Additional aspects of display 600 may be based on planned route 805. For example, display 600 of lane 610 may include lane transitions on the same road or transitions between roads. Furthermore, one or more selected primary object (PO) IDs 606 may depend on planned route 805 (e.g., current or predicted lane). For example, POID 606 may be in a lane different from the vehicle's, where the vehicle's path includes changes from the lane to a lane including POID 606. The GUI elements of the display 600 can depict the selected mode, such as an indication via lane detection 806, or the selected driving mode 808. For example, autonomous or semi-autonomous driving mode 808 can be displayed as not selected based on the lack of color, highlighting, or other prominent display features.

[0069] Figure 9 A graphical user interface 125 is depicted according to several aspects. Various elements of the GUI can be presented via a display 600. The GUI includes a planned lane change 902 instruction presented in response to path indication. The planned lane change 902 instruction can be received from the user (e.g., via actuation of a steering lever signal) or from an autonomous system in response to the environment, including detected or otherwise captured objects and roads. For example, the environment may include information received in a map or detected in sensor data 140 generated by sensor 105, and processed in a bird's-eye view (BEV) or other state space (e.g., from a single sensor or a BEV state space used for sensor fusion). In some cases, lane changes are provided in response to a planned route (e.g., moving into or out of an overtaking lane associated with overtaking maneuvers, or entering a left or right exit).

[0070] Various elements of the GUI can be provided in response to receiving a planned lane change 902. For example, the data processing system 100 can be configured to provide various elements of the GUI in response to receiving a lane change instruction. The data processing system 100 can provide a detection indication 802 with increased salience in response to the planned lane change 902 via the GUI. The data processing system 100 can provide a lane 610 relative to lane markings 608 via the GUI. In particular, the lane 610 is shown centered between lane markings 608. Lane markings 608 can be shown based on various aspects of the environment. Lane markings 608 can be shown as solid lines to indicate a lack of lane change availability due to dynamic conditions (e.g., other vehicles 604), even if the lane markings differ from such a view. For example, as shown in camera view 904, lane markings 608 indicating that lane changes are permitted can be depicted as not permitting lane changes in response to the detection of vehicle 604. The data processing system 100 can provide, via a GUI, lanes of interest 906 that are limited based on the detection of vehicle 604 (e.g., a vehicle depicted in a camera view 904), as detected by any other sensor among the various sensors 105 of the camera or vehicle.

[0071] Figure 10 A graphical user interface is depicted according to several aspects. Depending on the operation of the data processing system 100, the display 600, which provides elements of the GUI, can depict lanes of interest 906 available for use by the vehicle 602. For example, a destination 1002 in lane 906 is shown as available for vehicle use. Destination 1002 can be provided relative to the vehicle 602 or other vehicles 604 in the same environment, such that the destination can be shown as moving relative to the environment, including roads or other objects.

[0072] exist Figure 11 The diagram shows a lane 610 transitioning between lane markings 608. The data processing system 100 can make the display of lane 610 cross the lane markings to occupy portions of multiple lanes. The display may include lane marking 608 GUI elements for combinations of lanes occupied by vehicles, as indicated by elevated salience. Lane markings 608 showing lane-to-lane boundaries are further depicted as lacking elevated salience. One or more primary object IDs 606 can be selected in the lanes that the vehicle 602 leaves or enters.

[0073] Figure 12A graphical user interface (GUI) is depicted according to several aspects. Similar to other instances of GUIs disclosed herein, the GUI can be generated by the data processing system 100. Lane 610 is depicted as a portion of a lane defined by lane markings 608, which correspond to and are depicted by approximations of physical road markings. For example, a restricted portion 1202 of the road lane corresponding to lane 610 can be depicted as an element of the GUI. The restricted portion 1202 may correspond to an area identified by an autonomous system (such as a lane keeping assist system (LKAS) or route planner) for avoidance. The restricted portion 1202 can be generated, identified, displayed, or used by the autonomous system in response to the detection of vehicle 604 or other objects or their classification (e.g., as tunnel walls or construction obstacles).

[0074] Data processing system 100 can generate, identify, display, or use restricted portion 1202 based on priority object ID 606 (e.g., the category of priority object ID 606, the size of priority object ID 606 to a threshold size, or the position of priority object ID 606 relative to a threshold position, such as when priority object ID 606 corresponds to a vehicle intruding into lane 610). Such objects are sometimes referred to as confrontation objects. One or more classifications of confrontation objects can correspond to confrontation distances, such as varying based on the speed of vehicle 602, the confrontation objects, or the relative speed between them. For example, data processing system 100 can generate restricted portion 1202 to cause the vehicle to deviate from another vehicle, wall, or VRU. The autonomous system of vehicle 602 can center the vehicle in lane 610 excluding restricted portion 1202 so that vehicle 602 avoids encroaching on nearby objects. When vehicle 602 is no longer next to priority object ID 606, data processing system 100 can identify lane 610 without restricted portion 1202, so that vehicle 602 can follow the lane. Figure 13 The vehicle proceeds along route 1302 as depicted in the top view 300, thus avoiding the priority object 606. By avoiding close proximity to large objects, this choice of route 1302 can help ensure the comfort of the vehicle occupants and increase the time that the vehicle 602 has to react to any movement of another object, gusts of wind, or other changes in conditions.

[0075] refer to Figure 14According to several aspects, the graphical user interface (GUI) depicts the vehicle 602 with a selected autonomous or semi-autonomous driving mode 808. The GUI can present the driving mode indication with increased prominence when the driving mode 808 is selected. The autonomous or semi-autonomous driving mode 808 may include speed control based on any of a selected speed 1402 (sometimes referred to as the vehicle target speed), a speed limit 1404, or other aspects of the environment. The selected speed 1402 may include a speed selected by the user or a speed selected by the autonomous system (e.g., based on vehicle mileage or vehicle efficiency). The speed limit 1404 may be identified based on predefined map data or from vehicle sensors 105 (e.g., cameras, vehicle-to-vehicle communication, or vehicle-to-infrastructure communication).

[0076] The autonomous system can implement a driving speed 1406 based on environmental factors. The vehicle 602 can select a driving speed 1406 based on the speed of other objects (such as primary object ID 606). For example, if traffic slows down in a lane adjacent to the vehicle's lane 610, the vehicle 602 can reduce its speed below or above the selected speed to reduce the speed difference between lanes. The display 600 may include elements indicating the presence of adjacent traffic 1408 to indicate the reason for the difference between the selected speed and the speed commanded by the drive system or otherwise influenced. The presence of adjacent traffic 1408 or speed changes (e.g., the value of the speed difference) can be provided in response to the classification of vehicles 604 in adjacent lanes. For example, the data processing system 100 can adjust the speed in response to the category of vehicles or other objects (e.g., VRUs, other motor vehicles, or construction equipment).

[0077] Figure 15 It is based on the arrangement of some aspects of the environment of the vehicle 602 (e.g., Figure 2Another top view 300 of the electric vehicle 602 (602). The intended route 805 of the vehicle 602 may intersect with lanes other than the driver's lane 610 (such as bicycle lane 1502). The data processing system 100 of the vehicle 602 can detect a VRU (e.g., depicted as a cyclist 1504) and provide the user with an indication of the presence of the cyclist 1504. This operation may correspond to autonomous, semi-autonomous, or non-autonomous modes. For example, in semi-autonomous or autonomous mode, the data processing system 100 may adjust the vehicle's speed up or down to reach the intersection between the intended route 805 and bicycle lane 1502 at a threshold time or distance from the cyclist 1504. In semi-autonomous or non-autonomous mode, the data processing system 100 may depict a warning based on the intersection (e.g., simultaneous arrival) and the presence of the cyclist 1504 via GUI elements (or another indication) on the display 600. The data processing system 100 may provide steering torque feedback to help the driver maintain a threshold time or distance from the cyclist 1504.

[0078] Overall reference Figures 16-18 A block diagram of an example data processing system 100 is provided. Various block diagrams may refer to different systems or phased transitions to specific implementations of machine learning (ML) accelerators. The data processing system 100 or other implementations thereof may generate one or more elements for a GUI for display, or another component communicating with the data processing system 100 may generate GUI elements based on information determined by the data processing system 100. For example, Figures 16-18 The data processing system 100 (or as otherwise disclosed herein) can generate any GUI instance depicted herein.

[0079] Figure 16 This is a block diagram illustrating an example of a data processing system 100 according to some aspects. The data processing system 100 may include inputs 1602 that can be received by one or more controllers. For example, input 1602 may be received by a first machine learning (ML) accelerator 1604, which may be implemented in a highly parallel environment such as a graphics processing unit (GPU) or ML-specific hardware. Another controller 1606 may ingest the output of the ML accelerator 1604 and, based on the output, provide control outputs to the vehicle's autonomous system (e.g., generating control signals to perform navigation actions, such as steering, braking, acceleration, or turning signal indications). Controller 1606 may operate according to deterministic instructions.

[0080] Data processing system 100 includes or interfaces with input 1602, which includes sensors 105 and predefined data. For example, input 1602 may include a predicted route 1608 received from a user or via a route generator to route the vehicle to a destination received from the user or another predicted destination. Input 1602 may include camera data from any number of cameras 1610. Input may include radar data from any number of radar receivers 1612. For example, all radars of data processing system 100 may be captured by ML accelerator 1604 (e.g., bird's-eye view 1618), and a portion of the radar data may be used in other ways, such as via controller 1606 for path arbitration. A portion of the radar data (e.g., from individual radar transmitters 1612, 1614) may be provided to ML accelerator 1604 and controller 1606. Radar 1612 and corner radar 1614 may refer to sensors with overlapping, non-overlapping, or partially overlapping fields of view (FOV). Map data 1616 may include predefined road-related information, which can be used to align or improve the confidence of sensing data, determine route paths, etc. For example, map data 1616 may include HD maps, such as road data provided in map formats such as Open Street Map (OSM), GeoJSON, or Robot Operating System (ROS).

[0081] Input can be taken in by a sensor fusion unit, which can be configured to generate a bird's-eye view (BEV) state-space transformation for Model 1618 or another model. For example, the data processing system can process the data based on the data stream (e.g., Figure 4 or Figure 5 The data streams (block diagrams 400 and 500) are used to ingest sensor data, which are configured to generate object tracking information. The ingested signal information may include camera data and radar data, or data from any other sensor data available to the system. BEV 1618 may include a top-down view of the world with a local map and visible tracks. Tracks may include driving lanes or portions thereof, or lanes of a road (e.g., based on lane markings, map data, or flow data of other objects such as other vehicles). Data processing system 100 may include or interface with various cross-checkers external to BEV 1618 to verify any information about BEV model 1618. ML accelerator 1604 may fuse any of various inputs (e.g., data from camera 1610 and radar 1612) and provide output to an environmental tracker such as lane tracker 1620 or another tracker 1621 to track any objects in or associated with the environment.

[0082] The path arbiter 1622 can select a route description (e.g., based on the selection of polynomial coefficients, such as a cubic spline function or parameters of its segments). The path arbiter 1622 can consume polylines and fit them to the coefficients. The various components using the coefficients can calculate the coefficients separately, or share the calculated coefficients when the first component is determined. The lane keeping system 1630 can receive route indications to trigger the generation of control signals to trigger other indications of lane departure, such as display, tactile feedback, or lane departure, or generate control signals to maintain the lane (e.g., control signals to adjust or maintain the steering angle).

[0083] The primary object selector 1624 can select one or more objects associated with vehicle 602 based on stored route data received from the route arbitrator 1622. For example, vehicles or other objects in or adjacent to the route can be selected. The reference to a "primary" object should not be interpreted as limiting POID 606 to a single object. Some operational instances may select a single POID 606; some operational instances may select multiple POIDs 606. For example, POID 606 may be selected based on a comparison of its relevance to vehicle 602, determined based on the size, direction of travel, position, or classification of the object or vehicle 602. The primary object selector 1624 or other components (e.g., the automatic braking system AEB 1628) may receive input data separate from the ML accelerator 1604. For example, the primary object selector 1624 may receive angular radar data to indicate the presence of a vehicle in an adjacent lane, such as to avoid approaching a large vehicle (e.g., to cause lane 610 within the lane to deviate). Providing separate data can help some components function even when there is no sensor data or ML accelerator 1604.

[0084] The AEB system 1628 may receive information from the object tracker 1621 or the primary object selector 1624. The AEB 1628 may receive indications of objects that may intersect with the vehicle 602 to determine the application of braking to prevent intersection (e.g., collision avoidance). For example, the AEB system may determine the intersection point between the predicted path of the vehicle 602 and the predicted path of another object (e.g., static positioning or extrapolation based on current speed or direction of travel). The planner 1626 may include a route planner that can control vehicle operation without the operation of the AEB 1628 or LKS 1630. For example, the planner 1626 may cause the vehicle to proceed along a route based on BEV data (e.g., lane selection, position within the lane, or speed selection).

[0085] The data processing system 100 or its components may operate in shadow mode to collect operational data for comparison with another control system component, or operate in active mode to generate display or control signals to induce navigation actions.

[0086] Figure 17 This is a block diagram of an example of a data processing system 100 based on some aspects. The BEV 1618 receives input 1602 from sensor data 140, which includes corner radar data. The PO selector 1624 can omit the input so that the data stream passes through the lane tracker 1620 or other tracker 1621 and is transmitted on the planner 1626, AEB 1628, or LKS 1630.

[0087] Figure 18 This is a block diagram illustrating an example of a data processing system 100 based on several aspects. The diagram depicts additional functionalities implemented in the ML accelerator 1604. For example, the ML tracker 1802 can track or identify various objects, lane markings, and other aspects of the environment. The map fusion unit 1804 can fuse map data 1616 with other inputs 1602. The tracker 1802 and the map fusion unit 1804 can output data to a world predictor 1806 to predict the current or future state of the environment. The world predictor 1806 can provide the controller 1606 with aspects of the predicted environment to control signal generation. For example, the world predictor 1806 can generate outputs for road model 1810, planner 1626, and AEB system 1628.

[0088] Road model 1810 can generate deterministic outputs for LKS 1630, AEB 1628, and planner 1626 based on data received from tracker 1802 and world predictor 1806. For example, road model 1810 can model roads that include tracked objects (e.g., including environmental flow or occupancy) and predictions from world predictor 1806. AEB 1628 can generate control signals for braking based on tracker 1802, world predictor 1806, and road model 1810. LKS 1630 can operate based on road model 1810. Such operation can include generating depictions of a graphical user interface (e.g., any elements provided herein) or control signals to keep vehicle 602 in its lane (which may correspond to marked lanes, subsets thereof, or combinations of multiple lanes).

[0089] The components of the ML accelerator 1604 can be implemented as different components with discrete inputs and outputs, or according to a unified model. For example, the model can operate as an end-to-end model including each of the depicted components (e.g., outputting vehicle control signals based on inputs), including other combinations of components, or various components can be implemented as components of the same model. The illustrative examples of objects provided by the ML accelerator 1604 or another controller 1606 (e.g., a deterministic model or algorithm) should not be construed as limiting. Various functions can be implemented according to various components, or can be omitted, supplemented, or replaced using different components.

[0090] Based on several aspects, a method for visualizing perceptual information is provided. This method can be derived from... Figures 1-18 The method may be performed by one or more systems or components (including data processing system 100) depicted herein. For example, the method may be performed by one or more controllers of an electric vehicle 200 having a memory device communicatively coupled thereto.

[0091] Various operations of the method may include ACTs provided thereafter that can be inherited, copied, or adapted throughout the various aspects described in this disclosure.

[0092] The method may include an ACT (Active Action) receiving first sensor data from a first plurality of sensors of a first sensor type. The method may include an ACT receiving second sensor data from a second plurality of sensors of a second sensor type. The method may include generating an environmental map based on the first and second sensor data. The method may include an ACT (Active Action) identifying multiple objects and indicating routes based on the environmental map. The method may include classifying the multiple objects and routes. The method may include an ACT (Active Action) displaying multiple first visual representations corresponding to the multiple objects and second visual representations based on routes, based on the classification.

[0093] Some descriptions herein emphasize the structural independence of various aspects of system components or the grouping of the operations and responsibilities of these system components. Other groups performing similar overall operations are also within the scope of this application. Modules may be implemented in hardware or as computer instructions on a non-transitory computer-readable storage medium, and modules may be distributed across various hardware or computer-based components.

[0094] The system described above can provide any one of these components or multiple components of each component, and these components can be provided on a standalone system or on multiple instantiations in a distributed system. Furthermore, the system and method described above can be provided as one or more computer-readable programs or executable instructions embodied on or in one or more artifacts. The artifact can be cloud storage, a hard disk, a CD-ROM, a flash memory card, a PROM, RAM, ROM, or magnetic tape. Generally, the computer-readable program can be implemented in any programming language (such as LISP, PERL, C, C++, C#, PROLOG) or any bytecode language (such as JAVA). The software program or executable instructions can be stored as object code on or in one or more artifacts.

[0095] Examples and non-limiting module implementation elements include a sensor 105 that provides any value as defined herein, a sensor 105 that provides any value that is a precursor to the value defined herein, data link or network hardware including communication chips, oscillating crystals, communication links, cables, twisted pairs, coaxial cabling, shielded cabling, transmitters, receivers, or transceivers, logic circuits, hardwired logic circuits, reconfigurable logic circuits in a specific non-transient state configured according to the module specification, any actuator including at least an electric actuator, hydraulic actuator, or pneumatic actuator, solenoids, operational amplifiers, analog control elements (springs, filters, integrators, adders, dividers, gain elements), or digital control elements.

[0096] The subject matter and operations described herein can be implemented in digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in combinations thereof. The subject matter described herein can be implemented as one or more computer programs (e.g., one or more computer program instruction circuits) encoded on one or more computer storage media for execution by or control of the operation of a data processing device. Alternatively or additionally, program instructions can be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be or is included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. Although the computer storage medium is not a propagating signal, it can be a source or destination of computer program instructions encoded in artificially generated propagating signals. Computer storage media may also be, or be included in, one or more separate components or media (e.g., multiple CDs, discs, or other storage devices including cloud storage). The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0097] The terms "computing device," "component," or "data processing apparatus," etc., encompass a variety of devices, apparatuses, and machines for processing data, including, by example, programmable processors, computers, systems-on-a-chip, or many or combinations of the foregoing. The device may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The device and execution environment can implement various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0098] Computer programs (also known as programs, software, software applications, applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may correspond to a file in a file system. A computer program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as multiple collaborating files (e.g., a file storing one or more modules, subroutines, or code sections). A computer program can be deployed to execute on one or more computers located in one place or distributed across multiple locations and interconnected via a communications network.

[0099] The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform actions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuitry, and the apparatus can be implemented as special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Devices suitable for storing computer program instructions and data can include non-volatile memory, media, and memory devices, by way of example including semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory can be supplemented or incorporated therein by special-purpose logic circuitry.

[0100] The subject matter described herein can be implemented in a computing system that includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer with a graphical user interface or web browser through which a user can interact with a specific implementation of the subject matter described herein, or a combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0101] Although the operations are depicted in a specific order in the accompanying drawings, these operations do not need to be performed in the specific order shown or sequentially, and it is not necessary to perform all the illustrated operations. The actions described herein may be performed in different orders.

[0102] Some illustrative embodiments have now been described, and it is clear that the foregoing is illustrative rather than restrictive, and has been presented by way of example. Specifically, while many of the examples presented herein involve specific combinations of method actions or system elements, these actions and elements can be combined in other ways to achieve the same purpose. Actions, elements, and features discussed in conjunction with one embodiment are not intended to exclude similar roles in other embodiments.

[0103] The wording and terminology used in this text are for descriptive purposes and should not be considered restrictive. The use of “comprising,” “including,” “having,” “containing,” “involving,” “characterized in,” and variations thereof is intended to cover the items listed thereafter, their equivalents, and additional items, as well as alternative embodiments consisting only of the items listed thereafter. In one embodiment, the system and method described herein consist of one, more than one, or all of the elements, actions, or components described herein.

[0104] Any reference to any specific embodiment or element or action of the system and method cited herein in the singular may also include specific embodiments that include multiple such elements, and any reference to any specific embodiment or element or action herein in the plural form may also include specific embodiments that include only a single element. References in the singular or plural form are not intended to limit the currently disclosed system or method, its components, actions, or elements to a single or multiple configuration. References to any action or element based on any information, action, or element may include specific embodiments in which the action or element is at least partially based on any information, action, or element.

[0105] Any specific implementation disclosed herein may be combined with any other specific implementation or scheme, and references to “specific implementation,” “some specific implementations,” “one specific implementation,” etc., are not necessarily mutually exclusive, but are intended to indicate that a particular feature, structure, or characteristic described in connection with a specific implementation may be included in at least one specific implementation or scheme. Such terms as used herein do not necessarily all refer to the same specific implementation. Any specific implementation may be incorporated into or exclusively combined with any other specific implementation in any manner consistent with the aspects and specific implementations disclosed herein.

[0106] A reference to "or" can be interpreted as inclusive, such that any term described using "or" can indicate a single, more than one, or all of the stated terms. A reference to at least one of the terms in a connected list can be interpreted as inclusive OR, indicating a single, more than one, or all of the stated terms. For example, a reference to "at least one of 'A' and 'B'" can include only "A", only "B", or both "A" and "B". Such references, used in conjunction with "include" or other open-ended terms, can include additional items.

[0107] In the case of reference numerals following technical features in the drawings, detailed embodiments, or any claims, the inclusion of reference numerals is to enhance the comprehensibility of the drawings, detailed embodiments, and claims. Therefore, the presence or absence of reference numerals does not limit the scope of any claim element.

[0108] Modifications may be made to the components and operations without substantially departing from the teachings and advantages of the subject matter disclosed herein, such as variations in the size, dimensions, structure, shape and proportions, parameter values, installation arrangement, use of materials, color, and orientation of the various components. For example, an integrally formed component may be composed of multiple parts or elements, the positions of the components may be reversed or otherwise varied, and the nature, number, or position of the discrete components may be changed or altered. Other substitutions, modifications, alterations, and omissions may also be made in the design, operating conditions, and arrangement of the disclosed components and operations without departing from the scope of this disclosure.

[0109] For example, the descriptions of positive and negative electrical characteristics may be reversed. Further descriptions of relative parallel, perpendicular, vertical, or other positioning or orientation include variations within + / -10% or + / -10 degrees of purely vertical, parallel, or perpendicular positioning. Unless otherwise expressly stated, references to "approximately," "substantially," or other terms of degree include variations within + / -10% of a given measurement, unit, or range. Coupled elements may be directly coupled to each other or electrically, mechanically, or physically coupled using intermediary elements. Therefore, the scope of the systems and methods described herein is indicated by the appended claims rather than the foregoing description, and variations falling within the meaning and scope of the equivalence of the claims are included therein.

Claims

1. A system comprising one or more processors coupled to a memory, the system being configured to: Receive first sensor data from a first plurality of sensors of a first sensor type; Receive second sensor data from a second plurality of sensors of a second sensor type; An environmental map is generated based on the data from the first sensor and the data from the second sensor. Based on the environmental map, multiple objects are identified and route instructions are provided; The multiple objects and the route are classified; and Based on the classification, multiple first visual representations corresponding to the multiple objects and second visual representations based on the route are displayed.

2. The system of claim 1, further comprising a system for generating an environmental map based on the fusion of the following: The first feature extracted from the first sensor data; and The second feature extracted from the data of the second sensor. The identification of the plurality of objects is based on a third feature extracted from the environment map.

3. The system according to claim 1, wherein the identification of the route includes: Lane marking recognition; as well as The flow of one or more of the plurality of objects.

4. The system according to claim 1, wherein, The plurality of first visual representations corresponding to the classification include at least one first visual representation for the following: pedestrian; Vulnerable road users; Motor vehicles; and Traffic control equipment.

5. The system of claim 1, further comprising a system for performing the following operations: After identifying the plurality of objects and the indication of the route, and before displaying the plurality of objects and the route, the positions of the plurality of objects and the route are verified based on the time dependency between their positions and previous positions.

6. The system of claim 1, wherein the first sensor type and the second sensor type are optical cameras.

7. The system of claim 1, wherein the first sensor type is an optical camera, and the second sensor type is one of an ultrasonic sensor, a radar sensor, or a LiDAR sensor.

8. The system of claim 1, further comprising a system for performing the following operations: The identification of the plurality of objects is periodically updated; and The identification of the route indication is updated asynchronously with the update of the identification of the plurality of objects.

9. The system according to claim 1, wherein: Identifying the plurality of objects includes identifying a first object among a plurality of objects that are in the same lane as the vehicle, which includes the first plurality of sensors and the second plurality of sensors, and that are in front of the vehicle; and Depict a plurality of first visual representations corresponding to the first object, wherein the first object has increased salience relative to other objects among the plurality of objects.

10. The system according to claim 1, wherein, The display of the first visual representation and the second visual representation is based on the receipt of the user's selection of the driving mode.

11. The system of claim 1, further comprising a system for performing the following operations: Detecting omissions in the data from the first sensor; and In response to the detection of the omission, the environmental map is generated based on the second sensor data.

12. The system of claim 1, further comprising a system for classifying the route based on at least one of stored route data or Global Navigation Satellite System (GNSS) sensors.

13. The system of claim 1, further comprising a system for performing the following operations: Encode the first sensor data into a first data structure; Extract the first feature from the first data structure; The second sensor data is encoded into a second data structure; Extract the second feature from the second data structure; and The first feature and the second feature are used to generate the environment map.

14. A vehicle, the vehicle comprising: Cockpit displays; and One or more processors, said one or more processors coupled to memory, said one or more processors being used for: Receive first sensor data from a first plurality of sensors of a first sensor type; Receive second sensor data from a second plurality of sensors of a second sensor type; An environmental map is generated based on the data from the first sensor and the data from the second sensor. Based on the environmental map, multiple objects are identified and route instructions are provided; The multiple objects and the route are classified; and Based on the classification, multiple first visual representations corresponding to the multiple objects and a second visual representation based on the route are presented on the cockpit display.

15. The vehicle of claim 14, further comprising the one or more processors, the one or more processors being configured to: In response to the plurality of objects and the indication of the identified route based on the environment map, a navigation action is performed.

16. A method, the method comprising: One or more processors receive first sensor data from a first plurality of sensors of a first sensor type; The one or more processors receive second sensor data from a second plurality of sensors of a second sensor type; An environmental map is generated by the one or more processors based on the first sensor data and the second sensor data; The one or more processors identify multiple objects and route indications based on the environment map; The plurality of objects and the routes are classified by the one or more processors; and Based on the classification, the one or more processors display multiple first visual representations corresponding to the plurality of objects and second visual representations based on the routes.

17. The method of claim 16, comprising: The one or more processors classify the objects among the plurality of objects into confrontation objects; The processors display the self-lane in a subset of the region between lane markers corresponding to the self-lane, based on the classification of the confrontation objects. and The restricted portion of the region is displayed by the one or more processors based on the classification of the confrontation objects.

18. The method of claim 16, comprising: The one or more processors receive the instruction for lane change; The occupancy of lanes adjacent to the self-lane is determined by the one or more processors; and The destinations in the adjacent lanes are displayed by the one or more processors.

19. The method of claim 16, comprising: The one or more processors receive an indication of the target speed of the vehicle; The speed of a portion of the plurality of objects in a lane adjacent to the self-lane is identified by the one or more processors; and The one or more processors adjust the target vehicle speed to reduce the difference between the target vehicle speed and the speed associated with portions of the plurality of objects.

20. The method of claim 16, comprising: Determine the intersection point between the first lane and the second lane; and Based on the detection of vulnerable road users in the second lane, an indication of the intersection is presented before reaching the intersection.

Images