Enhanced user interface generation for parking based on occupancy machine learning model

Abstract

Systems and methods for enhanced user interface generation for parking based on an occupancy machine learning model. An example method includes acquiring images from a plurality of image sensors, the plurality of image sensors being placed around a vehicle; calculating a forward pass through the occupancy network to output at least information for each angular range around the vehicle reflecting whether the object is within a threshold distance of the vehicle within a single range and an estimated distance to the object; and presenting a user interface via a display of the vehicle, the user interface depicting a graphical representation of the vehicle and the output information.

Description

[0001] Cross-references to related applications This application claims priority to U.S. Provisional Application No. 63 / 515,033, filed July 21, 2023, the entire contents of which are incorporated herein by reference and used for all purposes. Technical Field

[0002] This disclosure relates to the development and implementation of machine learning models, and more specifically to the development and implementation of machine learning models to update user interfaces, thereby indicating one or more aspects of the relative position of an object compared to another object. Background Technology

[0003] Neural networks are relied upon for various purposes and are increasingly forming the foundation of technologies. For example, neural networks can be used to perform object classification on images acquired via user devices (e.g., smartphones). In this example, the neural network could represent a convolutional neural network, which applies convolutional layers, pooling layers, and one or more fully connected layers to classify objects depicted in an image. As another example, neural networks can be used for text translation between languages. For this example, the neural network could represent a recurrent neural network.

[0004] Complex neural networks are additionally used to enable autonomous or semi-autonomous driving functions in vehicles. For example, drones can partially utilize neural networks to navigate in real-world areas. In this example, the drone can use sensors to detect approaching objects and navigate around them. As another example, cars or trucks can implement neural networks to navigate in real-world areas. Currently, such neural networks may rely on expensive or error-prone sensors for input. Furthermore, such neural networks may lack accuracy in detecting and classifying moving and stationary (e.g., fixed) objects. This, in turn, can lead to inadequate autonomous or semi-autonomous driving performance. Summary of the Invention

[0005] To facilitate the generation of user interfaces during vehicle operation, this paper provides systems and methods for generating enhanced user interfaces for parking based on occupancy machine learning models. For example, the disclosed techniques relate to enhancements for autonomous or semi-autonomous (collectively referred to herein as autonomous) driving of a vehicle using image sensors (e.g., cameras) placed around the vehicle. Thus, the vehicle can navigate in real-world areas using vision-based sensor information.

[0006] This application describes vision-based machine learning models that rely on increased software complexity to enable reduced sensor-based hardware complexity, while increasing accuracy. For example, in some embodiments only image sensors are used. By using image sensors, such as cameras, the described models are able to implement a complex simulation of human vision-based driving. As will be described, the machine learning models can take images from the image sensors and combine (e.g., stitch or fuse) information included in the images. For example, the information can be combined into a vector space, which is then further processed by the machine learning models to extract objects, signals associated with the objects, etc.

[0007] In some embodiments, a system is disclosed. The system can include at least one processor configured to: receive data associated with a first image and a second image, the data associated with the first image and the second image generated by one or more sensors of a robotic system; determine, based on the first image, the second image, and an occupancy network, a plurality of distances to one or more objects relative to the robotic system, the occupancy network configured to receive, as input, the data associated with the first image and the second image and output the plurality of distances to the one or more objects relative to the robotic system; generate, based on the plurality of distances to the one or more objects relative to the robotic system, a graphical user interface, the graphical user interface including a representation of the robotic system and the one or more objects positioned relative to the robotic system; and provide, to at least one processor associated with a display screen, data associated with the graphical user interface, the data associated with the graphical user interface configured to cause the at least one processor to display the graphical user interface on the display screen.

[0008] In some implementations, the one or more processors configured to generate the graphical user interface are configured to: determine a contour of a portion of each of the one or more objects positioned relative to the robotic system; determine a placement of the contour relative to each of the one or more objects positioned relative to the robotic system; and generate the graphical user interface based on the contour and the placement of the contour relative to each of the one or more objects. The one or more processors configured to determine the contour can be configured to determine the contour of the portion of each of the one or more objects based on a distance in the plurality of distances corresponding to each of the one or more objects. In some implementations, the one or more processors configured to determine the contour are configured to determine a profile of the contour of the portion of each of the one or more objects positioned relative to the robotic system based on the plurality of distances corresponding to each of the one or more objects.

[0009] In some implementations, one or more processors configured to determine contours are configured to: determine contour profiles of a portion of the contour of each of the one or more objects located relative to the robot system, based on multiple distances corresponding to each of the one or more objects, each contour profile being represented in gradient form. One or more processors configured to determine multiple distances relative to the robot system to the one or more objects may be configured to: determine a subset of distances that satisfy a threshold range of distances. One or more processors configured to generate a graphical user interface based on multiple distances relative to the robot system to the one or more objects may be configured to: generate a graphical user interface based on multiple distances to the one or more objects that satisfy the threshold range of distances. In some implementations, one or more processors configured to generate a graphical user interface based on multiple distances relative to the robot system to the one or more objects are configured to: abandon generating a graphical user interface based on multiple distances to the one or more objects that do not satisfy the threshold range of distances.

[0010] In some embodiments, one or more processors configured to determine multiple distances relative to the robot system to one or more objects are configured to: determine the height of each of the multiple distances based on a first image and a second image; and determine that the height of each of the multiple distances satisfies a threshold range of height. One or more processors configured to generate a graphical user interface based on the multiple distances relative to the robot system to one or more objects may be configured to: generate the graphical user interface based on the multiple distances to one or more objects associated with heights that satisfy the threshold range of height. In some embodiments, one or more processors configured to generate a graphical user interface based on the multiple distances to one or more objects associated with heights that satisfy the threshold range of height are configured to: abandon generating a graphical user interface based on the multiple distances to one or more objects associated with heights that do not satisfy the threshold range of height.

[0011] In another embodiment, a method is disclosed. The method may include: at least one processor receiving data associated with a first image and a second image, the data being associated with the first image and the second image generated by one or more sensors of a robot system; at least one processor determining, based on the first image and the second image, multiple distances relative to the robot system and to one or more objects; at least one processor generating a graphical user interface (GUI) based on the multiple distances relative to the robot system and to the one or more objects, the GUI including representations of the robot system and one or more objects positioned relative to the robot system; and at least one processor providing the data associated with the GUI to at least one processor associated with a display screen, the data associated with the GUI being configured to cause at least one processor to display the GUI on the display screen.

[0012] In some implementations, generating a graphical user interface may include: at least one processor determining a contour of a portion of each of one or more objects positioned relative to a robot system; at least one processor determining the placement of the contour relative to the respective objects of the one or more objects positioned relative to the robot system; and at least one processor generating a graphical user interface based on the contour and the placement of the contour relative to the respective objects. Determining the contour may include: at least one processor determining the contour of a portion of each of the one or more objects based on a distance among a plurality of distances corresponding to each of the one or more objects. In some implementations, determining the contour may include: at least one processor determining a contour profile of the contour of a portion of the contour of a portion of the one or more objects positioned relative to the robot system based on a plurality of distances corresponding to each of the one or more objects. In some implementations, determining the contour may include: at least one processor determining a contour profile of the contour of a portion of the contour of a portion of the one or more objects positioned relative to the robot system based on a plurality of distances corresponding to each of the one or more objects, each contour profile being represented in the form of a gradient.

[0013] In some implementations, determining multiple distances relative to the robot system to one or more objects may include: at least one processor determining a subset of distances that satisfy a threshold range. Generating a graphical user interface based on multiple distances relative to the robot system to one or more objects may include: at least one processor generating a graphical user interface based on multiple distances to one or more objects that satisfy the threshold range. In some implementations, generating a graphical user interface based on multiple distances relative to the robot system to one or more objects may include: abandoning the generation of a graphical user interface based on multiple distances to one or more objects that do not satisfy the threshold range.

[0014] In some implementations, determining multiple distances relative to the robot system to one or more objects may include: at least one processor determining the height of each of the multiple distances based on a first image and a second image; and at least one processor determining that the height of each of the multiple distances satisfies a threshold range of height. Generating a graphical user interface based on multiple distances relative to the robot system to one or more objects may include: at least one processor generating a graphical user interface based on multiple distances to one or more objects associated with heights that satisfy the threshold range of height. In some implementations, generating a graphical user interface based on multiple distances to one or more objects associated with heights that satisfy the threshold range of height may include: at least one processor abandoning the generation of a graphical user interface based on multiple distances to one or more objects associated with heights that do not satisfy the threshold range of height.

[0015] In yet another embodiment, a non-transitory computer-readable medium is disclosed. The non-transitory computer-readable medium may store instructions thereon that, when executed by one or more processors, cause the one or more processors to: receive data associated with a first image and a second image, the data being associated with the first image and the second image generated by one or more sensors of a robot system; determine, based on the first image and the second image, multiple distances relative to the robot system and to one or more objects; generate a graphical user interface based on the multiple distances relative to the robot system and to one or more objects, the graphical user interface including representations of the robot system and one or more objects positioned relative to the robot system; and provide data associated with the graphical user interface to at least one processor associated with a display screen, the data associated with the graphical user interface being configured to cause the at least one processor to display the graphical user interface on the display screen.

[0016] In some implementations, instructions to cause one or more processors to generate a graphical user interface may cause one or more processors to: determine the outline of a portion of each of one or more objects positioned relative to the robot system; determine the placement of the outline relative to the respective objects among the one or more objects positioned relative to the robot system; and generate a graphical user interface based on the outline and the placement of the outline relative to the respective objects.

[0017] In one embodiment, a method is disclosed. The method may include: at least one processor acquiring images from a plurality of image sensors positioned around a vehicle; at least one processor calculating forward passes via an occupancy network to at least output information for various angular ranges surrounding the vehicle, reflecting whether an object is within a threshold distance of the vehicle in a single range and an estimated distance to the object; and at least one processor rendering a user interface via a display of the vehicle, the user interface depicting a graphical representation of the vehicle and the output information. In some embodiments, the graphical representation of the output information includes a contour positioned around the vehicle. Various portions of the contour may correspond to various angular ranges. In some embodiments, a color is assigned to each portion based on the proximity of the vehicle to the object within the corresponding angular range. The graphical representation of the output information includes a contour positioned around obstacles near the vehicle. In some embodiments, various portions of the contour correspond to various angular ranges. A color may be assigned to each portion based on the proximity of the vehicle to the object within the corresponding angular range. The user interface may be updated based on time or vehicle movement.

[0018] By implementing the systems and methods described herein, the sensor suite used in robotic systems can be simplified while maintaining or improving the overall system accuracy. For example, by implementing the techniques described herein, the system can more accurately determine the distance relative to the robotic system to a point along an object. This more accurate distance can then be used to generate a graphical user interface that provides visual indications of the proximity of the robotic system to one or more objects. Furthermore, by more accurately determining and / or indicating the distance between the robotic system and objects in the environment, the robotic system can be operated to allow for closer manipulation of objects, reducing the chance of unintentional contact and thus improving operational safety.

[0019] It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory, and are intended to provide further explanation of the embodiments described herein. Attached Figure Description

[0020] Non-limiting embodiments of this disclosure are described by way of example in relation to the accompanying drawings, which are schematic and not intended to be drawn to scale. Unless indicated as background art, the drawings illustrate aspects of this disclosure.

[0021] FIG. 1A Components of a system for collecting and analyzing data generated during vehicle operation, according to an embodiment, are shown.

[0022] FIG. 1B Various sensors associated with the vehicle according to an embodiment are shown.

[0023] FIG. 1C The components of a vehicle according to an embodiment are shown.

[0024] FIG. 2A A block diagram of an example autonomous driving system that outputs obstacle occupancy information according to an embodiment is shown.

[0025] FIGS. 2B-2D An example is shown. FIG. 2A A block diagram of an example implementation of an autonomous driving system.

[0026] FIG. 3 This is a flowchart of an example process for updating a user interface to depict obstacle occupancy information according to an embodiment.

[0027] FIG. 4 This is an example user interface depicting an area near a vehicle when parked, according to an embodiment.

[0028] FIG. 5A An example user interface depicting occupancy near a vehicle when parked, according to an embodiment, is shown.

[0029] FIG. 5B The following is illustrated: updates based on vehicle movement according to an embodiment. FIG. 5A Example user interface.

[0030] FIG. 6 A block diagram illustrating an implementation of an occupancy network according to an embodiment is shown.

[0031] FIG. 7 This is a flowchart of an example process for updating a user interface to depict obstacle occupancy information according to an embodiment.

[0032] The embodiments of this disclosure and their advantages are best understood by referring to the following detailed description. It should be understood that similar reference numerals are used to identify similar elements shown in one or more of the figures, wherein the illustrations in the figures are for illustrative purposes and not for limiting the scope of the embodiments of this disclosure. Detailed Implementation

[0033] Reference will now be made to the illustrative embodiments depicted in the accompanying drawings, and these illustrative embodiments will be described herein using specific language. However, it should be understood that this is not intended to limit the scope of the claims or the present disclosure. Changes and further modifications to the features shown herein, as well as additional applications of the principles of the subject matter illustrated herein, will be considered within the scope of the subject matter disclosed herein to those skilled in the art and those skilled in the art. Other embodiments may be used and / or other changes may be made without departing from the spirit or scope of this disclosure. The illustrative embodiments described in the detailed description are not intended to limit the subject matter presented.

[0034] As is readily apparent, humans are capable of driving vehicles using vision and a deep understanding of their real-world environment. For example, humans can quickly identify objects (e.g., pedestrians, road signs, lane markings, vehicles, etc.) and use these objects to inform the vehicle's driving. To further assist humans in performing driving operations, systems have been developed to assist drivers in performing certain tasks (e.g., braking); these systems are known as Advanced Driver Assistance Systems (ADAS). These systems often involve the implementation of one or more machine learning models capable of identifying and characterizing objects positioned around the vehicle. However, these machine learning models can be prone to error. For example, a simple model might generate incorrect predictions about objects and their positions around the vehicle. Furthermore, the generation and processing of sensor data from different sensors can introduce inconsistencies that affect the downstream use of inferences generated based on sensor data.

[0035] As will be described, the technology described herein can be employed to assist a driver in operating and / or parking a vehicle at one or more predetermined speeds. Additionally, this technology can be used as part of automated or semi-automated driving technology to operate and / or park a vehicle at one or more predetermined speeds. As will be understood, parking lots often contain obstacles that require the driver to avoid when parking their vehicle. For example, parking spaces may have bollards positioned near the boundary of the parking space. In this example, bollards can help protect pedestrians, infrastructure, etc. These bollards may be difficult to see reliably, and the driver may have difficulty determining the distance from the driver's vehicle to the bollard. Similarly, when parking on a street, there may be traffic cones, debris, parking timers, etc., requiring the driver to navigate around them.

[0036] Example vehicles may include emitting sensors, such as ultrasonic sensors, to detect the distance to the aforementioned object while the driver is parking. For example, an ultrasonic sensor could be included at the rear of the vehicle to determine the distance to an object positioned behind the vehicle. In this example, the vehicle could output a warning (e.g., an audible warning) when the driver is reversing. While the sensor can provide useful information, this information is specific to objects located within the sensor's beam angle. For example, this could represent 10 degrees, 15 degrees, etc., extending from the rear. While some vehicles may have another ultrasonic sensor at the front, parking obstacles can be located anywhere around the vehicle. Including a sufficient number of ultrasonic sensors to allow 360-degree coverage may also be impractical. Therefore, these sensors may provide limited visibility of the distance and location of these objects while parked.

[0037] In contrast, the technique described herein uses image sensors placed around the vehicle to provide 360-degree image data of the vehicle's surroundings. The vehicle can use machine learning techniques to identify objects approaching it, even if those objects might touch the vehicle while it is being operated (e.g., while parked). In some embodiments, the machine learning techniques can identify objects below a threshold height. Example threshold heights can include the vehicle's height, the vehicle's height plus a threshold distance (e.g., 0.5 meters, 1 meter). Example threshold heights can also include 0.75 meters, 1 meter, 1.2 meters, 1.5 meters, 2.5 meters, etc. In this way, the vehicle can ensure that objects posing a risk of colliding with it can be identified.

[0038] Example machine learning techniques can rely on neural networks trained to classify objects in a received image. For instance, a neural network may include convolutional layers followed by fully connected layers trained to assign classes to objects depicted in the image. Neural networks can be used to detect certain objects within the visible range of a vehicle running the neural network, such as cars, trucks, road signs, pedestrians, etc.

[0039] While understanding what an object is, such as its classification, may be important for achieving safe and reliable autonomous driving, the specific classification may not be as crucial during parking. For example, the presence of an object may be more important than its specific type. In this example, the driver or autonomous driving system might be more inclined to identify any object near the vehicle. Additionally, neural networks can assign cuboids around the classified objects. In this way, the location, size, etc., of the classified objects can be determined. However, such cuboids are often inflexible because they are sized to broadly enclose the object. Since parking involves navigating in confined areas, understanding the actual outline or boundaries of objects in a real-world environment is highly beneficial.

[0040] The vision-based machine learning model described herein determines the occupancy of an object in three-dimensional space. For example, as in U.S. Provisional Patent Application No. 63 / 375,199, U.S. Patent Application No. 18 / 440,764, filed February 13, 2024, and Patent Cooperation Treaty (PCT) Application No. PCT / US2023 / 032214, filed September 7, 2023, the entire contents of each of these applications are incorporated herein by reference and used for all purposes. The vision-based machine learning model can separate three-dimensional space into voxels, each voxel occupying a portion of the space. The model can then at least indicate whether a voxel is occupied by an object. In this way, the model determines the voxels that form each object physically present in a real-world environment. Thus, oddly shaped objects or objects with extending members (e.g., a truck with a ladder hanging from the rear) can be accurately mapped to the space occupied by voxels.

[0041] The vision-based machine learning model described above can additionally determine parking-related information. For example, the model can separate the 360-degree surroundings of a vehicle into a threshold number of degree increments (also referred to herein as angular ranges). In this example, the 360-degree surroundings can be separated into individual azimuth ranges. In some embodiments, there can be 72 individual ranges, each 5 degrees. In some embodiments, there can be fewer, more, or an adjustable number of ranges. For each range, the machine learning model can output a binary classification indicating whether that range is occupied by an object. The model can also output the distance from the vehicle (e.g., from a self-driving car) to the object. This output information is described herein as obstacle occupancy information.

[0042] In some embodiments, to provide an easier driving experience or display useful information to the driver during autonomous driving, the vehicle may present a graphical representation of obstacle occupancy information (e.g., a graphical user interface). For example, a display within the vehicle may depict a graphical representation during vehicle operation. Example graphical representations are shown in... FIG. 4 The diagram illustrates this, including the graphical boundary surrounding the vehicle. Sections of the graphical boundary can be assigned colors, chosen based on the proximity of each section to the object (e.g., red could indicate the object's proximity to the vehicle). Another example graphical representation is shown in... FIG. 5A and FIG. 5B As shown, and included on objects close to the vehicle, the graphic boundary can be positioned near the curb and curb portions, which are assigned colors based on the vehicle's proximity to the curb.

[0043] Therefore, the disclosed technology solves the technical problem. While some vehicles utilize transmitting sensors, such as ultrasonic sensors, at the front and / or rear of the vehicle, these sensors may have limited visibility of objects approaching the vehicle. Additionally, these sensors can have variable accuracy with respect to objects positioned at different heights. For example, these sensors may struggle to reliably detect curbs approaching the vehicle. In contrast, the technology described herein uses image sensors to achieve a 360-degree field of view around the vehicle, which is not hindered by the limitations of ultrasonic sensors. Therefore, the output of the image sensor can be processed to determine a more accurate distance to the object relative to a robotic system, such as a self-driving vehicle, and subsequently implemented by downstream systems when generating a graphical user interface and / or operating the self-driving vehicle. These improvements also enhance the operational safety of the robotic system, enabling it to operate closer to objects that would otherwise be impossible without considering the error tolerance of some of the sensors described above (e.g., transmitting sensors).

[0044] While this document includes descriptions relating to autonomous vehicles (e.g., self-driving cars denoted as automobiles), it will be understood that the technology can be applied to other autonomous vehicles (e.g., other self-driving cars). For example, the machine learning models described herein can be used in part to autonomously operate unmanned ground vehicles, unmanned boats, etc. Additionally, in some embodiments, references to autonomous vehicles may refer to vehicles that can be placed in autonomous driving mode. For example, a vehicle can autonomously drive or navigate on highways, expressways, etc. In some embodiments, a vehicle can autonomously drive or navigate on urban roads. Vehicles can also perform automatic parking in parking lots, city streets, etc.

[0045] Now for reference FIG. 1A The diagram illustrates non-limiting examples of system components in which the methods and systems discussed herein can be implemented. For example, an analytics server can collect and analyze data generated during vehicle operation. The analytics server can then provide the device with the event data (described herein) based on a received request for event data specifying one or more event types.

[0046] Continue to refer to FIG. 1A Example environment 100 may include analysis server 110a, system database 110b, administrator computing device 120, vehicles 140a-c (collectively referred to as "vehicle 140" and individually referred to as "vehicle 140"), vehicle computing devices 141a-c (collectively referred to as "vehicle computing device 141" and individually referred to as "vehicle computing device 141"), and server 160. Environment 100 is not limited to the components described herein and may include additional or other components not shown for brevity, which will be considered to be within the scope of the embodiments described herein.

[0047] The components mentioned herein can be interconnected via network 130 (e.g., establishing a connection for communication). Examples of network 130 may include, but are not limited to, private or public local area networks (LANs), wireless local area networks (WLANs), metropolitan area networks (MANs), wide area networks (WANs), and the Internet. Network 130 may include wired and / or wireless connections that facilitate communication according to one or more standards and / or via one or more transmission media.

[0048] Communication on network 130 can be performed according to various communication protocols, such as Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), and Institute of Electrical and Electronics Engineers (IEEE) communication protocols. In one example, network 130 may include wireless communication according to the Bluetooth specification set or another standard or proprietary wireless communication protocol. In another example, network 130 may also include communication via a cellular network, such as GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), or EDGE (Enhanced Data for Global Evolution) networks.

[0049] Environment 100 illustrates an example of a system architecture and components that can be used to train and execute one or more Artificial Intelligence (AI) models, such as AI model 110c. Specifically, as in... FIG. 1A As depicted and described herein, analysis server 110a can use the methods discussed herein (e.g., by using data streams 172 and 174) to train AI model 110c using data retrieved from vehicle 140. Once AI model 110c has been trained, each of the vehicles 140 can access and execute the trained AI model 110c. For example, vehicle 140a with vehicle computing device 141a can transmit its camera feed to the trained AI model 110c and can generate graphics defining road segments in the environment (e.g., data stream 174). Furthermore, data ingested and / or predicted by AI model 110c for vehicle 140 (during inference time) can also be used to improve AI model 110c. Therefore, environment 100 depicts a continuous loop that periodically improves the accuracy of AI model 110c. Additionally, environment 100 depicts a loop in which data received by vehicle 140 can also be used for the training phase, in addition to the inference phase.

[0050] Analysis server 110a can be configured to collect, process, and analyze navigation data (e.g., images captured during navigation) and various sensor data collected from vehicle 140. The collected data can then be processed and prepared into a training dataset. This training dataset can then be used to train one or more AI models, such as AI model 110c. Analysis server 110a can also be configured to collect visual data from vehicle 140. Using AI model 110c (trained using the methods and systems discussed herein), analysis server 110a can generate datasets and / or occupancy maps for vehicle 140. Analysis server 110a can display the occupancy map on vehicle 140 and / or transmit the occupancy map / dataset to vehicle computing device 141, administrator computing device 120, and / or server 160.

[0051] exist FIG. 1A In this diagram, AI model 110c is shown as a component of system database 110b, but AI model 110c can be stored in different or separate components, such as cloud storage or any other data repository accessible by analytics server 110a.

[0052] The analytics server 110a can also be configured to display an electronic platform that shows various training attributes used to train the AI ​​model 110c. The electronic platform can be displayed on the administrator computing device 120, allowing analysts to monitor the training of the AI ​​model 110c. Examples of electronic platforms generated and hosted by the analytics server 110a could be web-based applications or websites configured to display training datasets collected from the vehicle 140 and / or the training status / metrics of the AI ​​model 110c.

[0053] The analytics server 110a can be any computing device, including processors and non-transitory machine-readable storage capable of performing the various tasks and processes described herein. Non-limiting examples of such computing devices may include workstation computers, laptops, server computers, etc. While environment 100 includes a single analytics server 110a, environment 100 may include any number of computing devices operating in a distributed computing environment, such as a cloud environment.

[0054] Vehicle 140 can represent various systems including electronic data sources that transmit data associated with previous or current navigation sessions to analytics server 110a. Vehicle 140 can be any device configured for navigation, such as vehicle 140a and / or truck 140c. Vehicle 140 is not limited to vehicles and may also include robotic devices. For example, vehicle 140 may include robot 140b, which can represent a general-purpose, bipedal, autonomous humanoid robot capable of navigating various terrains. Robot 140b may be equipped with software to achieve balance, navigation, perception, or interaction with the physical world. Robot 140b may also include various cameras configured to transmit visual data to analytics server 110a.

[0055] In some embodiments, the vehicle 140 may include one or more electric motors that cause movement of the vehicle 140. The electric motors may include, for example, induction motors, permanent magnet motors, etc. Batteries (e.g., one or more battery packs, each including multiple batteries) may be used to power the electric motors. The vehicle 140 may also include a propulsion system capable of setting gears (e.g., direction of travel) for the vehicle. For electric vehicles, the propulsion system may adjust the operation of the electric motors to change the direction of travel. Additionally, the vehicle 140 may include the autonomous driving system described herein, which can process data, such as images received from cameras positioned around the vehicle 140. The processor system 120 may additionally output information to and receive information (e.g., user input) from a display included in the vehicle. For example, the display may present... FIG. 4 The user interface 400 shown in the figure FIGS. 5A-5B The user interface 500 shown in the image.

[0056] Although referred to herein as a "self-driving vehicle," the self-driving vehicle 140 may or may not be an autonomous device configured for automatic navigation. For example, in some embodiments, the self-driving vehicle 140 may be controlled by a human operator or a remote processor. The self-driving vehicle 140 may include various sensors, such as... FIG. 1B The sensors depicted are configured to collect data as the vehicle 140 navigates on various terrains (e.g., roads). An analytics server 110a can collect data provided by the vehicle 140. For example, the analytics server 110a can acquire navigation session and / or road / terrain data from various sensors (e.g., images of the roads navigated by the vehicle 140), such that the collected data is ultimately used by the AI ​​model 110c for training purposes.

[0057] As used herein, a navigation session corresponds to a journey along the route of vehicle 140, whether the journey is autonomous or human-controlled. In some embodiments, the navigation session may be used for data collection and model training purposes. However, in some other embodiments, vehicle 140 may refer to a vehicle purchased by a consumer, and the purpose of the journey may be categorized as everyday use. A navigation session may begin when vehicle 140 moves more than a threshold distance (e.g., 0.1 miles, 100 feet) or exceeds a threshold speed (e.g., more than 0 mph, more than 1 mph, more than 5 mph) from a non-moving location. A navigation session may end when vehicle 140 is returned to a non-moving location and / or shut down (e.g., when the driver leaves the vehicle).

[0058] Vehicle 140 can represent a set of vehicles monitored by analytics server 110a to train AI model 110c. For example, the driver of vehicle 140a can authorize analytics server 110a to monitor data associated with their respective vehicle. Therefore, analytics server 110a can utilize various methods discussed herein to collect sensor / camera data and generate a training dataset to train AI model 110c accordingly. Analytics server 110a can then execute the trained AI model 110c to analyze the data associated with vehicle 140 and predict the occupancy map of vehicle 140. Furthermore, additional / ongoing data associated with vehicle 140 can be processed and added to the training dataset, allowing analytics server 110a to recalibrate AI model 110c accordingly. Thus, environment 100 depicts a loop in which navigation data received from vehicle 140 can be used to train AI model 110c. Vehicle 140 may include a processor that executes the trained AI model 110c for navigation purposes. During navigation, the autonomous vehicle 140 can collect additional data about its navigation session, which can be used to calibrate the AI ​​model 110c. In other words, the autonomous vehicle 140 represents a vehicle that can be used to train, execute / use, and recalibrate the AI ​​model 110c. In a non-limiting example, the autonomous vehicle 140 represents a vehicle purchased by a customer, which can autonomously navigate using the AI ​​model 110c while simultaneously improving the AI ​​model 110c.

[0059] The autonomous vehicle 140 can be equipped with various technologies that allow it to collect data from its surroundings and (potentially) navigate autonomously. For example, the autonomous vehicle 140 can be equipped with an inference chip to run autonomous driving software.

[0060] Various sensors used in each vehicle 140 can monitor the collected data associated with different navigation sessions and transmit the collected data to the analysis server 110a. FIG. 1B and FIG. 1CA block diagram of a sensor integrated into a vehicle 140 according to an embodiment is shown. (Refer to...) FIG. 1B and FIG. 1C The number and location of the sensors discussed can depend on the FIG. 1A The types of autonomous vehicles discussed herein. For example, robot 140b may include sensors different from those of vehicle 140a or truck 140c. For example, robot 140b may not include airbag activation sensor 170q. Furthermore, the sensors of vehicle 140a and truck 140c may be compatible with those in… FIG. 1C The sensors for vehicle 140a and truck 140c shown are positioned differently.

[0061] As discussed herein, various sensors integrated within each vehicle 140 can be configured to measure various data associated with each navigation session. An analytics server 110a can periodically collect the data monitored and gathered by these sensors, wherein the data is processed according to the methods described herein and used to train and / or execute the AI ​​model 110c to generate a occupancy map.

[0062] refer to FIG. 1B The vehicle 140 may include a user interface 170a. The user interface 170a may refer to the vehicle's computing device (e.g., in...). FIG. 1A The user interface 170a is a display screen integrated with or coupled to the interior of the vehicle (such as a head-up display, touchscreen, etc.). The user interface 170a may include input devices such as touchscreens, knobs, buttons, keyboards, mice, gesture sensors, steering wheels, etc. In various embodiments, the user interface 170a may be adapted to input other devices or sensors of the vehicle 140 (e.g., in…). FIG. 1B Sensors shown in the diagram, such as controller 170c, provide user input (e.g., as a signal and / or sensor information).

[0063] User interface 170a may also be implemented using one or more logical devices, which may be adapted to execute instructions, such as software instructions, to implement any of the various processes and / or methods described herein. For example, user interface 170a may be adapted to establish communication links, transmit and / or receive communications (e.g., sensor signals, control signals, sensor information, user input, and / or other information), or perform various other processes and / or methods. In another example, a driver may use user interface 170a to control the temperature of vehicle 140 or activate features of vehicle 140 (e.g., autonomous driving or steering system 170o, sometimes referred to as a processor system). Therefore, user interface 170a may be combined with other sensors described herein to monitor and collect driving session data. User interface 170a may also be configured to display various data generated / predicted by analysis server 110a and / or AI model 110c.

[0064] Orientation sensor 170b may be implemented as one or more of a compass, buoy, accelerometer, and / or other digital or analog devices capable of measuring the orientation of vehicle 140 (e.g., the magnitude and direction of roll, pitch, and / or yaw relative to one or more reference orientations such as gravity and / or magnetic north). Orientation sensor 170b may be adapted to provide heading measurements to vehicle 140. In other embodiments, orientation sensor 170b may be adapted to provide roll, pitch, and / or yaw rates of vehicle 140 using time series of orientation measurements. Orientation sensor 170b may be positioned and / or adjusted to perform orientation measurements relative to a specific coordinate system of vehicle 140.

[0065] The controller 170c can be implemented as any suitable logic device (e.g., a processing device, microcontroller, processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), memory storage device, memory reader, or other device or combination of devices) that can execute, store, and / or receive appropriate instructions, such as software instructions implementing control loops for controlling various operations of the vehicle 140. Such software instructions can also be implemented as methods for processing sensor signals, determining sensor information, providing user feedback (e.g., via user interface 170a), querying device operating parameters, selecting device operating parameters, or performing any of the various operations described herein.

[0066] The communication module 170e can be implemented as any wired and / or wireless interface, any wired and / or wireless interface being configured to transmit sensor data, configuration data, parameters, and / or other data and / or signals to...FIG. 1A Any features shown in the document (e.g., analysis server 110a). As described herein, in some embodiments, the communication module 170e may be implemented in a distributed manner, such that portions of the communication module 170e are located in... FIG. 1B The communication module 170e is implemented within one or more of the components and sensors shown. In some embodiments, the communication module 170e may delay the transmission of sensor data. For example, when the vehicle 140 does not have a network connection, the communication module 170e may store the sensor data in a temporary data storage device and transmit the sensor data when it is determined that the vehicle 140 has a correct network connection.

[0067] The speed sensor 170d can be implemented as an electronic pitot tube, a metering gear or wheel, a water speed sensor, a wind speed sensor, an air volume sensor (e.g., direction and amplitude), and / or other devices capable of measuring or determining the linear speed of the vehicle 140 (e.g., in the surrounding medium and / or aligned with the longitudinal axis of the vehicle 140) and providing such measurements as sensor signals that can be transmitted to various devices.

[0068] The gyroscope / accelerometer 170f can be implemented as one or more electronic sextants, semiconductor devices, integrated chips, accelerometer sensors, or other systems or devices capable of measuring the angular velocity / accelerometer and / or linear acceleration (e.g., direction and magnitude) of the vehicle 140 and providing such measurements as sensor signals that can be transmitted to other devices such as the analysis server 110a. The gyroscope / accelerometer 170f can be positioned and / or adjusted to perform such measurements relative to a specific coordinate system of the vehicle 140. In various embodiments, the gyroscope / accelerometer 170f can be used in conjunction with... FIG. 1B Implemented in a shared housing and / or module for other components depicted herein, to ensure a common reference frame or known transformations between reference frames.

[0069] The Global Navigation Satellite System (GNSS) 170h can be implemented as a global positioning satellite receiver and / or, for example, as a device capable of determining the absolute and / or relative position of the vehicle 140 based on radio signals received from satellite sources and / or terrestrial sources, and capable of providing such measurements as sensor signals that can be transmitted to various devices. In some embodiments, the GNSS 170h can be adapted to determine the vehicle 140's velocity, rate, and / or yaw rate (e.g., using a time series of position measurements), such as the yaw component of the vehicle 140's absolute velocity and / or angular velocity.

[0070] Temperature sensor 170i can be implemented as a thermistor, electronic sensor, electronic thermometer, and / or other device capable of measuring the temperature associated with vehicle 140 and providing such measurement as a sensor signal. Temperature sensor 170i can be configured to measure the ambient temperature associated with vehicle 140, such as cabin or dashboard temperature, and can be used to estimate the temperature of one or more components of vehicle 140.

[0071] The humidity sensor 170j can be implemented as a relative humidity sensor, an electronic sensor, an electronic relative humidity sensor, and / or another device capable of measuring the relative humidity associated with the vehicle 140 and providing such measurement as a sensor signal.

[0072] The steering sensor 170g can be adapted to physically adjust the heading of the vehicle 140 based on one or more control signals and / or user input provided by a logic device such as controller 170c. The steering sensor 170g may include one or more actuators and control surfaces of the vehicle 140 (e.g., a rudder or other type of steering or trim mechanism) and can be adapted to physically adjust the control surfaces to various positive and / or negative steering angles / positions. The steering sensor 170g can also be adapted to sense the current steering angle / position of such steering mechanism and provide such measurements.

[0073] The propulsion system 170k can be implemented as a propeller, turbine, or other thrust-based propulsion system, a mechanical wheeled and / or tracked propulsion system, a wind / sail-based propulsion system, and / or other types of propulsion systems that can be used to power the vehicle 140. The propulsion system 170k can also monitor the direction of power and / or thrust of the vehicle 140 relative to its coordinate reference system. In some embodiments, the propulsion system 170k can be coupled to and / or integrated with the steering sensor 170g.

[0074] The occupant restraint sensor 170l can monitor seatbelt detection and locking / unlocking components, as well as other occupant restraint subsystems. The occupant restraint sensor 170l may include various environmental and / or state sensors, actuators, and / or other devices that facilitate the operation of safety mechanisms associated with the operation of the vehicle 140. For example, the occupant restraint sensor 170l can be configured to monitor... FIG. 1B Other sensors depicted receive motion and / or status data. Occupant restraint sensor 170l can determine whether safety measures (such as seat belts) are in use.

[0075] Camera 170m can refer to one or more cameras integrated into the vehicle 140, and can include multiple cameras integrated (or modified) into the vehicle 140, such as in FIG. 1CAs depicted in the image. Camera 170m can be an inward-facing or outward-facing camera on vehicle 140. For example, as in... FIG. 1C As depicted, the vehicle 140 may include one or more inward-facing cameras that can monitor and collect images of passengers in the vehicle 140. The vehicle 140 may include eight outward-facing cameras. For example, the vehicle 140 may include a front-facing camera 170m-1, a front-view side camera 170m-2, a front-view side camera 170m-3, a rear-view side camera 170m-4 on each front fender, a camera 170m-5 on each side (e.g., integrated into the B-pillar), and a rear-facing camera 170m-6. In some embodiments, the cameras 170m may generate sensor data that is used by an autonomous driving or steering system 170o (also referred to as the autonomous driving system 170o) to generate a 360-degree view around the vehicle 140. While this disclosure is described in relation to the generation of data from one or more cameras 170m, in some embodiments, sensor data generated by one or more other types of sensors (e.g., Light Detection and Ranging (LiDAR) sensors, radar sensor 170n, ultrasonic sensor 170p, etc.) may be used to generate data associated with the environment in which the vehicle 140 operates and is processed according to the techniques described herein.

[0076] In some embodiments, one or more of the cameras 170m may be positioned within a camera housing located near the top of the windshield of the vehicle 140. For example, the cameras 170m may provide a forward view of a real-world environment in which the vehicle is being driven. In this example, the cameras 170m may include three image sensors laterally offset from each other. For example, the camera housing may include three image sensors pointing forward relative to the vehicle 140. In this example, the first camera 170m may have a wide-angle (e.g., fisheye) lens. The second camera 170m may have a standard or conventional lens (e.g., 35 mm equivalent focal length, 50 mm equivalent focal length, etc.). The third camera 170m may have a zoom or narrow-angle lens. In this way, three images with different focal lengths can be acquired in the forward direction relative to the vehicle 140.

[0077] In some embodiments, one or more cameras 170m may be rear-facing and positioned on the left side of the vehicle 140. For example, camera 170m may be placed on a portion of the fender of the vehicle 140. Similarly, camera 170m may be rear-facing and positioned on the right side of the vehicle 140. For example, camera 170m may be placed on a portion of the fender of the vehicle 140.

[0078] In some embodiments, camera 170m may be positioned on a door pillar of vehicle 140 on the left and / or right side of vehicle 140. Camera 170m may be angled such that it points downwards and at least partially forwards relative to vehicle 140. In some embodiments, camera 170m may be angled such that it points downwards and at least partially backwards. Similarly, camera 170m may be positioned on a door pillar on the right side of vehicle 100. As described above, camera 170m may be angled such that it points partially downwards and forwards or backwards.

[0079] In some embodiments, the camera 170m may be positioned such that it points towards the rear of the vehicle 140 and acquires an image in the rearward direction of the vehicle 140 (e.g., assuming the vehicle 140 is moving forward). In some embodiments, the camera 170m may be placed above the license plate of the vehicle 140.

[0080] In some embodiments, the camera 170m, radar 170n, and / or ultrasonic sensor 170p can generate data based on specific frequencies, such as 30 Hz, 36 Hz, 51 Hz, 65 Hz, etc. In the example, sensor data can be generated at varying frequencies based on the placement of the sensors and / or sensors used to generate the sensor data. As described below, the sensor data generated by the sensors described herein can be processed by the autonomous driving system 170o to determine occupancy information associated with the real-world environment of the vehicle 140.

[0081] In some embodiments, radar 170n and ultrasonic sensor 170p can be configured to monitor the distance between the vehicle 140 and other objects, such as other vehicles or stationary objects (such as trees or garage doors). The vehicle 140 may also include an autonomous driving system 170o, which is configured to autonomously navigate the vehicle 140 using data collected via various sensors, such as radar 170n, speed sensor 170d, and / or ultrasonic sensor 170p.

[0082] Continue to refer to FIG. 1BThe autonomous driving system 170o may include and / or be implemented by a computing device, which is the same as or similar to the vehicle computing device 171. In some embodiments, the autonomous driving system 170o may analyze various data collected by one or more sensors described herein to identify driving data. For example, the autonomous driving system 170o may receive sensor data generated by one or more sensors described herein (e.g., camera 170m, radar 170n, ultrasonic sensor 170p, etc.) and calculate the risk of a forward collision based on the speed of the vehicle 140 and the distance between the vehicle 140 and another vehicle on the road. The autonomous driving system 170o may also determine whether the driver is touching the steering wheel. The autonomous driving system 170o may transmit the analyzed data to various features discussed herein, such as the analysis server 110a.

[0083] In some embodiments, the autonomous driving system 170o can acquire images from the camera 170m using the vision-based machine learning model described herein, and detect objects and information associated with those objects. Based on the objects, the autonomous driving system 170o can adjust one or more driving characteristics or features. For example, the autonomous driving system 170o can cause the vehicle 170o to turn, decelerate, brake, accelerate, etc. The autonomous driving system 170o can also render a user interface or cause the rendering of a user interface for presentation via a display. FIGS. 4-5B It includes a sample user interface.

[0084] In some embodiments, the autonomous driving system 170o may include one or more matrix processors configured to rapidly process information associated with a machine learning model. The autonomous driving system 170o may be used to perform convolutions associated with the forward pass via a convolutional neural network. For example, convolutions may be performed on input data and weight data. The autonomous driving system 170o may include multiple multiply-accumulate units that perform the convolutions. As an example, the matrix processors may use already organized or formatted input and weight data to facilitate larger convolution operations. In this example, the input data may be in the form of a three-dimensional matrix or tensor (e.g., two-dimensional data spanning multiple input channels). In this example, the output data may span multiple output channels. Therefore, the autonomous driving system 170o can process larger input data by merging or flattening each two-dimensional output channel into a vector, such that an entire channel or a significant portion of a channel can be processed by the autonomous driving system 170o. As another example, data can be efficiently reused, allowing weight data to be shared across convolutions. For an output channel, the weight data described herein may represent the weight data (e.g., a kernel) used to compute that output channel. Additional examples of autonomous driving systems 170o that may use one or more matrix processors are described in U.S. Patent Nos. 11,157,287, 11,409,692 and 11,157,441, the entire contents of which are incorporated herein by reference and form part of this disclosure, as set forth herein.

[0085] The airbag activation sensor 170q can anticipate or detect a collision and cause the activation or deployment of one or more airbags. The airbag activation sensor 170q can transmit data regarding airbag deployment, including data associated with the event that caused the deployment.

[0086] Return to reference FIG. 1A Administrator computing device 120 can represent a computing device operated by a system administrator. Administrator computing device 120 can be configured to display data retrieved or generated by analytics server 110a (e.g., various analytical metrics and risk scores), wherein the system administrator can monitor various models used by analytics server 110a, review feedback, and / or facilitate the training of AI model 110c maintained by analytics server 110a.

[0087] The vehicle 140 can be any device configured to navigate various routes, such as vehicle 140a or robot 140b. (See reference...) FIGS. 1B-1CAs discussed, the vehicle 140 may include various telemetry sensors. The vehicle 140 may also include a vehicle computing device 141. Specifically, each vehicle may have its own vehicle computing device 141. For example, truck 140c may have a vehicle computing device 141c. For simplicity, the vehicle computing device is collectively referred to as vehicle computing device 141. Vehicle computing device 141 can control the presentation of content on the infotainment system of the vehicle 140, process commands associated with the infotainment system, aggregate sensor data, manage data communication with electronic data sources, receive updates, and / or transmit messages. In one configuration, vehicle computing device 141 communicates with an electronic control unit. In another configuration, vehicle computing device 141 is an electronic control unit. Vehicle computing device 141 may include a processor and a non-transitory machine-readable storage medium capable of performing the various tasks and processes described herein. For example, the AI ​​model 110c described herein may be stored and executed (or directly accessed) by vehicle computing device 141. Non-limiting examples of the vehicle computing device 141 may include a vehicle multimedia and / or display system.

[0088] In one example of training AI model 110c, analytics server 110a can collect data from vehicle 140 to train AI model 110c. Before executing AI model 110c to generate or predict graphics of defined road segments, analytics server 110a can train AI model 110c using various methods. Training allows AI model 110c to ingest data from one or more cameras of one or more vehicles 140 (without receiving radar data) and predict occupancy data of the vehicle's surrounding environment. The operations described in this example can be performed by... FIG. 1A and FIG. 1B The distributed computing system described herein operates on any number of computing devices (e.g., the processor of vehicle 140).

[0089] To train the AI ​​model 110c, the analysis server 110a may first employ one or more of the vehicles 140 driving a specific route. While driving, the vehicles 140 may generate navigation session data using one or more of their sensors, including one or more cameras. For example, one or more vehicles 140 equipped with various sensors may navigate a specified route. As one or more vehicles 140 traverse terrain, their sensors may capture continuous (or periodic) data about their surroundings. The sensors may indicate the occupancy status of the surroundings of one or more vehicles 140. For example, sensor data may indicate various objects of mass in the surroundings of one or more vehicles 140 as they navigate their routes.

[0090] In operation, when one or more vehicles 140 are navigating, sensors of the one or more vehicles 140 collect data and transmit the data to an analytics server 110a, as depicted in data stream 172. In some embodiments, the one or more vehicles 140 may include one or more high-resolution cameras that capture a continuous stream of visual data from the surrounding environment of the one or more vehicles 140 as they navigate a route. The analytics server 110a can then use the camera feed to generate a second dataset, wherein the second dataset includes visual elements / descriptions of different voxels of the surrounding environment of the one or more vehicles 140. In operation, when one or more vehicles 140 are navigating, cameras of the one or more vehicles 140 collect data and transmit the data to the analytics server 110a, as depicted in data stream 172. For example, vehicle computing device 141 can use data stream 172 to transmit image data to analytics server 110a.

[0091] Analysis server 110a can generate a training dataset using data collected from vehicle 140 (e.g., camera feeds received from vehicle 140). The training dataset may identify or include a set of examples. Each example may identify or include input data and expected output data derived from the input data. In each example, the input may include collected data, such as sensor data from vehicle 140 (e.g., video or images from one or more cameras) and map data (e.g., a navigation map). The output may include environmental features (e.g., attributes acquired from the sensor data), map features (e.g., attributes in the navigation map, such as topological features and road layout), classifications (e.g., topology types), and output tokens (e.g., a combination of environmental features, map features, and classifications) to be included in a graph defining a road segment, etc. In some embodiments, the output may be created by a human reviewer who examines the input data.

[0092] Using a training dataset, analytics server 110a can feed a series of training datasets into AI model 110c and obtain a set of predicted outputs (e.g., environmental features, map features, classifications, and output tokens). Analytics server 110a can then compare the predicted data with ground truth data to determine discrepancies and train AI model 110c by adjusting its internal weights and parameters proportional to the determined discrepancies according to a loss function. Analytics server 110a can train AI model 110c in a similar manner until the trained AI model 110c's predictions are accurate to a certain threshold (e.g., recall or precision).

[0093] In some embodiments, the analysis server 110a may use supervised training methods. For example, using received ground truth and visual data, the AI ​​model 110c may train itself to predict outputs. Therefore, during training, the AI ​​model 110c may receive sensor data and map data, analyze the received data, and generate tokens. In some embodiments, the analysis server 110a may use unsupervised methods in which the training dataset is not labeled. Since labeling data within the training dataset can be time-consuming and may require excessive computational power, the analysis server 110a may utilize unsupervised training techniques to train the AI ​​model 110c.

[0094] As the AI ​​model 110c is established, the analytics server 110a can transmit, send, or otherwise distribute the weights of the AI ​​model 110c to each of the autonomous vehicle computing devices 141a-c. Upon receiving, the autonomous vehicle computing devices 141a-c can store and maintain the AI ​​model 110c on local memory. Once stored and loaded, the autonomous vehicle computing devices 141a-c can use it to process newly acquired data (e.g., sensor and map data) to create graphics defining road segments for autonomous navigation of the corresponding autonomous vehicle 140a-c through its environment. Sometimes, the analytics server 110a can transmit, send, or otherwise distribute updated weights of the AI ​​model 110c to update the instance of the AI ​​model 110c on the autonomous vehicle computing devices 141a-c.

[0095] Now for reference FIG. 2A This illustrates a non-limiting example of implementation 200a including an autonomous driving system 202, which is configured to determine obstacle occupancy information 210 based on image data 206 received from a sensor 204 (e.g., the same or similar sensor as a camera 170m, radar 170n, ultrasonic sensor 170p, etc.). In some embodiments, the autonomous driving system 202 and FIG. 1B The autonomous driving system is the same as or similar to 170o.

[0096] In some embodiments, image data 206 includes images surrounded by a vehicle (e.g., images of a vehicle surrounded by a vehicle). FIG. 1C The image data is associated with one or more images generated by one or more image sensors placed in a vehicle (identical to or similar to vehicle 140). In some embodiments, there may be eight (8) cameras, thus in FIG. 2AImage data 206 in the diagram represents eight (8) images. For example, the top row of image data 206 includes three images from a front-facing image sensor (e.g., images from the main camera, narrow camera, and fisheye camera); the middle row of image data 206 includes two images (e.g., images from cameras associated with the left and right pillars of the vehicle); and the bottom row of image data 206 includes three images from a rear-facing image sensor (e.g., images from the left blind spot camera, right blind spot camera, and backup camera). As described above, image data 206 can be received at a specific frequency such that the illustrated images represent specific timestamps of the images. In some embodiments, image data 206 can represent a High Dynamic Range (HDR) image. For example, different exposures can be combined to form an HDR image. As another example, images from image sensors can be preprocessed (e.g., using a machine learning model) to convert the images into HDR images.

[0097] In some embodiments, one or more of the sensors 204 (e.g., cameras) can acquire multiple exposures, each with a different shutter speed or integration time. For example, the difference between the different integration times can be greater than a threshold time difference. In this example, there can be three integration times, spaced approximately one order of magnitude apart in time. The autonomous driving system 202 or different computing devices associated with the vehicle can be configured to determine (e.g., select) one of the exposures based on a shearing measurement associated with the image. In some embodiments, the autonomous driving system 202 or different computing devices can form an image based on a combination of multiple exposures. For example, the autonomous driving system 202 can select each pixel of the formed image from one of the multiple exposures based on pixels that include or exclude values ​​(e.g., red, green, blue) that are sheared (e.g., exceed a threshold pixel value).

[0098] The autonomous driving system 202 can execute a vision-based machine learning model 208 (which may be the same as or similar to the occupancy network 217 described herein) to process image data 206. See also... FIG. 2B and FIG. 2C An example of a vision-based machine learning model 208 described for occupancy network 217 is described in more detail.

[0099] As described herein, a vision-based machine learning model 208 can combine information included in an image. For example, each image can be provided to a specific backbone. In some embodiments, the backbone can represent a convolutional neural network that extracts features (e.g., high-dimensional features in image space). In some examples, the outputs of these backbone networks can be combined (e.g., formed as tensors) or provided as separate tensors to one or more other parts of the model. In some embodiments, an attention part (e.g., a cross-attention part) can receive combinations of features. The attention part can query three-dimensional (3D) points to determine whether each 3D point is occupied. In some embodiments, the attention part generates 3D occupancy features and can be provided, for example, to an upsampling part (e.g., deconvolution) to produce denser features.

[0100] As will be described, the combined output can then be used to determine different obstacle occupancy information 210 associated with the real-world environment. Example occupancy information 210 may include a first vector or other data structure comprising a threshold number of elements, each element corresponding to a portion of the space surrounding the vehicle in which the autonomous driving system 202 is installed and / or implemented. For example, each portion of the space may represent an angular range (e.g., 3 degrees, 5 degrees, 6 degrees, etc.) extending from the vehicle in which the autonomous driving system 202 is implemented or installed. The first vector or other data structure can then indicate whether the angular range includes an object. Obstacle occupancy information 210 may also include a second vector or other data structure indicating the distance to the object within the angular range. Thus, if there is an object (e.g., a portion of an object) between the vehicle and (e.g., 180–185 degrees, assuming 0 degrees is forward), the first vector will indicate the presence of the object, and the second vector will indicate the distance to that portion of the object.

[0101] Now for reference FIGS. 2B-2D This illustrates a non-limiting example of implementation 200b including an autonomous driving system 202, which is configured to determine obstacle occupancy information 224 based on image data 206 received from example sensors 204 described herein (e.g., sensors identical or similar to camera 170m, radar 170n, ultrasonic sensor 170p, etc.). In some embodiments, the autonomous driving system 202 and... FIG. 1B The autonomous driving system 170o is the same as or similar to it. Furthermore, in some embodiments, the obstacle occupancy information 224 is the same as... FIGS. 2A-2D The obstacle occupancy information is the same as or similar to 210.

[0102] FIGS. 4-5BThis includes a block diagram illustrating details of an occupancy network 217 that outputs obstacle occupancy information 224. Occupancy network 217 can be executed by one or more systems associated with the vehicle 140, as described herein. Therefore, actions associated with occupancy network 217 can be understood as being executed by the autonomous driving system 202.

[0103] In the example shown, images 214a-214h are received by occupancy network 217. These images 214a-214h can be acquired from image sensors, such as camera 170m, placed around the vehicle 140. Occupancy network 217 includes feature engine 212 (e.g., a backbone portion), which receives the corresponding images as input. Therefore, feature engine 212 processes the raw pixels included in images 214a-214h. In some embodiments, feature engine 212 can be a convolutional neural network. For example, each backbone network may have 5 convolutional layers, 10 convolutional layers, 15 convolutional layers, etc. In some embodiments, feature engine 212 may include residual blocks, residual networks modulated by recurrent neural networks, etc. Additionally, feature engine 212 may include a weighted bidirectional feature pyramid network (BiFPN). The output of BiFPN can represent multi-scale features determined based on images 214a-214h. In some embodiments, Gaussian blur may be applied to portions of the images during training and / or inference time. For example, road edges may be sharp because they are clearly defined in an image. In this example, a Gaussian blur can be applied to the road edges to allow visual information to permeate, making the road edges detectable by a convolutional neural network.

[0104] Additionally, some feature engines 212 can preprocess the image, such as performing corrections, cropping, etc. For example, for cropping, the image 214c from the fisheye lens facing forward can be cropped vertically to remove certain elements (e.g., glare blockers) included on the windshield.

[0105] For the correction, the vehicles described herein can be examples of vehicles that can be used by millions or more end users. Due to manufacturing tolerances and / or usage variations in vehicles, image sensors in the vehicles may be positioned slightly differently at angles or otherwise (e.g., differences in roll, pitch, and / or yaw). Additionally, different models of the vehicle can perform the same vision-based machine learning model. These different models may have image sensors that are positioned and / or angled differently. The occupancy network 217 described herein can be trained, at least in part, using information aggregated from a fleet of vehicles used by end users. Therefore, differences in viewpoints of the images can be apparent due to slight differences in the angles or positions of image sensors in the vehicles included in the fleet.

[0106] Therefore, correction can be performed to address these differences. For example, a transformation (e.g., an affine transformation) can be applied to images 214a-214h or a portion of images 214a-214h to normalize the images. In this example, the transformation can be based on camera parameters associated with an image sensor (e.g., camera 170m), such as extrinsic and / or intrinsic parameters. In some embodiments, the image sensor can undergo an initial and optionally repeated calibration step. For example, when a vehicle is being driven, the camera can be calibrated to identify camera parameters that can be used in the correction process. In this example, specific markings (e.g., road lines) can be used to inform the calibration. Correction can optionally represent one or more layers of feature engine 212, in which values ​​for the transformation are learned based on training data.

[0107] Therefore, feature engine 212 can output feature maps (e.g., tensors), which are used downstream of occupancy network 217. In some embodiments, the output from feature engine 212 can be combined into a matrix or tensor. In some embodiments, the output can be provided as multiple tensors (e.g., eight (8) tensors in the example shown). In the example shown, the output is referred to as feature 216, which is input into occupancy engine 219. Although feature engine 212 and occupancy engine 219 are shown separately, in some embodiments, feature engine 212 and occupancy engine 219 can form part of the same network or model (e.g., occupancy network 217). Additionally, in some embodiments, feature engine 212 and occupancy engine 219 can be trained end-to-end.

[0108] Occupation engine 219 can use feature 214 to determine occupancy associated with an object positioned around a vehicle. For example, occupancy engine 219 can separate the real-world environment into voxels extending upwards from the ground in three-dimensional space. Each voxel can represent a portion of three-dimensional space, such as a cuboid or other three-dimensional shape. Objects or real-world features can be represented as combinations of voxels. For example, a truck can be represented as a combination of voxels that forms the volume of the truck when it is present in the real-world environment. Similarly, bollards can be represented as combinations of voxels. Additionally, curbs or sidewalks can be represented as combinations of voxels with sufficient detail to determine changes in curb elevation (e.g., changes in curb elevation due to tree root growth, previous earthquakes, etc.).

[0109] Therefore, the occupancy engine 219 determines occupancy in 3D based on input image features from the feature engine 200. The occupancy engine 219 includes a transformer engine 218, which uses query-based attention to determine 3D occupancy features. The output from the transformer engine 218 is provided to the alignment engine 220. To ensure object tracking during autonomous vehicle navigation, the alignment engine 220 queues the output from the transformer engine 218, even in cases of transient occlusion. The queued output can be used to track objects by aligning the occupancy features from the queue. For example, outputs can be pushed into the queue based on time and / or space. In this example, a time index can instruct the transformer engine 218 to store outputs based on time (e.g., pushing information at a specific frequency). A spatial index can instruct the transformer engine 218 to store outputs based on the vehicle's spatial movement. The alignment engine 220 may additionally include a video module that performs 3D convolution to track objects. In some embodiments, vehicle-associated kinematic information can be used as input to the alignment engine 220.

[0110] Output from alignment engine 220 can be provided to occupancy head 222 to determine obstacle occupancy information 224. Example output from occupancy head 222 may include indications of occupancy associated with voxels. Additionally, obstacle occupancy information 224 may include information indicating the presence of an object within various angular ranges surrounding the vehicle. Obstacle occupancy information 224 may also include the distance from the vehicle to the object within the angular range. For example, obstacle occupancy information 224 may indicate the presence of an object (e.g., a bollard) within several adjacent angular ranges behind the vehicle. Obstacle occupancy information 224 may also indicate the presence of a curb to the right of the vehicle. As will be understood, a curb may be curved, warped, etc. Therefore, the angular range can indicate the presence of a curb and the distance to the curb.

[0111] As described herein, object obstacle information 224 can be used to render a user interface that graphically indicates the presence of objects approaching the vehicle. FIG. 2D It includes a sample user interface.

[0112] Additional details relating to the occupation of network 217 are included in U.S. Provisional Patent Application No. 63 / 375,199, the entire contents of which are incorporated herein by reference and for all purposes.

[0113] Obstacle occupancy information 224 can be generated via forward propagation through occupancy network 217. In some embodiments, forward propagation can be calculated at a specific frequency (e.g., 24 Hz, 30 Hz, etc.).

[0114] Now for reference FIG. 2D A block diagram showing details of the output of the occupancy network 217 associated with parking is illustrated. In the example shown, vehicle 232 is parked in a parking lot. As described herein, vehicle 232 can perform the occupancy network 217 to determine obstacles associated with obstacles. Therefore, as vehicle 232 drives toward a parking space, vehicle 232 is generating occupancy obstacle information 224 associated with obstacles visible to the image sensors of vehicle 232.

[0115] In this way, vehicle 232 can detect at least obstacle A 234 (e.g., a bollard) and obstacle B 238 (e.g., a parking block or curb stop) near vehicle 232. As will be understood, when vehicle 232 is reversing, the rear-facing camera may lose visibility of obstacle B 238 (e.g., the obstacle may be located below the camera). However, even when obscured, the occupancy network 217 described herein adjusts the position of the object (e.g., based on the movement of vehicle 232, such as via kinematic information). In this way, vehicle 232 can maintain an understanding of the position of object 238.

[0116] exist FIG. 2DIn this context, obstacle A 234 is located to the left of vehicle 232. When vehicle 232 navigates to its final stationary position, occupancy network 217 outputs an obstacle classification vector 224a and an obstacle distance vector 224n. As described above, obstacle classification vector 224a can indicate the presence of an object located within an angular range that is optionally closer than a threshold distance from vehicle 232 (e.g., 3 meters, 4 meters, 7 meters, etc.) (e.g., binary classification). Obstacle distance vector 224n can indicate the distance to an object located within the angular range. As shown, obstacle A 234 is located within range A 234. Therefore, vector 224n indicates the presence of an object with respect to range A 234. Similarly, obstacle distance vector 224n indicates the range to obstacle A 234. In some embodiments, vectors 224a-224n can have a specific number of elements (e.g., 70 elements, 71 elements, 72 elements, etc.).

[0117] Vehicle 232 may include a display that includes a graphical representation of obstacle occupancy information 224. For example, the graphical representation (referred to as a graphical user interface (GUI) 230) may include an outline positioned around the graphical representation of vehicle 232, the color of which is adjusted based on the proximity of vehicle 232 to objects. For reference... FIG. 2D For example, the outline will have a color that reflects the angle area where the obstacle is visible, near the parking bollards and parking blocks. While specific colors can be varied and fall within the scope of this disclosure, in some embodiments, green may reflect distant or non-existent objects, and colors closer to red may indicate approaching objects. For example, colors may be selected based on a distance of 224n (e.g., a distance closer to zero may be a darker shade of red than a greater distance). In some embodiments, the outline may be fixed around the vehicle. In some embodiments, the outline may move closer to the vehicle as the vehicle gets closer to the object. The outline described herein may also be other shapes, such as squares, rectangles, polygons, etc.

[0118] The graphical representation can also represent the outline of the exterior of objects closer to the vehicle than a threshold distance (e.g., 3 meters, 4 meters, 7 meters, etc.). For example, the display may include a graphical representation of the vehicle 232 as well as a representation of objects within the threshold distance. Therefore, the outline can be positioned and colored along the exterior of the object, as described above. For example, and referring to... FIG. 3The first contour can be located along the outside of obstacle A 234, the second contour can be located along obstacle B 256, and so on. Since these objects 234, 238 will be displayed graphically, the driver will see a representation of the position of the driver's vehicle relative to the reference objects 234, 238. Additionally, the driver can use the contours to quickly determine how close the vehicle is to the represented object.

[0119] Now for reference FIG. 1B This illustrates a non-limiting example of a process 300 for updating a user interface to depict obstacle occupancy information. Process 300 is shown as a block diagram and includes details of an example bird's-eye view network. For convenience, process 300 will be described as being performed by what is referred to as an autonomous driving system (e.g., with...). FIGS. 2A-2D The autonomous driving system is implemented by a system that is the same as or similar to the autonomous driving system 170o and / or autonomous driving system 202. In some embodiments, the autonomous driving system may be included in one or more devices of the same or similar vehicle as vehicle 140 and / or implemented by one or more devices of the same or similar vehicle as vehicle 140.

[0120] At operation 302, the autonomous driving system acquires images from multiple image sensors placed around the vehicle. As described above, there may be 7, 8, 10, and / or similar image sensors (e.g., cameras) used to acquire images.

[0121] At operation 304, the autonomous driving system calculates the forward pass through the occupied network. (See reference...) FIG. 2D The description refers to the occupancy of objects in a real-world environment, as determined by the occupancy network.

[0122] At operation 306, the autonomous driving system acquires obstacle occupancy information. For example, obstacle occupancy information includes indications of voxels forming objects close to the vehicle. As described herein, this information also includes indications of objects within different angular ranges around the vehicle and their distances to the objects.

[0123] At operation 308, the autonomous driving system updates the user interface displaying obstacle occupancy information. (See reference...) FIGS. 4-5B The description describes an autonomous driving system that renders a user interface depicting obstacle occupancy information. For example, this may include contours surrounding a vehicle or an object, colored based on the vehicle's proximity to one or more objects. The color of the contours can be adjusted as the vehicle moves. For example, the contours can be positioned around the vehicle. Alternatively, one or more contours can be positioned around objects close to the vehicle. In this example, the system can render a contour surrounding the exterior of an object. Therefore, a contour can be drawn along one side of the object, where that side is closest to the vehicle.

[0124] As described above, the contour can be updated based on vehicle movement. For example, movement can cause a vehicle to move closer to or further away from nearby objects. Additionally, the contour can be updated over time by monitoring the positions of nearby objects. Therefore, if a vehicle is parked and another vehicle is moving simultaneously, the contour can be updated in real time to reflect the proximity of the other vehicle's update.

[0125] Now for reference FIG. 1A The diagram illustrates non-limiting examples of user interfaces 400 and 500, which can be accessed via a vehicle (e.g., with...). FIG. 2D The display screen of a bicycle (e.g., the same as or similar to bicycle 140) FIG. 4 The display screen (228) is the same as or similar to the display screen. For example, the display screen may be positioned so that the driver of the vehicle can see it. In some embodiments, the vehicle may autonomously navigate or perform semi-autonomous actions, such as parking, when the driver is inside or outside the vehicle. In these embodiments, a mobile application on a computing device (e.g., a phone, tablet, etc.) associated with the driver may include user interfaces 400 and 500. In this way, the driver can view a representation of the vehicle's actions.

[0126] Continue to refer to FIG. 5A User interface 400 depicts occupancy near vehicle 402 when parked. In the illustrated embodiment, vehicle 402 is parked between two vehicles 408, 410. User interface 400 depicts a representation of vehicle 402 and nearby objects, including the two vehicles 408, 410. Surrounding vehicle 402 are outlines 406 or other graphic shapes, the colors of which are adjusted to indicate proximity to vehicles 408, 410. For example, the leftmost portion may be colored red because it is closest to vehicle 410. The upper left portion may be colored orange or yellow because the left headlight is not too close to vehicle 410. In some embodiments, a nearest distance 404 to nearby objects (e.g., 40 inches) may be included.

[0127] Continue to refer to FIG. 5B User interface 500 depicts occupancy near vehicle 402 when parked. In the example shown, a graphical representation of vehicle 402 (e.g., a vehicle being driven) is included. Vehicle 504 is positioned to the right of vehicle 402 along with an outline 506 surrounding vehicle 504. For example, the outline may be shown as being pasted or wrapped around the exterior of vehicle 504. As described herein, the color of outline 506 can be selected based on the proximity of vehicle 402 to vehicle 504.

[0128] Continue to refer to FIG. 5AThis shows the updates based on the movement of vehicle 402. FIG. 6 The user interface 500. In the example shown, the vehicle 402 moves backward to perform a parallel parking maneuver. Therefore, the position of the vehicle 402 relative to the vehicle 504 is updated. The outline 506 is now included around the exterior of the vehicle 504 and along the curb 508. In this way, the driver can quickly see how close they are to the curb 508.

[0129] Now for reference FIGS. 1A-1C This illustrates a non-limiting example of a block diagram of an implementation 600 occupying network 602. Any of the components described herein, such as a vehicle (e.g., as described herein), can be used. FIG. 2C One or more components of a vehicle (identical to or similar to vehicle 140) are used to implement occupancy network 602. In some embodiments, one or more aspects or components of occupancy network 602 may be related to... FIG. 2B The occupying network 217 has one or more aspects or components that are the same as or similar to it.

[0130] Camera 170m can generate image data 206, as described herein. For example, camera 170m can generate image data based on the actions of one or more vehicles in the environment. The actions of the vehicles may include moving the vehicle along a drivable and / or walkable surface from an initial (starting) point via one or more intermediate points to a destination point. As will be understood, such actions may be performed based on one or more paths traversing the environment.

[0131] Occupancy network 602 can receive image data 206. For example, occupancy network 602 can be implemented by a computing device associated with the vehicle (e.g., by an autonomous driving system 170o and / or a vehicle computing device 171 that can implement the autonomous driving system 170o). In this example, occupancy network 602 can receive image data 206 based on image data 206 generated by one or more sensors of the vehicle.

[0132] In some embodiments, the occupancy network 602 may provide a portion of the image data 206 to one or more corresponding backbones of the sensor backbone 604. For example, the occupancy network may include a set of backbones 604a-604h configured to receive sensor data generated by a corresponding sensor and process the sensor data to generate features (e.g., values ​​corresponding to one or more features represented by the outputs of the respective backbones 604a-604h). In some embodiments, the sensor backbone 604 may be the same as or similar to components of the feature engine 212, and the features generated by the sensor backbone 604 may be... FIG. 2C Features 216 are the same or similar.

[0133] In some embodiments, features output by sensor backbone 604 can be provided to bird's-eye-view (BEV) converter 606a. For example, sensor backbone 604 can be provided to BEV converter 606a to generate an output. The output can include a set of distances relative to the vehicle and to one or more objects in the environment during the vehicle's operation in the environment. For example, BEV converter 606a can be trained based on a dataset comprising multiple image pairs and corresponding distances to points partially represented by the two images. The output can be associated with a given point in time (e.g., the current point in time, also referred to as time t=0). The output of BEV converter 606a can then be provided to downsampling system 608. In some embodiments, the output of BEV converter 606a can also be provided to fusion system 610.

[0134] In some embodiments, downsampling system 608 may receive data output from BEV converter 606a and downsample the data. For example, downsampling system 608 may include a neural network (e.g., a convolutional neural network, etc.) having one or more layers configured to downsample the output of BEV converter 606a. Downsampling system 608 may be configured to downsample the output of BEV converter 606a before providing the downsampled output to fusion system 614 described herein.

[0135] In some embodiments, features output by the sensor backbone 604 may be provided to the Birds-Eye-View (BEV) converter 606a at one or more earlier time points (times t-1, t-2, etc.) and stored and / or retrieved by the occupancy network 602 at a given time point (e.g., time t=0). For example, the sensor backbone 604 may be provided to the BEV converter 606a at times t-1, t-2, etc., so that the BEV converter 606a generates a corresponding output. The output may include a set of distances relative to the vehicle and to one or more objects in the environment at times t-1, t-2, etc., during the vehicle's operation in the environment. The output of the BEV converter 606a at one or more earlier time points may then be provided to the fusion system 610 so that the fusion system 610 combines the output at one or more earlier time points with the output of the BEV converter 606a at the current time point.

[0136] In some embodiments, the output of the fusion system 610 can be provided to the alignment system 612 (e.g., it can be used with...). FIG. 7(The system is the same as or similar to the alignment engine 220). In some embodiments, the alignment system 612 may include a machine learning model configured to acquire the output of the fusion system 610 and update the output of the fusion system 610 based on one or more changes in the vehicle's motion and / or position relative to the environment. For example, the alignment system 612 may be configured to receive the output of the fusion system 610 along with kinematic data associated with the motion of the vehicle at each corresponding time point (e.g., time t=0, t-1, t-2, etc.). The alignment system 612 may then output an updated set of distances relative to the vehicle and to one or more objects within the environment during the vehicle's operation in the environment. Thus, the alignment system 612 may output an updated set of distances relative to the vehicle and to one or more objects within the environment during the vehicle's operation in the environment, wherein the updated set of distances is refined based on the known motion of the vehicle within the environment. In some embodiments, the output of the alignment system may be downsampled by the alignment system 612 or by a separate downsampling system (not explicitly shown) as described herein with respect to downsampling system 608.

[0137] In some embodiments, the outputs of downsampling system 608 and alignment system 612 are provided to fusion system 614. For example, the output of downsampling system 608 (e.g., a set of downsampled distances relative to the vehicle and to one or more objects in the environment at time t=0 during the vehicle's operation in the environment) and the output of alignment system 612 (e.g., a set of downsampled distances relative to the vehicle and to one or more objects in the environment at times prior to time t=0 during the vehicle's operation in the environment) can be provided to fusion system 614. Fusion system 614 may include a machine learning model configured to receive the outputs of downsampling system 608 and alignment system 612 and determine an updated set of distances relative to the vehicle and to one or more objects in the environment during the vehicle's operation in the environment. The resulting updated set of distances output by fusion system 614 can then be provided to distance head 616. Distance head 616 may include a machine learning model configured to perform one or more convolutions and output a set 618 of distances between the vehicle and one or more objects in the environment.

[0138] In some embodiments, the output of the fusion system 614 can be provided to an object prediction network. For example, the output of the fusion system 614 can be provided to an object prediction network that includes a machine learning model configured to receive the output of the fusion system 614 and determine a correspondence between the output of the fusion system 614 and one or more predetermined object types. In this example, the one or more object types may include vehicles, trucks, another vehicle, pedestrians, etc. In some embodiments, the output of the object prediction network can be used to generate a GUI, as described herein. For example, the representation of an object when included in the GUI can be determined based on the output of the object prediction network that identifies predetermined object types. In this way, the GUI described herein can be generated such that the GUI provides a more accurate visual representation of an object (by a vehicle and / or relative to a vehicle) that is approaching.

[0139] Now for reference FIG. 1B This document illustrates a non-limiting example of a process 700 for generating a graphical user interface based on distance to an object relative to the robotic system. Components described herein, such as one or more components of an autonomous driving system (e.g., with...), can be used. FIGS. 2A-2D 170° and / or autonomous driving system FIG. 1A The analysis server is the same as or similar to the analysis server in the autonomous driving system 202, and any of the implementation processes 700. In some embodiments, it may be provided by one or more computing devices in one or more vehicles (e.g., with FIGS. 1A-1C The autonomous driving system is implemented using one or more computing devices that are the same as or similar to the vehicle computing devices 141a-141c. Although process 700 includes operations 702-708 as described herein, other embodiments may include additional or alternative operations, or one or more operations may be omitted.

[0140] In some implementations, in conjunction with one or more vehicles (e.g., with...) FIG. 1A When an autonomous driving system is associated with (e.g., installed in one or more of the same or similar autonomous vehicles) as vehicle 140, the autonomous driving system executes process 700. However, one or more steps of process 700 may be executed by one or more other computing devices separate from and / or included in the autonomous driving system, such as by an analytics server (e.g., with...). FIG. 1A Analysis server 110a is the same as or similar to the analysis server, and one or more computing devices of at least one vehicle (e.g., with the analysis server 110a). FIGS. 1A-1C One or more computing devices that are the same as or similar to the vehicle computing devices 141a-141c) and / or in a distributed computing system (e.g., with FIGS. 2A-2DThe process can be performed by one or more other computing devices operating in a distributed computing system (similar to or similar to the distributed computing system described in the text). For example, one or more computing devices in a vehicle can perform some or all of the steps described in process 700 locally.

[0141] At operation 702, the autonomous driving system receives data associated with the first and second images. For example, during operation of a robotic system (referred to as a vehicle) within an environment, the autonomous driving system may receive data associated with the first and second images. In this example, the vehicle may also be stationary or moving relative to the environment, and one or more objects may be stationary or moving relative to the vehicle. In some embodiments, the data associated with the first and second images may be generated by one or more sensors (e.g., cameras, radar, ultrasonic sensors, LiDAR sensors, etc.). For example, a first camera may generate the first image, and a second camera may generate the second image. In some embodiments, the data associated with the first image is generated by a first sensor, and the data associated with the second image is generated by a second sensor, wherein both the first and second sensors are positioned on the vehicle such that the fields of view of the first and second sensors at least partially overlap each other. While this disclosure has been discussed with reference to certain vehicles, it should be understood that the described vehicles are not intended to be limiting, and the systems and methods described herein can also be applied to other robotic systems not explicitly discussed.

[0142] At operation 704, the autonomous driving system determines multiple distances relative to the vehicle and to one or more objects based on the first and second images. For example, the autonomous driving system may be based on the references herein. FIG. 6 and / or FIGS. 4-5B The described occupancy network determines multiple distances. In these examples, the autonomous driving system can provide the occupancy network with data associated with a first image and a second image, causing the occupancy network to perform a forward pass through the occupancy network. The occupancy network can generate an output based on the forward pass. For example, the occupancy network can generate an output based on the forward pass, wherein the output includes data associated with indications of one or more angular ranges, distances from points along the vehicle to one or more objects, and distances to points along the one or more objects. In this way, the autonomous driving system can cause the occupancy network to output one or more distances relative to the vehicle and to one or more points along the surface of the object.

[0143] At operation 706, the autonomous driving system generates a graphical user interface (GUI) based on multiple distances. For example, the autonomous driving system may generate the GUI based on multiple distances corresponding to one or more objects relative to the vehicle. In this example, the GUI may include a representation of the vehicle and representations of one or more objects located relative to the vehicle within the environment in which the vehicle is operating. (Refer to above) FIG. 1B An example of a GUI is discussed. In the described example, the autonomous driving system can generate the GUI based on the autonomous driving system determining that the representation of the vehicle is fixed relative to the GUI (e.g., the vehicle can be placed in the center of the GUI or along a side portion of the GUI), and subsequently determining the position of objects relative to the vehicle.

[0144] In some embodiments, the autonomous driving system determines a contour associated with each of one or more objects. For example, the autonomous driving system may determine the contour based on distances corresponding to each of the one or more objects positioned relative to the vehicle, these distances representing the relative position of one or more points along the surface of each object relative to the vehicle. In some embodiments, the autonomous driving system determines the placement of the contour relative to the object associated with the contour. For example, the autonomous driving system may determine the placement of the contour relative to the object associated with the contour based on the position of each object relative to the vehicle determined by the autonomous driving system. The autonomous driving system may then match the contour with corresponding portions of the surfaces of the respective objects and determine the placement of the contour when included in a GUI. The autonomous driving system may then generate a GUI based on the contour and the placement of the contour relative to the corresponding object in the environment.

[0145] In some embodiments, the autonomous driving system can determine a profile of a contour. For example, the autonomous driving system can determine a profile of a given object based on the position of the object and the contour relative to the vehicle. In an example, the autonomous driving system can determine the profile of the contour based on multiple distances corresponding to objects that match the contour. In some embodiments, the autonomous driving system can determine one or more aspects of the profile based on multiple distances, wherein the profile is continuous and adjusts as a portion of the contour changes distance relative to a portion of the vehicle. As an example, where the distance of a point along the contour relative to at least one point along the vehicle increases continuously, the autonomous driving system can determine one or more colors or shadows that form a gradient that changes proportionally to the change in distance (e.g., where closer distances correspond to darker shadows or colors, and progressively farther distances correspond to lighter shadows or colors). In another example, where the distance between points along the contour and at least one point along the vehicle increases continuously, the autonomous driving system can determine one or more colors or shades of the contour that change proportionally with the change in distance (e.g., where closer parts of the contour correspond to darker shades or colors, and progressively farther parts of the contour correspond to lighter shades or colors).

[0146] In some embodiments, the autonomous driving system determines whether one or more of the distances relative to the vehicle and to an object meet a threshold range of distances. For example, the autonomous driving system may determine the distances relative to the vehicle and to an object, and then compare these distances to a threshold range of distances. For example, the threshold range may include a range that includes a lower limit (e.g., 5 cm, 4 cm, etc.) and / or an upper limit (e.g., 100 cm, 125 cm, etc.). In this example, the autonomous driving system may generate a GUI based on whether the distances relative to the vehicle and to the object meet the threshold range. For example, the autonomous driving system may determine that the distance meets the threshold range (e.g., meets the lower limit and / or upper limit) and generate a GUI as described herein based on multiple distances corresponding to the object. In another example, the autonomous driving system may determine that the distance does not meet the threshold range (e.g., does not meet the lower limit or upper limit) and generate a GUI as described herein by discarding the distances that do not meet the threshold range. In this way, the autonomous driving system may discard the generation of a GUI to include objects outside the threshold range.

[0147] In some embodiments, the autonomous driving system determines one or more audible tones (e.g., a ringtone) to be output based on one or more distances between the vehicle and an object. For example, as the vehicle approaches an object, the autonomous driving system may determine that the distance between the vehicle and the object relative to the vehicle is decreasing (e.g., the vehicle and / or the object relative to the vehicle are moving closer to each other). The autonomous driving system may then identify, based on the distance, the audible tone or a series of audible tones to be output by the vehicle's user interface (e.g., a speaker). For example, the autonomous driving system may determine that a single ringtone, a series of ringtones, etc., should be output by the speaker. Additionally or alternatively, the autonomous driving system may determine a pattern associated with the audible tone to be output based on one or more distances. For example, the autonomous driving system may determine a tone pattern that includes a set of tones whose frequency and / or volume increase as the vehicle and / or the object relative to the vehicle move closer to each other. In some embodiments, the autonomous driving system may then provide data associated with the audible tone or series of audible tones to the speaker or a computing device controlling the speaker to cause the speaker to output the tone or series of audible tones.

[0148] In some embodiments, the autonomous driving system determines whether one or more of the distances relative to the vehicle and to an object indicate whether the height of the object corresponding to the distance meets a height threshold range. For example, the autonomous driving system may determine the height of the object relative to the vehicle based on multiple distances corresponding to the object and then compare the height to a height threshold range. For example, the height threshold range may include a range that includes a lower limit (e.g., 5 cm, 4 cm, etc.) and / or an upper limit (e.g., 100 cm, 125 cm, etc.). In this example, the autonomous driving system may generate a GUI based on whether the height of the object meets the height threshold range. For example, the autonomous driving system may determine that the distance meets the height threshold range (e.g., meets the lower limit and / or upper limit) and generate a GUI as described herein based on multiple distances corresponding to the object. In another example, the autonomous driving system may determine that the distance does not meet the height threshold range (e.g., does not meet the lower limit or upper limit) and generate a GUI as described herein by discarding the distance that does not meet the threshold range. In this way, the autonomous driving system may discard generating a GUI to include objects outside the threshold range, such as ramps, curbs, or parking garage ceilings that the body of the vehicle cannot contact by lateral or longitudinal movement.

[0149] At operation 708, the autonomous driving system can provide data associated with the GUI to enable the GUI to be displayed. For example, the autonomous driving system can send data to a display screen, such as the vehicle's user interface (e.g., with...). ​The user interface 170a (or a similar user interface) provides data associated with the GUI. This data can be configured to cause one or more processors associated with the display to show the GUI. For example, the autonomous driving system can provide the GUI-associated data to the display when the vehicle is moving at a predetermined speed (e.g., 0 miles per hour (mph), less than 5 mph, less than 10 mph, etc.). In another example, the autonomous driving system can provide the GUI-associated data to the display during the operation of the vehicle. In this example, in response to receiving input from a user (e.g., an individual operating the vehicle) requesting the display of a GUI including its outline, the autonomous driving system can provide the GUI-associated data to the display during the operation of the vehicle. In some embodiments, the autonomous driving system can provide the GUI-associated data to the display in response to determining that one or more distances satisfy a minimum distance threshold. For example, as the vehicle approaches or is approached by one or more objects and the distance between the vehicle and the objects gradually decreases, the autonomous driving system can determine that a minimum distance threshold has been met and provide the GUI-associated data to cause the GUI to be displayed in response to the satisfaction of the minimum distance threshold.

[0150] All the processes described herein can be embodied in software code modules executed by a computing system comprising one or more computers or processors, and can be fully automated via these software code modules. The code modules can be stored on any type of non-transitory computer-readable medium or other computer storage device. Some or all of the methods can be embodied in dedicated computer hardware.

[0151] Many other variations besides those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain actions, events, or functions of any of the algorithms described herein may be performed in a different order and may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary for the practice of the algorithm). Furthermore, in some embodiments, actions or events may be performed concurrently, for example, through multithreaded processing, interrupt handling, or concurrent execution by multiple processors or processor cores, or concurrent execution on other parallel architectures, rather than sequentially. Additionally, different tasks or processes may be performed by different machines and / or computing systems that can work together.

[0152] The various illustrative logic blocks, modules, and engines described in conjunction with the embodiments disclosed herein can be implemented or executed by machines, such as processing units or processors designed to perform the functions described herein, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The processor may be a microprocessor, but alternatively, it may be a controller, microcontroller, or state machine, a combination thereof, etc. The processor may include circuitry configured to process computer-executable instructions. In another embodiment, the processor includes an FPGA or other programmable device that performs logic operations without processing computer-executable instructions. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Although this document is primarily described with respect to digital technologies, the processor may also primarily include analog components. For example, some or all of the signal processing algorithms described herein may be implemented in analog circuitry or mixed analog and digital circuitry. Computing environments can include any type of computer system, including but not limited to microprocessor-based computer systems, mainframe computers, digital signal processors, portable computing devices, device controllers, or computing engines within machinery, to name just a few.

[0153] Unless otherwise specified, conditional language such as “may,” “may,” “can,” or “is able” should be understood in the context of certain features, elements, and / or steps that are included in some embodiments but not in others. Therefore, such conditional language is not generally intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments must include logic for determining whether to include or perform such features, elements, or steps in any particular embodiment, with or without user input or prompting.

[0154] Unless otherwise expressly stated, disjunctive language such as the phrase “at least one of X, Y, or Z” is understood in conjunction with the context in which items, terms, etc., can be X, Y, Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive language is generally not intended to, and should not, imply that some embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0155] Any process description, element, or block described herein and / or depicted in the flowcharts in the accompanying drawings should be understood as potentially representing portions of modules, segments, or code, which include one or more executable instructions for implementing a specific logical function or element in the process. Depending on the functionality involved, alternative implementations are included within the scope of the embodiments described herein, within which elements or functions may be omitted, performed, or discussed in the order shown, including performing elements or functions substantially simultaneously or in reverse order, as will be understood by those skilled in the art.

[0156] Unless otherwise explicitly stated, articles such as “a” or “one” should generally be interpreted as including one or more of the described items. Therefore, phrases such as “the device is configured to” are intended to include one or more referenced devices. Such one or more referenced devices can also be collectively configured to perform the references. For example, “processors configured to perform references A, B, and C” could include a first processor configured to perform reference A, which works in conjunction with a second processor configured to perform references B and C.

[0157] This document describes some embodiments of the present disclosure in conjunction with thresholds or threshold ranges. As described herein, satisfying a threshold can refer to a value greater than a threshold, exceeding a threshold, higher than a threshold, greater than or equal to a threshold, less than a threshold, less than a threshold, lower than a threshold, less than or equal to a threshold, equal to a threshold, etc.

[0158] It should be emphasized that many variations and modifications can be made to the above embodiments, and the elements of the above embodiments should be understood as one of other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure.

Claims

1. A system comprising: at least one processor configured to: receive data associated with a first image and a second image, the data associated with the first image and the second image generated by one or more sensors of a robotic system; determine, based on the first image, the second image, and an occupancy network, a plurality of distances to one or more objects relative to the robotic system, the occupancy network configured to receive, as input, data associated with the first image and the second image and output the plurality of distances to one or more objects relative to the robotic system; generate, based on the plurality of distances to one or more objects relative to the robotic system, a graphical user interface, the graphical user interface comprising a representation of the robotic system and the one or more objects positioned relative to the robotic system; and provide, to at least one processor associated with a display screen, data associated with the graphical user interface, the data associated with the graphical user interface configured to cause the at least one processor to display the graphical user interface on the display screen. the one or more processors configured to generate the graphical user interface are configured to:

2. The system of claim 1, wherein, determine a contour of a portion of each of the one or more objects positioned relative to the robotic system; determine a placement of the contour relative to each of the one or more objects positioned relative to the robotic system; and generate the graphical user interface based on the contour and the placement of the contour relative to each object. the one or more processors configured to determine the contour are configured to: determine a contour of a portion of each of the one or more objects based on a distance in the plurality of distances corresponding to each of the one or more objects.

3. The system of claim 2, wherein, the one or more processors configured to determine the contour are configured to: determine a contour profile of a contour of a portion of each of the one or more objects positioned relative to the robotic system based on the plurality of distances corresponding to each of the one or more objects.

4. The system of claim 3, wherein, the one or more processors configured to determine the contour are configured to: determine a contour profile of a contour of a portion of each of the one or more objects positioned relative to the robotic system based on the plurality of distances corresponding to each of the one or more objects, each contour profile represented in a gradient form.

5. The system of claim 3, wherein, the one or more processors configured to determine the plurality of distances to one or more objects relative to the robotic system are configured to: determine a subset of distances of the plurality of distances that satisfy a threshold range of distances, and 6. The system of claim 3, wherein, wherein the one or more processors configured to generate the graphical user interface based on the plurality of distances to one or more objects relative to the robotic system are configured to: generate the graphical user interface based on the plurality of distances to the one or more objects that satisfy the threshold range of distances. ​ ​ 7. The system of claim 6, wherein, The one or more processors configured to generate the graphical user interface based on the plurality of distances to the one or more objects relative to the robotic system are configured to: generate the graphical user interface based on a plurality of distances to the one or more objects relative to the robotic system.

8. The system of claim 3, wherein, The one or more processors configured to determine the plurality of distances to the one or more objects relative to the robotic system are configured to: determine a height of each distance of the plurality of distances based on the first image and the second image; and determine that the height of each distance of the plurality of distances satisfies a threshold range of heights, and wherein the one or more processors configured to generate the graphical user interface based on the plurality of distances to the one or more objects relative to the robotic system are configured to: generate the graphical user interface based on a plurality of distances to the one or more objects associated with heights that satisfy the threshold range of heights.

9. The system of claim 8, wherein, The one or more processors configured to generate the graphical user interface based on a plurality of distances to the one or more objects associated with heights that satisfy the threshold range of heights are configured to: generate the graphical user interface based on a plurality of distances to the one or more objects associated with heights that satisfy the threshold range of heights.

10. A method comprising: receiving, by at least one processor, data associated with a first image and a second image, the data associated with the first image and the second image generated by one or more sensors of a robotic system; determining, by the at least one processor, a plurality of distances to one or more objects relative to the robotic system based on the first image and the second image; generating, by the at least one processor, a graphical user interface based on the plurality of distances to the one or more objects relative to the robotic system, the graphical user interface including a representation of the robotic system and the one or more objects positioned relative to the robotic system; and providing, by the at least one processor, data associated with the graphical user interface to at least one processor associated with a display screen, the data associated with the graphical user interface configured to cause the at least one processor to display the graphical user interface on the display screen.

11. The method of claim 10, wherein, Generating the graphical user interface includes: determining, by the at least one processor, a contour of a portion of each of the one or more objects positioned relative to the robotic system; determining, by the at least one processor, a placement of the contour relative to each of the one or more objects positioned relative to the robotic system; and generating, by the at least one processor, the graphical user interface based on the contour and the placement of the contour relative to each object.

12. The method of claim 11, wherein, Determining the contour includes: determining, by the at least one processor, a contour of a portion of each of the one or more objects based on a distance of the plurality of distances corresponding to each of the one or more objects.

13. The method of claim 12, wherein, Determining the contour includes: The at least one processor determines a profile profile section of a profile of a portion of each of the one or more objects positioned relative to the robotic system based on the plurality of distances corresponding to each of the one or more objects.

14. The method of claim 12, wherein, Determining the profile includes: The at least one processor determines a profile profile section of a profile of a portion of each of the one or more objects positioned relative to the robotic system based on the plurality of distances corresponding to each of the one or more objects, each profile section being represented in gradient form.

15. The method of claim 12, wherein, Determining the plurality of distances to one or more objects relative to the robotic system includes: The at least one processor determines a subset of distances of the plurality of distances that satisfy a threshold range of distances, and Wherein generating the graphical user interface based on the plurality of distances to one or more objects relative to the robotic system includes: The at least one processor generates the graphical user interface based on the plurality of distances to the one or more objects that satisfy the threshold range of distances.

16. The method of claim 15, wherein, Generating the graphical user interface based on the plurality of distances to one or more objects relative to the robotic system includes: Abandoning generating the graphical user interface based on the plurality of distances to the one or more objects that do not satisfy the threshold range of distances.

17. The method of claim 12, wherein, Determining the plurality of distances to one or more objects relative to the robotic system includes: The at least one processor determines a height of each distance of the plurality of distances based on the first image and the second image; and The at least one processor determines that the height of each distance of the plurality of distances satisfies a threshold range of heights, and Wherein generating the graphical user interface based on the plurality of distances to one or more objects relative to the robotic system includes: The at least one processor generates the graphical user interface based on the plurality of distances to the one or more objects associated with heights that satisfy the threshold range of heights.

18. The method of claim 17, wherein, Generating the graphical user interface based on the plurality of distances to one or more objects associated with heights that satisfy the threshold range of heights includes: The at least one processor abandons generating the graphical user interface based on the plurality of distances to the one or more objects associated with heights that do not satisfy the threshold range of heights.

19. A method comprising: obtaining, by at least one processor, images from a plurality of image sensors positioned about a vehicle; computing, by the at least one processor, a forward pass through an occupancy network to output, at least, information reflecting, for each angular range about the vehicle, whether an object is within a threshold distance of the vehicle in a single range and an estimated distance to the object; and presenting, by the at least one processor, a user interface via a display of the vehicle, the user interface depicting a graphical representation of the vehicle and the output information.

20. The method of claim 19, wherein, The graphical representation of the output information includes a profile positioned about the vehicle.

Images