Determining lane information

Abstract

Systems and techniques for determining lane information are described herein. For example, a method for determining lane information is provided. The method can include obtaining an image representing one or more lanes of a roadway and an object, wherein the object is adjacent to the roadway; and determining coordinates of an object-to-lane association point for at least one lane of the one or more lanes of the roadway, wherein the coordinates are associated with the object.

Description

Technical Field

[0001] This disclosure relates in general to determining lane information. For example, aspects of this disclosure include systems and techniques for determining the relationship between roadside signs and road lanes. Background Technology

[0002] An object detector can detect objects in an image. For each detected object, the object detector can generate a bounding box consisting of image coordinates associated with the corresponding detected object. For example, the bounding box can define a portion of the image representing the object. For example, the bounding box can define the pixels representing the object in the image.

[0003] A lane detector can detect lanes in an image of a road. The lane detector can generate lane boundaries, which can be or may include image coordinates associated with lane lines. For example, in the physical world, lane lines can represent lanes of a road. The lane detector can define lane boundaries that define the pixels in the image representing the lane lines. Lane boundaries can be used to define portions of the image representing various lanes of the road. For example, the lane detector can receive an image and identify a first portion of the image representing a first lane (e.g., a driving lane) (e.g., between two lane boundaries), a second portion of the image representing a second lane (e.g., the lane to the left of a driving lane) (e.g., between two lane boundaries), and a third portion of the image representing a third lane (e.g., the lane to the right of a driving lane) (e.g., between two lane boundaries).

[0004] Using bounding boxes (e.g., from an object detector) and lane boundaries (e.g., from a lane detector), it is possible to determine which lane a vehicle is in. For example, if a vehicle is traveling on a road that includes several lanes and captures images of other vehicles on the road, the vehicle's computing system can determine which lane the other vehicles are in (e.g., by comparing the bounding boxes of the other vehicles with the lane boundaries). Summary of the Invention

[0005] The following is a simplified summary of the invention relating to one or more aspects disclosed herein. Therefore, this summary should not be considered an exhaustive overview relating to all conceived aspects, nor should it be considered to identify key or decisive elements relating to all conceived aspects or to depict the scope associated with any particular aspect. Accordingly, the following outline presents certain concepts in a simplified form relating to one or more aspects of the mechanisms disclosed herein, preceding the detailed description that follows.

[0006] Systems and techniques for determining lane information are described. Based on at least one example, a method for determining lane information is provided. The method includes: obtaining an image representing one or more lanes of a road and an object adjacent to the road; and determining coordinates from the object to a lane-associated point in at least one of the one or more lanes of the road, wherein the coordinates are associated with the object.

[0007] In another example, an apparatus for determining lane information is provided, the apparatus comprising: at least one memory; and at least one processor (e.g., configured in a circuit) coupled to the at least one memory. The at least one processor is configured to: acquire an image representing one or more lanes of a road and an object, wherein the object is adjacent to the road; and determine coordinates of the object to a lane-associated point in at least one of the one or more lanes of the road, wherein the coordinates are associated with the object.

[0008] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: obtain an image representing one or more lanes of a road and an object, wherein the object is adjacent to the road; and determine coordinates of the object to a lane-associated point in at least one of the one or more lanes of the road, wherein the coordinates are associated with the object.

[0009] In another example, an apparatus for determining lane information is provided. The apparatus includes: components for acquiring images representing one or more lanes of a road and an object adjacent to the road; and components for determining coordinates of the object to a lane-associated point in at least one of the one or more lanes of the road, wherein the coordinates are associated with the object.

[0010] In some aspects, one or more of the devices described herein are, may be part of, or may include: extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), vehicles (or computing devices, systems, or components of vehicles), mobile devices (e.g., mobile phones or so-called "smartphones," tablet computers, or other types of mobile devices), smart or connected devices (e.g., Internet of Things (IoT) devices), wearable devices, personal computers, laptop computers, video servers, televisions (e.g., connected TVs), robotic devices or systems, or other devices. In some aspects, each device may include one image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, each device may include one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, each device may include one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, each device may include one or more sensors. In some cases, the one or more sensors may be used to determine the location of the device, the state of the device (e.g., tracking state, operating state, temperature, humidity level, and / or another state), and / or for other purposes.

[0011] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim.

[0012] The foregoing and other features and aspects will become more apparent from the following description, claims and accompanying drawings. Attached Figure Description

[0013] The following description, with reference to the accompanying drawings, details exemplary examples of this application:

[0014] Figure 1 This is a block diagram illustrating an example system for determining lane information according to various aspects of this disclosure;

[0015] Figure 2 Example images of three lanes of a road and two objects adjacent to the road are provided to illustrate concepts of various aspects of this disclosure;

[0016] Figure 3 An example bird's-eye view representation including three lanes of a road and two objects adjacent to the road is provided to illustrate the concepts of various aspects according to this disclosure.

[0017] Figure 4Example images of three lanes of a road and five objects adjacent to the road are provided to illustrate concepts of various aspects of this disclosure.

[0018] Figure 5 An example bird's-eye view representation of the three lanes of the road and five objects adjacent to the road is provided to illustrate the concepts of various aspects of this disclosure;

[0019] Figure 6 Example images including two lanes of a road and an object adjacent to the road are provided to illustrate concepts of various aspects of this disclosure;

[0020] Figure 7 This is a block diagram illustrating another example system for determining lane information according to various aspects of this disclosure;

[0021] Figure 8 This is a block diagram illustrating yet another example system for determining lane information according to various aspects of this disclosure.

[0022] Figure 9 This is a block diagram illustrating yet another example system for determining lane information according to various aspects of this disclosure.

[0023] Figure 10 This is a flowchart illustrating another example process for determining lane information according to various aspects of this disclosure;

[0024] Figure 11 This is a block diagram illustrating examples of deep learning neural networks that can be used to implement a perception module and / or one or more verification modules, based on some aspects of the disclosed techniques.

[0025] Figure 12 This is a block diagram illustrating an example computing device architecture that can implement the various technologies described herein. Detailed Implementation

[0026] Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently, and some may be applied in combination, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation in order to provide a thorough understanding of the various aspects of this application. However, it will be apparent that various aspects may be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0027] The following description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of exemplary aspects will provide those skilled in the art with descriptions that can be used to implement the exemplary aspects. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0028] The terms “exemplary” and / or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and / or “example” is not necessarily to be construed as superior to or better than other aspects. Similarly, the term “aspects of this disclosure” does not require that all aspects of this disclosure include the features, advantages, or modes of operation discussed.

[0029] As described above, a vehicle's lane can be determined using bounding boxes (e.g., from an object detector) and lane boundaries (e.g., from a lane detector). However, such techniques may not be able to associate objects adjacent to the road with lanes. For example, signs located beside or above the road can provide information for driving on the road (e.g., instructions, restrictions, and / or navigation information). However, the bounding boxes of such signs (e.g., determined by an object detector) may not be between lane boundaries (e.g., determined by a lane detector). For example, the bounding box of a sign beside the road may be outside all lane boundaries. Similarly, the bounding box of a sign above the road but not aligned with a lane may not be between lane boundaries. Therefore, techniques used to associate vehicles in a lane with lanes may not be able to associate signs with lanes. Furthermore, in some cases, such signs may provide information associated with some lanes of the road but not others. For example, on a multi-lane highway, a sign above the highway may indicate that certain lanes are tied to certain destinations (e.g., at an upcoming lane merging point). Therefore, it may be important to be able to associate objects (e.g., signs) that are adjacent to lanes (e.g., located beside or above the road) with the lanes to which those objects belong.

[0030] This document describes systems, apparatus, methods (also referred to as processes), and computer-readable media (collectively, "systems and techniques") for determining lane information. For example, systems and techniques may identify objects (e.g., signs) in an image and determine which lanes of a road the objects belong to. For example, systems and techniques may acquire images representing a road (e.g., a multi-lane road) and objects adjacent to the road. Objects may be signs located on the side of or above the road. Systems and techniques may determine coordinates from the object to a lane-associated point for at least one lane of the road. In some aspects, coordinates may be or may include image coordinates (e.g., indicating pixel locations). In some aspects, coordinates may be or may include three-dimensional coordinates. Three-dimensional coordinates may be relative to the camera that captured the image. Alternatively, three-dimensional coordinates may be relative to a reference coordinate system (e.g., latitude and longitude). Coordinates may be associated with an object. For example, for a stop sign on the side of a road, systems and techniques may determine an object to a lane-associated point indicating the lane edge of the road to which the stop sign belongs.

[0031] Systems and techniques can use a trained machine learning model (e.g., a trained neural network) to determine coordinates. For example, a machine learning model can be trained on a training data corpus (e.g., via a backpropagation training process) comprising images of roads and signs, annotated with coordinates representing lane edges associated with the signs (e.g., image coordinates or 3D coordinates). More specifically, the machine learning model can be provided with images of roads and signs. The machine learning model can predict the coordinates of lane edges associated with the signs. The predicted coordinates can be compared with coordinates in the image annotations, and the difference (e.g., error) between the predicted and annotated coordinates can be determined. Parameters (e.g., weights) of the machine learning model can be adjusted to minimize error in future iterations, and the process can be repeated multiple times using various images and annotations from the training data corpus. Once trained, the machine learning model can be provided with images (e.g., not part of the training data corpus), and the machine learning model can generate coordinates based on the images.

[0032] In some aspects, systems and technologies (or automated or semi-automated driving systems using these systems and technologies) can use coordinates to control a vehicle. For example, in some aspects, systems and technologies can model coordinates as points in a three-dimensional (or two-dimensional) map of the vehicle's environment and control the vehicle based on lane-related information provided by signs. In some aspects, systems and technologies can use coordinates to obtain information (e.g., the association between signs and lanes). Such information can be provided to the driver or stored (e.g., in a map). For example, coordinates can be used to determine whether a speed limit is relevant to the lane in which the vehicle is traveling. In non-automated vehicles, this information (the current or upcoming speed limit) can be provided to the driver as information, and / or a warning can be issued if the vehicle is traveling too fast. Additionally or alternatively, coordinates can be used to generate warnings, for example, for stop signs, pedestrian crossings, etc. For example, coordinates can be used to determine whether a sign (e.g., a stop sign or pedestrian crossing sign) is relevant to the vehicle.

[0033] Additionally or alternatively, in some aspects, the system and techniques may track coordinates across multiple images, for example, to optimize the position of coordinates by updating coordinates based on tracking. Additionally or alternatively, the system and techniques may track points based on coordinates over time (e.g., points on a map used by an automated or semi-automated driving system to control a vehicle) (e.g., based on several images captured over time), for example, to optimize the position of points in a map by updating points based on tracking.

[0034] For automated / semi-automated driving systems (e.g., in autonomous vehicles), following issued instructions and restrictions (e.g., speed limits and stop signs) is important. Additionally, interpreting navigational instructions from signs may be important for automated vehicles. Signs may include instructions that are not reflected on a map (e.g., detours in construction zones or temporary speed limits). Therefore, the ability to deduce information from signs may be important for automated / semi-automated driving systems. Furthermore, correctly associating objects (e.g., signs) with the lanes associated with those objects may be important for automated / semi-automated driving systems. These capabilities may become even more important for higher levels of autonomy (such as Level 3 and above). For example, Level 0 autonomy requires full driver control because the vehicle does not have an automated driving system, and Level 1 autonomy involves basic assistance features such as cruise control, in which case the driver has full control of the vehicle. Level 2 autonomy refers to semi-automated driving, where the vehicle can perform functions such as driving in a straight path, staying in a specific lane, controlling distance to other vehicles in front of the vehicle, or other functions. Levels 3, 4, and 5 autonomy include even greater levels of autonomy. For example, Level 3 autonomy refers to an onboard automated driving system that can take over all driving functions in certain situations, with the driver remaining ready to take over at any time (if needed). Level 4 autonomy refers to a fully automated driving experience without user assistance, even in complex driving situations (e.g., on highways and in busy city traffic). In Level 4 autonomy, the person remains in the driver's seat behind the steering wheel. Vehicles operating at Level 4 autonomy can communicate with other vehicles and inform them of upcoming maneuvers (e.g., the vehicle is changing lanes, turning, stopping, etc.). Level 5 autonomy vehicles are fully automated driverless vehicles that operate automatically in all conditions. No human operator is required to take any action. Therefore, automated / semi-automated driving systems are examples of systems and technologies that can be adopted. Additionally, systems and technologies can be used in non-automated (e.g., human-controlled) vehicles. For example, systems and technologies can provide the driver with information about signs and lanes. For example, systems and technologies can present the driver with information related to signs based on signs belonging to the lane the driver is driving in.

[0035] Various aspects of this application will be described below with reference to the accompanying drawings.

[0036] Figure 1This is a block diagram illustrating an example system 100 for determining lane information according to various aspects of this disclosure. System 100 includes a machine learning model 102 that receives an image 104 as input and generates an object-to-lane association point 106 as output. System 100 may be included in an automated or semi-automated driving system. The object-to-lane association point 106 may be used by the automated or semi-automated driving system to control the vehicle (e.g., based on instructions, restrictions, and / or navigation information provided by objects such as signs in the image 104). Additionally or alternatively, the object-to-lane association point 106 may be used to provide information to the driver.

[0037] Image 104 may be an image of a road and one or more objects adjacent to the road. The road may include multiple lanes. Multiple lanes may include multiple lanes for travel in the same direction (e.g., a multi-lane highway) or multiple intersecting lanes. One or more objects may include objects located beside the road and / or above the road. Image 104 may be captured by a camera of a vehicle.

[0038] The object-to-lane association point 106 may be or primarily comprise the coordinates of the lane edge. In some cases, the object-to-lane association point 106 may be or may comprise image coordinates, such as pixel coordinates describing the location where the lane edge appears in image 104. In other cases, the object-to-lane association point 106 may be or may comprise three-dimensional coordinates describing the lane edge in three dimensions. The three-dimensional coordinates may be relative to the camera that captured image 104 (e.g., pitch, yaw, and distance, or meters in three dimensions, such as x, y, and z relative to the camera). Additionally or alternatively, the three-dimensional coordinates may be relative to a reference coordinate system (e.g., latitude, longitude, and altitude). The object-to-lane association point 106 may be associated with an object in image 104. For example, the object-to-lane association point 106 may comprise two or more coordinates for each object (e.g., a sign) in image 104. For example, image 104 may comprise a representation of a sign located next to a lane on a multi-lane road. The object-to-lane association point 106 may be coordinates corresponding to the edge of the lane to which the sign belongs.

[0039] Machine learning model 102 may be or may include a trained neural network (e.g., a transformer). Machine learning model 102 may be trained (e.g., via a backpropagation training process) to generate coordinates (e.g., image coordinates or 3D coordinates) of lane edges associated with objects based on images. For example, prior to deployment in system 100, machine learning model 102 may be trained via an iterative training process involving providing machine learning model 102 with multiple images of roads and signs from a training data corpus. During the training process, machine learning model 102 may predict the coordinates of lane edges associated with objects in the images. The predicted coordinates may be compared with the coordinates of annotations to the images (the annotations may be part of the training data corpus), and the error between the predicted coordinates and the annotation coordinates may be determined. Parameters (e.g., weights) of machine learning model 102 may be adjusted such that in future iterations of the iterative training process, machine learning model 102 can more accurately determine the coordinates. Once trained, machine learning model 102 may be deployed in system 100 and may determine object-to-lane association points 106 based on image 104.

[0040] Figure 2 Example image 200, including three lanes of a road and two objects adjacent to the road, illustrates concepts according to various aspects of this disclosure. Image 200 includes a representation of a road (e.g., pixels representing the road) comprising lanes 202, 204, and 206. In image 200, lanes 202, 204, and 206 are all intended for travel in the same direction (e.g., entering the plane of image 200). Image 200 may be... Figure 1 Example of image 104.

[0041] Image 200 includes representations of sign 210 (e.g., pixels representing a sign) and sign 220 (e.g., pixels representing a sign). Sign 210 and sign 220 are examples of objects. Sign 210 and sign 220 are adjacent to a road. For example, sign 210 is located next to lane 206, and sign 220 is located above lane 206. According to the example in image 200, both sign 210 and sign 220 belong to lane 206, but not to lane 202 or lane 204. For example, sign 210 may indicate that the speed limit is 60 kilometers per hour (km / h) when driving in lane 206, and sign 220 may indicate navigation information associated with sign 220 (e.g., sign 220 may indicate that lane 206 is heading towards Trento).

[0042] Lines 214 and 224 are also illustrated in image 200. Lines 214 and 224 can be superimposed on an image captured by a camera. Lines 214 and 224 can be constructed using the systems and techniques of this disclosure (e.g., by...). Figure 1The system 100) determines this. Line 214 may be defined by object-to-lane association point 212, and line 224 may be defined by object-to-lane association point 222. Therefore, the system and technology can determine object-to-lane association point 212 and object-to-lane association point 222 (e.g., additionally or alternatively determine line 214 and line 224). Object-to-lane association point 212 and object-to-lane association point 222 may be... Figure 1 An example of object-to-lane association point 106. Object-to-lane association points 212 and 222 can be, for example, derived from... Figure 1 The machine learning model 102 was determined.

[0043] Systems and technologies (e.g.) Figure 1 System 100 can determine line 214 (or object-to-lane association point 212) based on and associated with sign 210. For example, the system and technology can determine the coordinates of an object to lane association point 212 in lane 206. The system and technology can determine the coordinates of an object to lane association point 212 based on sign 210. Furthermore, the system and technology can associate the coordinates of the object to lane association point 212 with sign 210. For example, the system and technology can detect sign 210 (or receive indications (such as bounding boxes) for sign 210). The system and technology can determine the coordinates of an object to lane association point 212 and associate the coordinates of the object to lane association point 212 with sign 210. By associating the object to lane association point 212 with sign 210, the system and technology can determine or indicate that sign 210 belongs to lane 206. For example, the system and technology can determine that sign 210 provides instructions, restrictions, or navigation information relative to lane 206.

[0044] Line 214 may be at the road level in image 200. For example, although sign 210 is located at the depth of sign 210 in image 200, above the road level, and / or although sign 210 is above the horizon in image 200, line 214 may be located at the depth of sign 210 in image 200, at the road level. Furthermore, line 214 may be laterally offset relative to sign 210. For example, line 214 may be located to the left of sign 210 in image 200. Additionally, line 214 may be perpendicular to the direction of travel of lane 206.

[0045] Similarly, systems and technologies (e.g., Figure 1 The system 100 can determine line 224 (or object-to-lane association point 222) based on sign 220 and associated with that sign. Line 224 may be at the road level in image 200. For example, although sign 220 is located above the road in image 200, line 224 may be located at the depth of sign 220 in image 200, at the road level. Furthermore, line 224 may be perpendicular to the direction of travel of lane 206.

[0046] Object to lane association point (e.g., Figure 1 The object-to-lane association points 106, 212, and / or 222 can accurately represent lane edges based on real-world conditions. These object-to-lane association points can represent different ranges of traffic conditions and / or lane layouts. The task of generating these object-to-lane association points can be learned by a machine learning model, enabling the model to repeat the task on new images.

[0047] In some aspects, systems and technologies (e.g.) Figure 1 System 100 can track coordinates (e.g., object-to-lane association point 212 and / or object-to-lane association point 222 of image 200) across multiple images and / or in three dimensions (e.g., in a three-dimensional model of the system and technology's environment). For example, a camera capturing image 200 can capture additional images (e.g., multiple images per second). The system and technology can generate object-to-lane association point coordinates (image coordinates and / or three-dimensional coordinates) based on each image in the images, and track the position of the coordinates across multiple images and / or across time in three dimensions. By tracking the coordinates, the system and technology can update the coordinates (e.g., using Kalman filtering techniques), and thereby optimize the coordinates.

[0048] Figure 3 An example bird's-eye view representation 300, including three lanes of a road and two objects adjacent to the road, is provided to illustrate concepts of various aspects according to this disclosure. The bird's-eye view representation 300 includes a representation of the road (e.g., an example two-dimensional map representing the road). Autonomous or semi-autonomous driving systems can use maps such as those illustrated by the bird's-eye view representation 300 to control autonomous (or semi-autonomous) vehicles. For example, an autonomous or semi-autonomous driving system can make decisions regarding lane changes, acceleration, deceleration, and / or turning based on maps such as those represented by the bird's-eye view representation 300. For descriptive purposes, the bird's-eye view representation 300 is used in conjunction with the map illustrated by the bird's-eye view representation 300. Figure 2 The image 200 captures the scene corresponding to that scene.

[0049] The bird's-eye view of road 300 includes lanes 202, 204, and 206. Furthermore, the bird's-eye view of road 300 includes signs 210 and 220. Systems and technologies (e.g., Figure 1System 100 may generate lines 214 and 224 (and / or object-to-lane association points 212 and 222) based on an image (e.g., image 200). The system and techniques may include representations of lines 214 and 224 (and / or object-to-lane association points 212 and 222) in a map (e.g., as represented by bird's-eye view representation 300). In other words, in addition to generating lines 214 and 224 (and / or object-to-lane association points 212 and 222) in an image plane (e.g., as image coordinates), the system and techniques may also model lines 214 and 224 (and / or object-to-lane association points 212 and 222) in a two-dimensional and / or three-dimensional map, which may be used by an automated or semi-automated driving system to control the vehicle and / or provide information to the driver. For example, the system and techniques can project lines 214 and 224 (and / or object-to-lane association point 212 and object-to-lane association point 222) from the image plane into a three-dimensional model of the vehicle's environment that captures the image 200, such as as represented by the bird's-eye view representation 300.

[0050] In some aspects, systems and technologies (e.g.) Figure 1 System 100 can (e.g., based on receiving multiple corresponding images to determine multiple coordinates) track object-to-lane association points 212 and 222 in a 3D model over time. For example, a camera capturing image 200 can capture additional images (e.g., multiple images per second). The system and techniques can generate object-to-lane association point coordinates (e.g., 3D coordinates) based on each image and model the coordinates as points in the 3D model. The system and techniques can track the object-to-lane association point coordinates over time. By tracking the object-to-lane association points, the system and techniques can update the object-to-lane association points (e.g., using Kalman filtering techniques) and thereby optimize the object-to-lane association point position.

[0051] Figure 4 Example image 400, including three lanes of a road and five objects adjacent to the road, illustrates concepts of various aspects according to this disclosure. Image 400 includes a representation of a road (e.g., pixels representing the road) comprising lanes 402, 404, and 406. In the lower portion of image 400, lanes 402, 404, and 406 are all intended to travel in the same direction (e.g., into the plane of image 400), but further up in image 400, lanes 402 and 404 branch off from lane 406 and travel in a different direction than lane 406. Image 400 may be... Figure 1 Example of image 104.

[0052] Image 400 includes representations of the following: a sign 410 located above lane 402, a sign 420 located above lane 404, a sign 430 located beside lanes 404 and 406, a sign 440 located beside lane 406, and a sign 450 located beside lane 406. Systems and technologies (e.g., Figure 1 System 100 can determine the object-to-lane association point 412 and / or line 414 based on sign 410, the object-to-lane association point 422 and / or line 424 based on sign 420, the object-to-lane association point 432 and / or line 434 based on sign 430, the object-to-lane association point 442 and / or line 444 based on sign 440, and the object-to-lane association point 452 and / or line 454 based on sign 450. The object-to-lane association point 412, object-to-lane association point 422, object-to-lane association point 432, object-to-lane association point 442, and object-to-lane association point 452 can be... Figure 1 An example of an object associated with lane connection point 106.

[0053] Systems and technologies (e.g.) Figure 1 The system 100 can receive images 400 as input and can use a trained machine learning model (e.g., Figure 1 The machine learning model 102 determines object-to-lane association points 412, 422, 432, 442, and / or 452. In some cases, the system and technology may detect signs 410, 420, 430, 440, and / or 450, for example, the system and technology may (e.g., using an object detector) determine bounding boxes indicating pixels representing each of signs 410, 420, 430, 440, and / or 450. Additionally or alternatively, the system and technology may receive indications of bounding boxes from another source (e.g., from an object detector outside the system and technology). In other cases, the system and technology may not explicitly use bounding boxes, but may determine object-to-lane association points 412, 422, 432, 442, and / or 452 based on image 400 without additional input.

[0054] Line 414 may (e.g., through systems and technologies, such as) Figure 1System 100 is determined to be located below sign 410 and (based on the viewpoint of image 400) laterally offset relative to sign 410. Furthermore, line 414 may be determined to be associated with lane 402. Sign 410 provides navigation information related to lane 402. Line 424 may (e.g., through system and technology) be determined to be located below sign 420 and associated with lane 404. Sign 420 provides navigation information related to lane 404. Line 434 may (e.g., through system and technology) be determined to be located at the depth of sign 430, at the road level. Line 434 may be determined to be associated with all lanes 402, 404, and 406. For example, system and technology may determine that the information provided by sign 430 is related to all lanes 402, 404, and 406. Sign 430 may indicate the branching between lane 402 and lanes 404 and 406. Line 444 may (e.g., through systems and technologies) be determined to be laterally offset relative to sign 440 and associated with lane 406. Sign 440 may provide navigation information related to lane 406. Line 454 may (e.g., through systems and technologies) be determined to be laterally offset relative to sign 450 and associated with lane 406. Sign 450 may indicate a speed limit related to lane 406.

[0055] Figure 5 An example bird's-eye view representation 500, including three lanes of a road and five objects adjacent to the road, is provided to illustrate concepts of various aspects according to this disclosure. For descriptive purposes, bird's-eye view representation 500 is associated with... Figure 4 The image 400 corresponds to the scene captured.

[0056] The bird's-eye view of road 500 shows lanes 402, 404, and 406. Furthermore, the bird's-eye view of road 500 includes signs 410, 420, 430, 440, and 450. Systems and technologies (e.g., Figure 1 System 100 may generate lines 414, 424, 434, 444, and 454 based on an image (e.g., image 400). The system and techniques may include representations of lines 414, 424, 434, 444, and 454 in a map (e.g., as represented by bird's-eye view representation 500). For example, the system and techniques may project lines 414, 424, 434, 444, and 454 from an image plane onto a three-dimensional model of the environment of the vehicle that captured image 400, such as as represented by bird's-eye view representation 500.

[0057] Figure 6An example image 600, including two lanes of a road and an object adjacent to the road, is provided to illustrate concepts according to various aspects of this disclosure. Image 600 includes a representation of the road (e.g., pixels representing the road), which includes lanes 602 and 604. In image 600, both lanes 602 and 604 are intended for travel in the same direction (e.g., into the plane of image 600). Image 600 may be... Figure 1 Example of image 104.

[0058] Image 600 includes a representation of sign 610 located next to lane 602. Systems and technologies (e.g., Figure 1 System 100 can determine the object-to-lane association point 612 and / or line 614 based on sign 610. Even though a portion of lane 602 is obscured by vehicle 622, the system and technology can still determine the object-to-lane association point 612 and / or line 614. For example, even when a portion of the lane boundary and / or a portion of the lane is obscured in image 600, the system and technology are able to determine the object-to-lane association point 612 and / or line 614 based on sign 610. The object-to-lane association point 612 can be... Figure 1 An example of an object associated with lane connection point 106.

[0059] Figure 7 This is a block diagram illustrating an example system 700 for determining lane information according to various aspects of this disclosure. System 700 includes a machine learning model 702 that receives an image 704 as input and generates an object-to-lane association point 706 as output. System 700 may be included in an automated or semi-automated driving system. The object-to-lane association point 706 may be used by the automated or semi-automated driving system to control the vehicle (e.g., based on instructions, restrictions, and / or navigation information provided by objects (such as signs) in the image 704). Additionally or alternatively, the object-to-lane association point 706 may be used to provide information to the driver.

[0060] Image 704 may be an image of a road and one or more objects adjacent to the road. Image 704 may be identical to or substantially similar to... Figure 1 Image 104. Figure 2 Image 200 Figure 4 Image 400 and / or Figure 6Image 600 can be an example of image 704. The object-to-lane association point 706 can be the coordinates of the edge of the lane. The coordinates can be image coordinates describing the edge of the lane when it appears in image 704. Additionally or alternatively, the coordinates can be three-dimensional coordinates of the lane edge (e.g., relative to the camera that captured image 704 or in a reference coordinate system). The object-to-lane association point 706 can be associated with an object in image 704. The object-to-lane association point 706 can be the same as or substantially similar to an object in image 704. Figure 1 The object is associated with lane connection point 106. Figure 2 Object-to-lane association point 212 and object-to-lane association point 222 Figure 4 Object to lane association point 412, object to lane association point 422, object to lane association point 432, object to lane association point 442 and object to lane association point 452, and / or Figure 6 The object-to-lane association point 612 can be an example of the object-to-lane association point 706.

[0061] Machine learning model 702 may be or may include a trained neural network (e.g., a transformer). Machine learning model 702 may be trained (e.g., via a backpropagation training process) to generate object-to-lane association points (e.g., image coordinates or 3D coordinates) based on an image. Machine learning model 702 may be identical to or substantially similar to... Figure 1 The machine learning model 102, and / or can perform the same or substantially the same operations as it.

[0062] Additionally, in some cases, machine learning model 702 can generate lane associations 708. For example, in addition to generating lane edges 706, machine learning model 702 can be trained to generate lane associations 708. Machine learning model 102 can be trained (e.g., via a backpropagation training process) to generate object-related lane associations based on images. For example, prior to deployment in system 700, machine learning model 702 can be trained via an iterative training process involving providing machine learning model 702 with multiple images of roads and signs from a training data corpus. During the training process, machine learning model 702 can predict lane associations related to objects in the images. The predicted lane associations can be compared with lane associations annotated to the images (the annotations may be part of the training data corpus), and the error between the predicted lane associations and the annotated lane associations can be determined. The parameters (e.g., weights) of machine learning model 702 can be adjusted so that in future iterations of the iterative training process, machine learning model 702 can more accurately determine lane associations. Once trained, the machine learning model 702 can be deployed in the system 700 and lane associations 708 can be determined based on the image 704.

[0063] Lane association 708 may be or may include an association between an object (e.g., a sign) in image 704 and a lane of the road. Furthermore, lane association 708 may include an identifier relative to the lane from which image 704 was captured. For example, image 704 may be captured by a camera of a vehicle traveling in the lane. This lane may be referred to as a "vehicle" lane. Other lanes in the road may be referenced relative to the vehicle lane. For example, a lane immediately to the left of the vehicle lane may be referred to as "One Lane Left". A lane two lanes to the left of the vehicle lane may be referred to as "Two Lanes Left", etc. A similar lane to the right of the vehicle lane may be referred to as "One Lane Right", etc. For example, lane association 708 may include an association between an object (e.g., a sign) and an identifier of the lane to which the object belongs.

[0064] Lane association 708 can include not only the association between an object (e.g., a sign) and coordinates (as included in object-to-lane association point 706), but also the association between the object and a lane relative to the vehicle's lane. In some cases, such lane associations may be useful for automated or semi-automated driving systems because they may be more directly related to driving decisions. For example, an automated or semi-automated driving system may make a decision about changing lanes based on lane association 708. However, in some cases, the actual road may not be neatly classified into categories relative to the vehicle's lane. For example, in situations where lanes branch and the branched lanes branch again, it may be difficult to classify some of the branched lanes captured from the image relative to the vehicle's lane. In such cases, lane association 708 may be complex and / or not particularly useful. However, even in such cases, object-to-lane association point 706 remains relevant because object-to-lane association point 706 can associate an object (e.g., a sign) with a lane when the lane appears in the image (e.g., as image coordinates) and / or when the lane is in a 3D model of the environment, without relying on the ability to associate lanes relative to the vehicle's lane. The coordinates can be fixed in time and space, and therefore can be tracked regardless of lane changes. Furthermore, the coordinates can be tracked over time (e.g., image coordinates and / or 3D coordinates indicating lane edges) and / or modeled. Lane association 708 is optional in system 700. The optional nature of lane association 708 in system 700 is illustrated by using dashed lines to represent lane association 708.

[0065] Additionally, in some cases, machine learning model 702 may generate bounding boxes 710 indicating the image coordinates of objects (e.g., signs) in image 704. For example, in addition to generating object-to-lane association points 706 and / or lane associations 708, machine learning model 702 may be trained to generate bounding boxes 710. Machine learning model 702 may be trained (e.g., via a backpropagation training process) to generate bounding boxes of objects in an image based on the image. For example, prior to deployment in system 700, machine learning model 702 may be trained via an iterative training process involving providing machine learning model 702 with multiple images of roads and signs from a training data corpus. During the training process, machine learning model 702 may predict bounding boxes of objects in the images. The predicted bounding boxes may be compared with the bounding boxes of the image's annotations (which may be part of the training data corpus), and the error between the predicted bounding boxes and the annotated bounding boxes may be determined. The parameters (e.g., weights) of machine learning model 702 may be adjusted such that in future iterations of the iterative training process, machine learning model 702 can more accurately determine the image coordinates. Once trained, the machine learning model 702 can be deployed in the system 700 and can determine the bounding box 710 based on the image 704.

[0066] The bounding box 710 may include image coordinates that define the shape describing the position of an object (e.g., a sign) in the image 704. For example, Figure 6 The bounding box 616 describes the location of the mark 610 in the image 600. The bounding box 710 is optional in the system 700. The optional nature of the bounding box 710 in the system 700 is indicated by using a dashed line to indicate the bounding box 710.

[0067] Figure 8 This is a block diagram illustrating an example system 800 for determining lane information according to various aspects of this disclosure. System 800 includes a machine learning model 802 that receives an image 804 as input and generates an object-to-lane association point 806 as output. System 800 further includes a lane associate 820 that associates the object-to-lane association point 806 with a lane boundary 814 to generate a lane association 808. System 800 may be included in an automated or semi-automated driving system. The object-to-lane association point 806 and / or lane association 808 may be used by the automated or semi-automated driving system to control the vehicle (e.g., based on instructions, restrictions, and / or navigation information provided by objects such as signs in image 804). Additionally or alternatively, the object-to-lane association point 806 and / or lane association 808 may be used to provide information to the driver.

[0068] Image 804 may be an image of a road and one or more objects adjacent to the road. Image 804 may be identical to or substantially similar to... Figure 1 Image 104 and / or Figure 7 Image 704. Figure 2 Image 200 Figure 4 Image 400 and / or Figure 6 Image 600 can be an example of image 804. The object-to-lane association point 806 can be the coordinates of the edge of the lane. The coordinates can be image coordinates describing the edge of the lane when it appears in image 804. Additionally or alternatively, the coordinates can be three-dimensional coordinates of the lane edge (e.g., relative to the camera that captured image 804 or in a reference coordinate system). The object-to-lane association point 806 can be associated with an object in image 804. The object-to-lane association point 806 can be the same as or substantially similar to an object in image 804. Figure 1 Object to lane association point 106 and / or Figure 7 The object is associated with lane connection point 706. Figure 2 Object-to-lane association point 212 and object-to-lane association point 222 Figure 4 Object to lane association point 412, object to lane association point 422, object to lane association point 432, object to lane association point 442 and object to lane association point 452, and / or Figure 6 The object-to-lane association point 612 can be an example of the object-to-lane association point 806.

[0069] Machine learning model 802 may be or may include a trained neural network (e.g., a transformer). Machine learning model 802 may be trained (e.g., via a backpropagation training process) to generate object-to-lane-associated points (e.g., image coordinates or 3D coordinates) based on an image, relating lane edges to objects. Machine learning model 802 may be identical to or substantially similar to... Figure 1 Machine learning model 102 and / or Figure 7 The machine learning model 702, and / or can perform the same or substantially the same operations as it.

[0070] Additionally, in some cases, machine learning model 802 can generate bounding boxes 810 indicating the image coordinates of objects (e.g., signs) in image 804. For example, in addition to generating object-to-lane association points 806, machine learning model 802 can be trained to generate bounding boxes 810. Machine learning model 802 can be trained (e.g., via a backpropagation training process) to generate bounding boxes of objects in an image based on the image. Once trained, machine learning model 802 can be deployed in system 800 and can determine bounding boxes 810 based on image 804.

[0071] The bounding box 810 may include image coordinates that define a shape describing the location of an object (e.g., a sign) in the image 804. For example, Figure 6 The bounding box 616 describes the location of the mark 610 in the image 600. The bounding box 810 is optional in the system 800. The optional nature of the bounding box 810 in the system 800 is indicated by using a dashed line to indicate the bounding box 810.

[0072] Lane boundary 814 may include image coordinates (or three-dimensional coordinates) corresponding to lane lines in image 804. For example, lane boundary 814 may include... Figure 6 The image coordinates (or three-dimensional coordinates) corresponding to lane boundaries 606 and 608 of image 600. In some cases, lane boundary 814 may be associated with a lane. For example, in some cases, lane boundary 814 may include an identifier of the lane relative to the vehicle's lane.

[0073] In some aspects, lane association 820 may generate lane association 808 based on lane boundary 814 and object-to-lane association point 806 (and in some cases, bounding box 810). For example, in some aspects, lane association 820 may be or may include a machine learning model trained to generate lane associations based on object-to-lane association point and / or lane boundary (and in some cases, bounding box). In other aspects, lane association 820 may generate lane association 808 based on rules. In either case, lane association 820 may take object-to-lane association point 806 and lane boundary 814 (and in some cases, bounding box 810) as input and generate lane association 808 as output.

[0074] Lane association 808 may include not only the association between an object (e.g., a sign) and coordinates (as included in the object-to-lane association point 806), but also the association between the object and a lane relative to the vehicle's lane. In some cases, such lane associations may be useful for automated or semi-automated driving systems because such associations may be more directly related to driving decisions.

[0075] Figure 9This is a block diagram illustrating an example system 900 for determining lane information according to various aspects of this disclosure. System 900 includes a machine learning model 902 that receives an image 904 as input and generates an object-to-lane association point 906 as output. System 900 may be included in an automated or semi-automated driving system. The object-to-lane association point 906 may be used by the automated or semi-automated driving system to control the vehicle (e.g., based on instructions, restrictions, and / or navigation information provided by objects (such as signs) in the image 904). Additionally or alternatively, the object-to-lane association point 906 may be used to provide information to the driver.

[0076] Image 904 may be an image of a road and one or more objects adjacent to the road. Image 904 may be identical to or substantially similar to... Figure 1 Image 104 Figure 7 Image 704 and / or Figure 8 Image 804. Figure 2 Image 200 Figure 4 Image 400 and / or Figure 6 Image 600 can be an example of image 904. The object-to-lane association point 906 can be the coordinates of the edge of the lane. The coordinates can be image coordinates describing the edge of the lane when it appears in image 904. Additionally or alternatively, the coordinates can be three-dimensional coordinates of the lane edge (e.g., relative to the camera that captured image 904 or in a reference coordinate system). The object-to-lane association point 906 can be associated with an object in image 904. The object-to-lane association point 906 can be the same as or substantially similar to an object in image 904. Figure 1 Object to lane association point 106 Figure 7 Object to lane association point 706 and / or Figure 8 The object is associated with lane connection point 806. Figure 2 Object-to-lane association point 212 and object-to-lane association point 222 Figure 4 Object to lane association point 412, object to lane association point 422, object to lane association point 432, object to lane association point 442 and object to lane association point 452, and / or Figure 6 The object-to-lane association point 612 can be an example of the object-to-lane association point 906.

[0077] Machine learning model 902 may be or may include a trained neural network (e.g., a transformer). Machine learning model 902 may be trained (e.g., via a backpropagation training process) to generate coordinates (e.g., image coordinates or 3D coordinates) of lane edges associated with an object based on an image. Machine learning model 902 may be identical to or substantially similar to... Figure 1 Machine learning model 102 Figure 7Machine learning model 702 and / or Figure 8 The machine learning model 802, and / or can perform the same or substantially the same operations as it.

[0078] Additionally, in some cases, machine learning model 902 can generate lane associations 908. For example, in addition to generating object-to-lane association points 906, machine learning model 902 can be trained to generate lane associations 908. Lane associations 908 may be identical to or substantially similar to... Figure 7 Lane association 708 and / or Figure 8 Lane association 808. Lane association 908 is optional in system 900. The optional nature of lane association 908 in system 900 is illustrated by using a dashed line to represent lane association 908.

[0079] In some aspects, system 900 may include lane detector 910, which can generate lane boundaries 914 based on image 904. Lane detector 910 may use any suitable technique to generate lane boundaries 914. For example, lane detector 910 may include a neural network (e.g., a transformer) trained to detect lanes. In some aspects, lane boundaries 914 may be derived from map 912 (e.g., a map of the environment in which the vehicle is traveling, capturing image 904). In any case, lane boundaries 914 may include image coordinates (or three-dimensional coordinates) corresponding to lane lines in image 904. For example, lane boundaries 914 may include... Figure 6 Image coordinates (or 3D coordinates) corresponding to lane boundaries 606 and 608 of image 600. In some cases, lane boundary 914 may be associated with lane associations. For example, in some cases, lane boundary 914 may include an identifier of a lane relative to the vehicle's lane. In some aspects, machine learning model 902 may generate objects to lane association points 906 and / or lane associations 908 based on lane boundary 914. For example, machine learning model 902 may take image 904 and lane boundary 914 as input and generate objects to lane association points 906 and / or lane associations 908 as output. For example, machine learning model 902 may have been trained through a training process involving providing images and lane boundaries as input. Lane detector 910, map 912, and lane boundary 914 are optional in system 900. The optional nature of lane detector 910, map 912, and lane boundary 914 in system 900 is illustrated by using dashed lines to represent lane detector 910, map 912, and lane boundary 914.

[0080] In some aspects, system 900 may include object detector 916 that can generate bounding boxes 918 based on image 904. Object detector 916 may use any suitable technique to generate bounding boxes 918. For example, object detector 916 may include a neural network (e.g., a transformer) trained to detect objects. Bounding boxes 918 may be identical to or substantially similar to Figure 7 The bounding box 710 and / or Figure 8 The bounding box 810. In some aspects, the machine learning model 902 may generate object-to-lane association points 906 and / or lane associations 908 based on the bounding box 918. For example, the machine learning model 902 may take the image 904 and the bounding box 918 as input and generate object-to-lane association points 906 and / or lane associations 908 as output. For example, the machine learning model 902 may have been trained by a training process involving providing the image and the bounding box as input. The object detector 916 and the bounding box 918 are optional in the system 900. The optional nature of the object detector 916 and the bounding box 918 in the system 900 is illustrated by using dashed lines to represent the object detector 916 and the bounding box 918.

[0081] Figure 10 This is a flowchart illustrating a process 1000 for determining lane information according to various aspects of this disclosure. One or more operations of process 1000 may be performed by a computing device (or apparatus) or a component of such computing device (e.g., chipset, codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device (such as a watch), an extended reality (XR) device (such as a virtual reality (VR) device or an augmented reality (AR) device), a vehicle or a component or system of a vehicle, a desktop computing device, a tablet computing device, a server computer, a robotic device, and / or any other computing device having the resource capability to perform process 1000. One or more operations of process 1000 may be implemented as software components that execute and run on one or more processors.

[0082] At box 1002, a computing device (or one or more components thereof) may obtain an image representing one or more lanes and objects of a road, wherein the objects are adjacent to the road. For example, Figure 1 The machine learning model 102 can obtain image 104. For example, Figure 7 Machine learning model 702 can obtain image 704. For example, Figure 8 The machine learning model 802 can obtain image 804. For example, Figure 9 The machine learning model 902 can obtain image 904. Figure 2Image 200 may be an example of an image, such as including lanes 202, 204 and 206 of a road, and signs 210 and 220 adjacent to the road. Figure 2 Image 200 may be an example of an image, such as including lanes 202, 204 and 206 of a road, and signs 210 and 220 adjacent to the road. Figure 4 Image 400 may be an example of an image, such as including lanes 402, 404 and 406 of a road and signs 410, 420, 430, 440 and 450 adjacent to the road. Figure 6 Image 600 can be an example of an image, such as including lanes 602 and 604 of a road and a sign 610 adjacent to the road.

[0083] At box 1004, the computing device (or one or more components thereof) can determine the coordinates of an object in at least one of one or more lanes of a road to a lane-associated point, where the coordinates are associated with the object. For example, Figure 1 The machine learning model 102 can determine the lane association point 106 from the object. For example, Figure 7 The machine learning model 702 can generate objects to lane-related points 706. For example, Figure 8 The machine learning model 802 can generate objects to lane-related points 806. For example, Figure 9 The machine learning model 902 can generate objects to lane-related points 906.

[0084] In some respects, coordinates may be or may include image coordinates. For example, Figure 2 The object-to-lane association point 212 and the object-to-lane association point 222 can be examples of the determined object-to-lane association points. For example, the object-to-lane association point 212 can be or may include image coordinates of image 200.

[0085] In some respects, coordinates may be or may include image coordinates. For example, Figure 4 Object-to-lane association points 412, 422, 432, 442, and 452 can be examples of the determined object-to-lane association points. For example, object-to-lane association points 412, 422, 432, 442, and 452 can be or may include image coordinates of image 400.

[0086] In some respects, coordinates may be or may include image coordinates that are laterally offset from objects in the image. For example, object-to-lane-associated point 212 is offset from sign 210 in image 200, line 434 is laterally offset from sign 430, object-to-lane-associated point 452 is laterally offset from sign 450, and object-to-lane-associated point 612 is laterally offset from sign 610.

[0087] In some respects, coordinates may be or may include image coordinates at the level of a road in the image. For example, object-to-lane association point 212 and object-to-lane association point 222 may be at the level of lane 206 in image 200, object-to-lane association point 412 and object-to-lane association point 422 may be at the level of lanes 402 and 404 in image 400, object-to-lane association point 432 may be at the level of lanes 402, 404 and 406 in image 400, object-to-lane association point 442 and object-to-lane association point 452 may be at the level of lane 406 in image 400, and object-to-lane association point 612 may be at the level of lane 602 in image 600.

[0088] In some respects, coordinates may be or may include image coordinates lower than those of the object in the image. For example, object-to-lane-association point 212 is lower than sign 210 in image 200, object-to-lane-association point 222 is lower than sign 220 in image 200, object-to-lane-association point 412 is lower than sign 410 in image 400, object-to-lane-association point 422 is lower than sign 420 in image 400, object-to-lane-association point 432 is lower than sign 430 in image 400, object-to-lane-association point 442 is lower than sign 440 in image 400, object-to-lane-association point 452 is lower than sign 450 in image 400, and object-to-lane-association point 612 is lower than sign 610 in image 600.

[0089] In some respects, the coordinates include image coordinates, and the lines between the image coordinates are substantially perpendicular to the direction of travel of at least one lane.

[0090] For example, line 214 between an object and lane association point 212 may be perpendicular to the direction of travel in lane 206; line 224 between an object and lane association point 222 may be perpendicular to the direction of travel in lane 206; line 414 between an object and lane association point 412 may be perpendicular to the direction of travel in lane 402 and / or lane 404 (or the average direction of travel between lane 402 and lane 404); and line 424 between an object and lane association point 422 may be perpendicular to the direction of travel in lane 402 and / or lane 404 (or the average direction of travel between lane 402 and lane 404). The average driving direction), the line 434 between the object and the lane association point 432 may be perpendicular to the driving direction in lanes 402, 404 and / or lane 406 (or the average driving direction between lanes 402, 404 and / or lane 406), the line 444 between the object and the lane association point 442 may be perpendicular to the driving direction in lane 406, the line 454 between the object and the lane association point 452 may be perpendicular to the driving direction in lane 406, and the line 614 between the object and the lane association point 612 may be perpendicular to the driving direction in lane 602.

[0091] In some respects, coordinates may be or may include three-dimensional coordinates. For example, Figure 3 Object-to-lane association point 212 and object-to-lane association point 222 can be examples of the determined object-to-lane association points. For example, object-to-lane association point 212 can be or may include three-dimensional coordinates (e.g., mapped onto a bird's-eye view representation 300 for illustrative purposes). Object-to-lane association point 212 may be associated with sign 210, and object-to-lane association point 222 may be associated with sign 220.

[0092] In some respects, coordinates may be or may include three-dimensional coordinates. For example, Figure 5 Object-to-lane association points 412, 422, 432, 442, and 452 can be examples of the determined object-to-lane association points. For example, object-to-lane association points 412, 422, 432, 442, and 452 can be or may include three-dimensional coordinates (e.g., mapped onto a bird's-eye view representation 500 for illustrative purposes). Object-to-lane association point 412 may be associated with sign 410, object-to-lane association point 422 may be associated with sign 420, object-to-lane association point 432 may be associated with sign 430, object-to-lane association point 442 may be associated with sign 440, and object-to-lane association point 452 may be associated with sign 450. Figure 6The object-to-lane association point 612 can be an example of the determined object-to-lane association point. The object-to-lane association point 612 can be associated with the sign 610.

[0093] In some aspects, three-dimensional coordinates can be relative to the camera that captures the image. For example, coordinates may include indications of distance from the camera in three orthogonal directions. Alternatively, coordinates may include indications of azimuth, elevation, and the distance between the coordinates and the camera. In some aspects, three-dimensional coordinates can be relative to a reference coordinate system. For example, coordinates may include latitude and longitude.

[0094] In some aspects, the object may be or may include signs relating to at least one lane. In some aspects, the object may be or may include road signs that provide information relating to at least one lane. Figure 2 Marks 210 and 220 Figure 4 The signs 410, 420, 430, 440, and 450, and Figure 6 The flag 610 is an example of an object.

[0095] In some respects, the coordinates may indicate at least one lane associated with the object. For example, object-to-lane association point 212 may indicate the lane to which sign 210 belongs, object-to-lane association point 222 may indicate the lane to which sign 220 belongs, object-to-lane association point 412 may indicate the lane to which sign 410 belongs, object-to-lane association point 422 may indicate the lane to which sign 420 belongs, object-to-lane association point 432 may indicate the lane to which sign 430 belongs, object-to-lane association point 442 may indicate the lane to which sign 440 belongs, and object-to-lane association point 452 may indicate the lane to which sign 450 belongs.

[0096] In some respects, to determine coordinates, a computing device (or one or more components thereof) may feed an image to a neural network trained to determine coordinates representing the coordinates of an object associated with a lane-connected point; and obtain the coordinates from the neural network. For example, the computing device (or one or more components thereof) may feed an image to... Figure 1 The machine learning model 102 provides image 104, and the machine learning model 102 can generate objects to lane-related points 106. Alternatively, a computing device (or one or more components thereof) can provide... Figure 7 The machine learning model 702 provides image 704, and the machine learning model 702 can generate object-to-lane association points 706. Alternatively, a computing device (or one or more components thereof) can provide... Figure 8 The machine learning model 802 provides image 804, and the machine learning model 802 can generate objects to lane-related points 806. Alternatively, a computing device (or one or more components thereof) can provide... Figure 9The machine learning model 902 provides image 904, and the machine learning model 902 can generate objects to lane-related points 906.

[0097] In some respects, coordinates may be or may include image coordinates, and a neural network may be trained to determine the image coordinates of an object to a lane-related point. For example, machine learning model 102, machine learning model 702, machine learning model 802, and / or machine learning model 902 may be trained to generate image coordinates based on an image.

[0098] In some respects, the coordinates may be or may include three-dimensional coordinates, and the neural network may be trained to determine the three-dimensional coordinates of the object to the lane-related point. For example, machine learning model 102, machine learning model 702, machine learning model 802, and / or machine learning model 902 may be trained to generate three-dimensional coordinates based on an image.

[0099] In some respects, the computing device (or one or more components thereof) can obtain lane boundaries associated with an image and associate the lane boundaries with objects based on coordinates. For example, Figure 8 The system 800 can obtain lane boundary 814 and associate objects in image 804 with lane boundary 814 based on object-to-lane association points 806.

[0100] In some respects, to obtain lane boundaries, a computing device (or one or more components thereof) can feed an image to a neural network trained to determine lane boundaries based on the image, and obtain the lane boundaries from the neural network. For example, Figure 9 System 900 may provide image 904 to lane detector 910. Lane detector 910 may generate lane boundary 914 based on image 904. Machine learning model 902 may associate objects in image 904 with lane boundary 914 based on object-to-lane association points 906. In some aspects, lane boundary may be based on map information. For example, lane detector 910 may generate lane boundary 914 based on map 912.

[0101] In some respects, a computing device (or one or more components thereof) can feed an image to a neural network, which is trained to determine coordinates representing lane edges associated with objects and lane boundaries; obtain the coordinates from the neural network; and obtain the lane boundaries from the neural network. For example, Figure 9 The system 900 can provide an image 904 to the lane detector 910. The lane detector 910 can generate a lane boundary 914 based on the image 904.

[0102] In some respects, a computing device (or one or more components thereof) may feed an image to a neural network, which is trained to determine bounding boxes, and from which object-related bounding boxes are obtained. For example, system 900 may provide image 904 to object detector 916, and object detector 916 may determine bounding box 918 based on image 904.

[0103] In some respects, coordinates are determined based on bounding boxes. For example, machine learning model 902 can determine the object-to-lane association point 906 based at least in part on bounding box 918.

[0104] In some aspects, a computing device (or one or more components thereof) can determine, based on coordinates, the bird's-eye view coordinates corresponding to an object-to-lane association point in at least one of one or more lanes of a road. For example, machine learning models 102, 702, 802, and / or 902 can determine the bird's-eye view coordinates. For example, as... Figure 3 The bird's-eye view representation of the object-to-lane association point 212 and object-to-lane association point 222 illustrated in 300 can be bird's-eye view coordinates. For example, as... Figure 5 The object-to-lane association points 412, 422, 432, 442, and 452 illustrated in the bird's-eye view representation 500 can be bird's-eye view coordinates. In some respects, the computing device (or one or more components thereof) may track the bird's-eye view coordinates based on consecutive images.

[0105] In some respects, a computing device (or one or more components thereof) can determine, based on coordinates, the three-dimensional coordinates corresponding to an object-to-lane association point in at least one of one or more lanes of a road. For example, machine learning model 102, machine learning model 702, machine learning model 802, and / or machine learning model 902 can determine the three-dimensional coordinates.

[0106] For example, such as Figure 3 The object-to-lane association point 212 and object-to-lane association point 222 illustrated in the bird's-eye view representation 300 can be three-dimensional coordinates (e.g., projected onto a two-dimensional map such as bird's-eye view representation 300). For example, as Figure 5 The object-to-lane association points 412, 422, 432, 442, and 452 illustrated in the bird's-eye view representation 500 can be three-dimensional coordinates (e.g., projected onto a two-dimensional map such as the bird's-eye view representation 500). In some aspects, the computing device (or one or more components thereof) can track the three-dimensional coordinates based on successive images.

[0107] In some respects, computing devices (or one or more components thereof) can control vehicles based on coordinates. In other respects, computing devices (or one or more components thereof) can provide information to the driver of a vehicle based on coordinates.

[0108] In some examples, as previously noted, the methods described herein (e.g., Figure 10 The process 1000 and / or other methods described herein may be performed wholly or partially by a computing device or apparatus. In one example, one or more of these methods may be performed by... Figure 1 System 100 Figure 7 System 700, Figure 9 The system 900 or is executed by another system or device. For example, these methods (e.g., Figure 10 One or more of the processes 1000 and / or other methods described herein may be used by Figure 12 The computing device architecture 1200 shown is implemented wholly or partially. For example, it has Figure 12 The computing device of the computing device architecture 1200 shown may include or be included in components of system 100, system 700, and / or system 900, and may perform the operation of process 1000 and / or other processes described herein. In some cases, the computing device or apparatus may include various components such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0109] A component capable of implementing a computing device in a circuit. For example, the component may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0110] Process 1000 and / or other processes described herein are illustrated as logic flowcharts, whose operations represent sequences of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0111] Additionally, process 1000 and / or other processes described herein may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, implemented in hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0112] As noted above, various aspects of this disclosure may utilize machine learning models or systems.

[0113] Figure 11 This is an exemplary example of a neural network 1100 (e.g., a deep learning neural network) that can be used to implement machine learning-based object detection, lane detection, lane association, lane edge determination, feature segmentation, implicit neural representation generation, rendering, classification, image recognition (e.g., face recognition, object recognition, scene recognition, etc.), feature extraction, authentication, gaze detection, gaze prediction, and / or automation. For example, neural network 1100 could be... Figure 1 Machine learning model 102 Figure 7 Machine learning model 702 Figure 9 Machine learning model 902 Figure 9 Lane detector 910 and / or Figure 9 Examples of object detectors 916, or implementations thereof.

[0114] Input layer 1102 includes input data. In an exemplary example, input layer 1102 may include representations... Figure 1 Image 104 Figure 2 Image 200 Figure 4 Image 400 Figure 6 Image 600 Figure 7Image 704 Figure 9 Image 904 Figure 9 Lane boundaries 914 and / or Figure 9 The data for the bounding box 918. The neural network 1100 includes multiple hidden layers 1006a, 1006b through 1006n. Hidden layers 1006a, 1006b through 1006n comprise "n" hidden layers, where "n" is an integer greater than or equal to one. Multiple hidden layers can be made to include as many layers as needed for a given application. The neural network 1100 further includes an output layer 1104, which provides the output produced by the processing performed by the hidden layers 1006a, 1006b through 1006n. In an exemplary example, output layer 1104 provides... Figure 1 Object to lane association point 106 Figure 2 Object to lane association point 212, line 214, object to lane association point 222 and / or line 224 Figure 4 Object to lane association point 412, line 414, object to lane association point 422, line 424, object to lane association point 432, line 434, object to lane association point 442, line 444, object to lane association point 452 and / or line 454, Figure 6 Object to lane association point 612 and / or line 614, Figure 7 Object to lane association point 706, lane association 708 and / or bounding box 710, and / or Figure 9 The object is associated with lane association point 906 and / or lane association point 908.

[0115] Neural network 1100 may be or may include a multi-layer neural network with interconnected nodes. Each node may represent a piece of information. The information associated with these nodes is shared between different layers, and each layer retains information while processing it. In some cases, neural network 1100 may include a feedforward network, in which case there are no feedback connections in which the network's output is fed back into itself. In some cases, neural network 1100 may include a recurrent neural network, which may have loops that allow information to be carried across nodes when reading input.

[0116] Information can be exchanged between nodes via node-to-node interconnects between layers. Nodes in input layer 1102 can activate the node set in the first hidden layer 1006a. For example, as shown, each input node in input layer 1102 is connected to each node in the first hidden layer 1006a. Nodes in the first hidden layer 1006a can transform the information of each input node by applying an activation function to the input node information. The information derived from this transformation can then be passed to nodes in the next hidden layer 1006b, activating those nodes, which can then perform their own specified functions. Example functions include convolution, upsampling, data transformation, and / or any other suitable functions. The output of hidden layer 1006b can then activate nodes in the next hidden layer, and so on. The output of the last hidden layer 1006n can activate one or more nodes in output layer 1104, at which the output is provided. In some cases, although a node in neural network 1100 (e.g., node 1108) is shown as having multiple output lines, the node has a single output and is shown as all lines output from the node representing the same output value.

[0117] In some cases, each node or the interconnection between nodes may have weights, which are a set of parameters derived from the training of the neural network 1100. Once the neural network 1100 is trained, it can be referred to as a trained neural network, which can be used to perform one or more operations. For example, the interconnection between nodes may represent a piece of information about what the interconnected nodes have learned. The interconnection may have tunable numerical weights that can be tuned (e.g., based on the training dataset), thereby allowing the neural network 1100 to adapt to the input and learn as more and more data is processed.

[0118] The neural network 1100 can be pre-trained to process features from the data in the input layer 1102 using different hidden layers 1006a, 1006b to 1006n, so as to provide an output through the output layer 1104. In an example where the neural network 1100 is used to identify features in an image, the neural network 1100 can be trained using training data that includes both images and labels, as described above. For example, training images can be input into the network, where each training image has a label indicating features in the image (for feature segmentation machine learning systems) or a label indicating the category of activity in each image. In an example where object classification is used for illustrative purposes, the training images may include images of the number 2, in which case the label of the image may be [0 0 1 0 0 0 0 0 0 0].

[0119] In some cases, the neural network 1100 can use a training process called backpropagation to adjust the weights of its nodes. As noted above, the backpropagation process can include forward pass, loss function, back pass, and weight update. For each training iteration, forward pass, loss function, back pass, and parameter update are performed. For each set of training images, this process can be repeated up to a certain number of iterations until the neural network 1100 is trained well enough to accurately tune the weights of each layer.

[0120] For an example of identifying objects in an image, the forward pass may include passing a training image through a neural network 1100. The weights are initially randomized before training the neural network 1100. As an illustrative example, the image may include a numerical array representing the pixels of the image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or lightness and two chroma components, etc.).

[0121] As noted above, for the first training iteration of the neural network 1100, the output will likely include values ​​due to the weights being randomly selected during initialization without prioritizing any particular class. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each class may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 1100 cannot determine low-level features and therefore cannot make an accurate judgment about what the object might be classified as. A loss function can be used to analyze the error in the output. Any suitable loss function can be defined, such as cross-entropy loss. Another example of a loss function includes mean squared error (MSE), which is defined as... The loss can be set to equal E. 总和 The value of .

[0122] For the first training image, the loss (or error) will be high because the actual value will be significantly different from the predicted output. The goal of training is to minimize the loss so that the predicted output matches the training labels. The Neural Network 1100 performs backpropagation by determining which inputs (weights) contribute most to the network's loss and can adjust the weights to reduce and eventually minimize the loss. The derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight at a specific layer) can be calculated to determine the weights that contribute most to the network's loss. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. A weight update can be represented as... Where w represents the weight, w i Let represent the initial weights, and η represent the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes larger weight updates, while a lower value indicates smaller weight updates.

[0123] Neural Network 1100 can include any suitable deep network. An example includes a Convolutional Neural Network (CNN), which includes an input layer and an output layer, with multiple hidden layers between the input and output layers. The hidden layers of a CNN include a series of convolutional layers, non-linear layers, pooling layers (for downsampling), and fully connected layers. Neural Network 1100 can include any other deep network besides CNNs, such as transformers, autoencoders, deep belief networks (DBNs), recurrent neural networks (RNNs), etc.

[0124] Figure 12 Example computing device architecture 1200 illustrates example computing devices that can implement the various technologies described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device in a vehicle), or other devices. For example, computing device architecture 1200 may include, implement Figure 1 System 100 Figure 7 System 700 and / or Figure 9 The system 900 may contain any or all of the following systems, or may be included therein. Additionally or alternatively, the computing device architecture 1200 may be configured to execute process 1000 and / or other processes described herein.

[0125] The components of computing device architecture 1200 are shown to communicate electrically with each other using a connection 1212, such as a bus. Example computing device architecture 1200 includes a processing unit (CPU or processor) 1202 and a computing device connection 1212 that couples various computing device components, including computing device memories 1210 (such as read-only memory (ROM) 1208 and random access memory (RAM) 1206), to the processor 1202.

[0126] The computing device architecture 1200 may include a cache of high-speed memory that is directly connected to, very close to, or integrated as part of the processor 1202. The computing device architecture 1200 may copy data from memory 1210 and / or storage device 1214 to cache 1204 for fast access by the processor 1202. In this way, the cache can provide performance improvements by avoiding latency for the processor 1202 while waiting for data. These and other modules may control or be configured to control the processor 1202 to perform various actions. Other computing device memory 1210 may also be available. Memory 1210 may include various different types of memory with different performance characteristics. The processor 1202 may include any general-purpose processor and hardware or software services configured to control the processor 1202 (such as services 11216, 1218, and 31220 stored in storage device 1214), as well as dedicated processors in which software instructions are incorporated into the processor design. The processor 1202 may be a self-contained system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0127] To enable user interaction with the computing device architecture 1200, input device 1222 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. Output device 1224 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device, etc. In some instances, multi-mode computing devices allow users to provide multiple types of input to communicate with computing device architecture 1200. Communication interface 1226 typically controls and manages user input and computing device output. There are no limitations on operation on any particular hardware arrangement, and therefore the underlying features here can be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0128] Storage device 1214 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as a magnetic tape cassette, flash memory card, solid-state memory device, digital multifunction disk, magnetic tape cartridge, random access memory (RAM) 1206, read-only memory (ROM) 1208, and hybrid forms thereof. Storage device 1214 may include services 1216, 1218, and 1220 for controlling processor 1202. Other hardware or software modules are envisioned. Storage device 1214 may be connected to computing device connection 1212. In one aspect, a hardware module performing a particular function may include software components for performing that function stored in a computer-readable medium connected to necessary hardware components such as processor 1202, connection 1212, output device 1224, etc.

[0129] With reference to a given parameter, property, or condition, the term "substantially" may mean that a person skilled in the art would understand that a given parameter, property, or condition is satisfied with a small degree of variance (such as, for example, within acceptable manufacturing tolerances). For example, depending on the specific parameter, property, or condition that is substantially satisfied, the parameter, property, or condition may be satisfied at least 90%, at least 95%, or even at least 99%.

[0130] Various aspects of this disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although devices having or coupled to a light projector are described below, various aspects of this disclosure are applicable to devices having any number of light projectors and are therefore not limited to any particular device.

[0131] The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that implement at least some parts of this disclosure. Although the following description and examples use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or specific aspects. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. Although the following description and examples use the term "system" to describe various aspects of this disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0132] Specific details are provided in the foregoing description to provide a thorough understanding of the aspects and examples presented herein. However, those skilled in the art will understand that these aspects can be practiced without these specific details. For clarity, in some cases, the technology may be presented as comprising individual functional blocks, including functional blocks comprising devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these aspects in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary detail to avoid obscuring the aspects.

[0133] Various aspects described above can be presented as processes or methods, depicted as flowcharts, diagrams, data flow graphs, structure diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations within an operation can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying diagrams. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0134] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure, cause or otherwise configure, a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion of the computer resources used may be accessible via a network. Computer-executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0135] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored and which do not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, magnetic disks or optical disks, USB devices equipped with non-volatile memory, network storage devices, any suitable combinations thereof, etc. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, independent variables, parameters, data, etc., can be transmitted, forwarded, or sent through any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0136] In some respects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0137] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or interlocking cards. By further example, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0138] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0139] In the foregoing description, aspects of this application have been described with reference to their specific aspects, but those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative aspects of this application have been described in detail herein, it is to be understood that various inventive concepts may be embodied and employed in various other ways, and the appended claims are not intended to be construed as including these variations unless limited by prior art. The various features and aspects of the applications described above may be used individually or in combination. Furthermore, aspects may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that, in alternative aspects, the methods may be performed in a different order than described.

[0140] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols without departing from the scope of this description.

[0141] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0142] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0143] Claim language or other languages ​​that state "at least one of" and / or "one or more of" in a set indicate that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any repeating information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the language of a claim stating "at least one of A and B" or "at least one of A or B" may refer to A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases "at least one" and "one or more" are used interchangeably herein.

[0144] Claims using phrases such as "at least one processor, the at least one processor being configured to," "at least one processor being configured to," "one or more processors, the one or more processors being configured to," or "one or more processors being configured to," or other languages, indicate that one or more processors (in any combination) are capable of performing associated operations. For example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each assigned a specific subset of tasks to perform operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" could mean that any single processor can perform only at least one subset of operations X, Y, and Z.

[0145] When referring to one or more elements that perform functions (e.g., steps of a method), one element may perform all functions, or more than one element may jointly perform these functions. When more than one element jointly performs these functions, each function does not need to be performed by every single element (e.g., different functions may be performed by different elements), and / or each function does not need to be performed by only one element as a whole (e.g., different elements may perform different sub-functions of a function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform functions, one element may be configured to cause another element to perform all functions, or more than one element may be jointly configured to cause another element to perform these functions.

[0146] When referring to an entity that performs or is configured to perform functions (e.g., steps of a method) (e.g., any entity or device described herein), the entity may be configured to cause one or more elements (individually or collectively) to perform those functions. One or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more of those functions, and / or any combination thereof. When referring to an entity that performs functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to perform those functions collectively. When the entity is configured to cause more than one component to perform those functions collectively, each function does not need to be performed by every single component (e.g., different functions may be performed by different components), and / or each function does not need to be performed by only one component as a whole (e.g., different components may perform different sub-functions of a function).

[0147] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been broadly described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be construed as departing from the scope of this application.

[0148] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0149] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor), multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein.

[0150] The exemplary aspects of this disclosure include:

[0151] Aspect 1. An apparatus for determining lane information, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: acquire an image representing one or more lanes of a road and an object, wherein the object is adjacent to the road; and determine coordinates of the object to a lane-associated point of at least one of the one or more lanes of the road, wherein the coordinates are associated with the object.

[0152] Aspect 2. The apparatus according to aspect 1, wherein the coordinates include image coordinates.

[0153] Aspect 3. The apparatus according to any one of Aspect 1 or 2, wherein the coordinates include three-dimensional coordinates.

[0154] Aspect 4. The apparatus according to aspect 3, wherein the three-dimensional coordinates are relative to the camera that captures the image.

[0155] Aspect 5. The apparatus according to any one of Aspects 3 or 4, wherein the three-dimensional coordinates are relative to a reference coordinate system.

[0156] Aspect 6. The apparatus according to any one of Aspects 1 to 5, wherein the object includes a sign associated with the at least one lane.

[0157] Aspect 7. The apparatus according to any one of Aspects 1 to 6, wherein the object includes a road sign that provides information relating to the at least one lane.

[0158] Aspect 8. The apparatus according to any one of aspects 1 to 7, wherein the coordinates indicate the at least one lane associated with the object.

[0159] Aspect 9. The apparatus according to any one of Aspects 1 to 8, wherein the coordinates include image coordinates that are laterally offset in the image relative to the object in the image.

[0160] Aspect 10. The apparatus according to any one of Aspects 1 to 9, wherein the coordinates include image coordinates of the layer of the road in the image.

[0161] Aspect 11. The apparatus according to any one of Aspects 1 to 10, wherein the coordinates include image coordinates in the image that are lower than those of the object.

[0162] Aspect 12. The apparatus according to any one of Aspects 1 to 11, wherein the coordinates include image coordinates, and wherein the line between the image coordinates is substantially perpendicular to the direction of travel of the at least one lane.

[0163] Aspect 13. The apparatus according to any one of Aspects 1 to 12, wherein, in order to determine the coordinates, the at least one processor is configured to: provide the image to a neural network, the neural network being trained to determine coordinates representing an object-to-lane association point associated with an object; and obtain the coordinates from the neural network.

[0164] Aspect 14. The apparatus according to aspect 13, wherein the coordinates include image coordinates, and wherein the neural network is trained to determine the image coordinates of an object to a lane-related point.

[0165] Aspect 15. The apparatus according to any one of Aspects 13 or 14, wherein the coordinates include three-dimensional coordinates, and wherein the neural network is trained to determine the three-dimensional coordinates of an object to a lane-associated point.

[0166] Aspect 16. The apparatus according to any one of Aspects 1 to 15, wherein the at least one processor is further configured to: obtain a lane boundary associated with the image; and associate the lane boundary with the object based on the coordinates.

[0167] Aspect 17. The apparatus according to aspect 16, wherein, in order to obtain the lane boundary, the at least one processor is configured to: provide the image to a neural network, the neural network being trained to determine the lane boundary based on the image; and obtain the lane boundary from the neural network.

[0168] Aspect 18. The apparatus according to any one of Aspects 16 or 17, wherein the lane boundaries are based on map information.

[0169] Aspect 19. The apparatus according to any one of Aspects 1 to 18, wherein the at least one processor is further configured to: provide the image to a neural network, the neural network being trained to determine coordinates representing lane edges associated with an object and a lane boundary; obtain the coordinates from the neural network; and obtain the lane boundary from the neural network.

[0170] Aspect 20. The apparatus according to any one of aspects 1 to 19, wherein the at least one processor is further configured to: provide the image to a neural network, the neural network being trained to determine bounding boxes; and obtain bounding boxes associated with the object from the neural network.

[0171] Aspect 21. The apparatus according to aspect 20, wherein the coordinates are determined based on the bounding box.

[0172] Aspect 22. The apparatus according to any one of aspects 1 to 21, wherein the at least one processor is further configured to determine, based on the coordinates, bird's-eye view coordinates corresponding to the object-to-lane association point of at least one of the one or more lanes of the road.

[0173] Aspect 23. The apparatus according to aspect 22, wherein the at least one processor is further configured to track the bird's-eye view coordinates based on continuous images.

[0174] Aspect 24. The apparatus according to any one of aspects 1 to 23, wherein the at least one processor is further configured to determine, based on the coordinates, three-dimensional coordinates corresponding to the object-to-lane association point of at least one lane in one or more lanes of the road.

[0175] Aspect 25. The apparatus according to aspect 24, wherein the at least one processor is further configured to track the three-dimensional coordinates based on continuous images.

[0176] Aspect 26. The apparatus according to any one of aspects 1 to 25, wherein the at least one processor is further configured to control the vehicle based on the coordinates.

[0177] Aspect 27. The apparatus according to any one of Aspects 1 to 26, wherein the at least one processor is further configured to provide information to the driver of the vehicle based on the coordinates.

[0178] Aspect 28. A method for determining lane information, the method comprising: obtaining an image representing one or more lanes of a road and an object, wherein the object is adjacent to the road; and determining coordinates of the object to a lane-associated point in at least one of the one or more lanes of the road, wherein the coordinates are associated with the object.

[0179] Aspect 29. The method according to aspect 28, wherein the coordinates include image coordinates.

[0180] Aspect 30. The method according to any one of Aspects 28 or 29, wherein the coordinates include three-dimensional coordinates.

[0181] Aspect 31. The method according to aspect 30, wherein the three-dimensional coordinates are relative to the camera that captures the image.

[0182] Aspect 32. The method according to any one of Aspects 30 or 31, wherein the three-dimensional coordinates are relative to a reference coordinate system.

[0183] Aspect 33. The method according to any one of Aspects 28 to 32, wherein the object includes a sign associated with the at least one lane.

[0184] Aspect 34. The method according to any one of Aspects 28 to 33, wherein the object includes a road sign that provides information relating to the at least one lane.

[0185] Aspect 35. The method according to any one of Aspects 28 to 34, wherein the coordinates indicate the at least one lane associated with the object.

[0186] Aspect 36. The method according to any one of Aspects 28 to 35, wherein the coordinates include image coordinates that are laterally offset in the image relative to the object in the image.

[0187] Aspect 37. The method according to any one of Aspects 28 to 36, wherein the coordinates include image coordinates of the layer of the road in the image.

[0188] Aspect 38. The method according to any one of Aspects 28 to 37, wherein the coordinates include image coordinates in the image that are lower than those of the object.

[0189] Aspect 39. The method according to any one of Aspects 28 to 38, wherein the coordinates include image coordinates, and wherein the line between the image coordinates is substantially perpendicular to the direction of travel of the at least one lane.

[0190] Aspect 40. The method according to any one of Aspects 28 to 39, wherein determining the coordinates comprises: providing the image to a neural network, the neural network being trained to determine coordinates representing an object-to-lane association point associated with an object; and obtaining the coordinates from the neural network.

[0191] Aspect 41. The method according to aspect 40, wherein the coordinates include image coordinates, and wherein the neural network is trained to determine the image coordinates of an object to a lane-related point.

[0192] Aspect 42. The method according to any one of Aspects 40 or 41, wherein the coordinates include three-dimensional coordinates, and wherein the neural network is trained to determine the three-dimensional coordinates of the object to the lane association point.

[0193] Aspect 43. The method according to any one of Aspects 28 to 42, the method further comprising: obtaining a lane boundary associated with the image; and associating the lane boundary with the object based on the coordinates.

[0194] Aspect 44. The method according to aspect 43, wherein obtaining the lane boundary comprises: providing the image to a neural network, the neural network being trained to determine the lane boundary based on the image; and obtaining the lane boundary from the neural network.

[0195] Aspect 45. The method according to any one of Aspects 43 or 44, wherein the lane boundary is based on map information.

[0196] Aspect 46. The method according to any one of Aspects 28 to 45, the method further comprising: providing the image to a neural network, the neural network being trained to determine coordinates representing lane edges associated with an object and a lane boundary; obtaining the coordinates from the neural network; and obtaining the lane boundary from the neural network.

[0197] Aspect 47. The method according to any one of Aspects 28 to 46, the method further comprising: providing the image to a neural network, the neural network being trained to determine bounding boxes; and obtaining bounding boxes associated with the object from the neural network.

[0198] Aspect 48. The method according to aspect 47, wherein the coordinates are determined based on the bounding box.

[0199] Aspect 49. The method according to any one of Aspects 28 to 48, the method further comprising: determining, based on the coordinates, bird's-eye view coordinates corresponding to an object-to-lane association point of at least one of the one or more lanes of the road.

[0200] Aspect 50. The method according to aspect 49, the method further comprising: tracking the coordinates of the bird's-eye view based on continuous images.

[0201] Aspect 51. The method according to any one of Aspects 28 to 50, the method further comprising: determining, based on the coordinates, three-dimensional coordinates corresponding to an object-to-lane association point of at least one of the one or more lanes of the road.

[0202] Aspect 52. The method according to aspect 51, the method further comprising: tracking the three-dimensional coordinates based on continuous images.

[0203] Aspect 53. The method according to any one of Aspects 28 to 52, the method further comprising: controlling the vehicle based on the coordinates.

[0204] Aspect 54. The method according to any one of Aspects 28 to 53, the method further comprising: providing information to the driver of the vehicle based on the coordinates.

[0205] Aspect 55. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions causing the at least one processor, when executed, to perform any one of aspects 28 to 54.

[0206] Aspect 56. An apparatus for providing virtual content for display, the apparatus comprising one or more components for performing operations according to any one of aspects 28 to 54.

Claims

1. An apparatus for determining lane information, the apparatus comprising: At least one memory; and At least one processor, the at least one processor being coupled to the at least one memory and being configured to: Obtain an image representing one or more lanes and objects of a road, wherein the objects are adjacent to the road; and Determine the coordinates of an object to a lane-associated point in at least one of the one or more lanes of the road, wherein the coordinates are associated with the object.

2. The apparatus according to claim 1, wherein the coordinates include image coordinates.

3. The apparatus according to claim 1, wherein the coordinates include three-dimensional coordinates.

4. The apparatus of claim 3, wherein the three-dimensional coordinates are relative to the camera that captures the image.

5. The apparatus of claim 3, wherein the three-dimensional coordinates are relative to a reference coordinate system.

6. The apparatus of claim 1, wherein the object includes a sign associated with the at least one lane.

7. The apparatus of claim 1, wherein the object comprises a road sign that provides information relating to the at least one lane.

8. The apparatus of claim 1, wherein the coordinates indicate the at least one lane associated with the object.

9. The apparatus of claim 1, wherein the coordinates include image coordinates that are laterally offset relative to the object in the image.

10. The apparatus of claim 1, wherein the coordinates include image coordinates of the layer of the road in the image.

11. The apparatus of claim 1, wherein the coordinates include image coordinates in the image that are lower than those of the object.

12. The apparatus of claim 1, wherein the coordinates include image coordinates, and wherein the line between the image coordinates is substantially perpendicular to the direction of travel of the at least one lane.

13. The apparatus according to claim 1, wherein, To determine the coordinates, the at least one processor is configured to: The image is fed to a neural network, which is trained to determine coordinates representing the object's coordinates to a lane-associated point; and The coordinates are obtained from the neural network.

14. The apparatus of claim 13, wherein the coordinates include image coordinates, and wherein the neural network is trained to determine the image coordinates of an object-to-lane association point.

15. The apparatus of claim 13, wherein the coordinates include three-dimensional coordinates, and wherein the neural network is trained to determine the three-dimensional coordinates of the object to the lane-associated point.

16. The apparatus of claim 1, wherein the at least one processor is further configured to: Obtain the lane boundaries associated with the image; and The lane boundary is associated with the object based on the coordinates.

17. The apparatus according to claim 16, wherein, In order to obtain the lane boundary, the at least one processor is configured to: The image is fed to a neural network, which is trained to determine lane boundaries based on the image; and The lane boundary is obtained from the neural network.

18. The apparatus of claim 16, wherein the lane boundaries are based on map information.

19. The apparatus of claim 1, wherein the at least one processor is further configured to: The image is fed into a neural network, which is trained to determine coordinates representing lane edges associated with objects and lane boundaries; The coordinates are obtained from the neural network; and Lane boundaries are obtained from the neural network.

20. The apparatus of claim 1, wherein the at least one processor is further configured to: The image is fed to a neural network, which is trained to determine bounding boxes; and Obtain the bounding box associated with the object from the neural network.

21. The apparatus of claim 20, wherein the coordinates are determined based on the bounding box.

22. The apparatus of claim 1, wherein the at least one processor is further configured to determine, based on the coordinates, bird's-eye view coordinates corresponding to the object-to-lane association point of at least one of the one or more lanes of the road.

23. The apparatus of claim 22, wherein the at least one processor is further configured to track the bird's-eye view coordinates based on continuous images.

24. The apparatus of claim 1, wherein the at least one processor is further configured to determine, based on the coordinates, three-dimensional coordinates corresponding to the object-to-lane association point of at least one lane in one or more lanes of the road.

25. The apparatus of claim 24, wherein the at least one processor is further configured to track the three-dimensional coordinates based on continuous images.

26. The apparatus of claim 1, wherein the at least one processor is further configured to control the vehicle based on the coordinates.

27. The apparatus of claim 1, wherein the at least one processor is further configured to provide information to the driver of the vehicle based on the coordinates.

28. A method for determining lane information, the method comprising: Obtain an image representing one or more lanes and objects of a road, wherein the objects are adjacent to the road; as well as Determine the coordinates of an object to a lane-associated point in at least one of the one or more lanes of the road, wherein the coordinates are associated with the object.

29. The method of claim 28, wherein the coordinates include image coordinates.

30. The method of claim 28, wherein the coordinates include three-dimensional coordinates.

Images