Landmark Position Reconstruction in Autonomous Machine Applications

Abstract

The present invention discloses a reconstruction of landmark positions in autonomous machine applications. In various examples, 3D reconstruction can be used to identify the positions of directional landmarks (such as vertical landmarks). A set of observations of directional landmarks (e.g., images captured from a moving vehicle) can be reduced to a one-dimensional lookup table by modifying the observations to align the directional landmarks along a specific direction of the observations. Object detection can be applied, and a corresponding one-dimensional lookup table can be generated to represent the presence of vertical landmarks detected in the image.

Description

Background Art

[0001] Embodiments of the present disclosure relate to reconstruction of three-dimensional (3D) objects, and more particularly, to reconstruction of 3D directional, linear and / or vertical objects. 3D reconstruction has applications in many fields, such as computer vision and computer graphics, usually to capture the shape and appearance of real-world objects.

[0002] Traditional 3D reconstruction techniques have many drawbacks. For example, direct triangulation attempts to reconstruct features (e.g., 3D features) from a set of two-dimensional (2D) observations and associated camera poses (e.g., orientation and position). However, direct triangulation has many disadvantages. For example, direct triangulation requires data association across 2D observations (e.g., 2D observation trajectories given 3D points), but data association is often difficult to achieve in the presence of fast and large motions. In some cases, the reconstructed features may not appear in certain 2D observations. In addition, direct triangulation may be sensitive to noisy observations and camera orientation, may not be easy to parallelize, and may be difficult to guarantee an upper limit on the computation in the presence of noisy observations.

[0003] Another traditional technique for performing 3D reconstruction is volumetric reconstruction, which samples 3D space and finds the markers of 3D objects in a set of 2D observations. However, volumetric reconstruction requires solving a large number of independent problems: namely, the calculation of the joint probability at each point sampled in 3D space. Due to the computational complexity, traditional volumetric reconstruction methods usually cannot be executed in sufficient time for real-time applications. Summary of the invention

[0004] In the context of autonomous driving, 3D landmarks can be used for positioning (e.g., locating a vehicle on a map), while vertical landmarks can provide important longitude accuracy. Embodiments of the present disclosure relate to techniques for identifying the location of directional landmarks using 3D reconstruction. More specifically, systems and methods for identifying the location of directional landmarks, such as vertical landmarks, are disclosed. Vertical landmarks may include utility poles, sign poles, straight trees, light poles, and / or objects such as these. Image data and / or data from other indicia of directional landmarks can be used to determine their locations. Compared to conventional systems, the system of the present disclosure can merge marker data into a simplified form reflecting directional landmarks without the need to store, retrieve, or analyze all original marker data. The merged marker data can be used to reconstruct the location of directional landmarks in 3D space. Compared to conventional systems, 3D reconstruction can be simplified to a 2D task, for example, by determining the location of directional landmarks on a 2D plane (e.g., the ground plane of 3D space). The location of the detected landmark can be embedded in a map, provided to a map provider for integration into a map, or otherwise. In embodiments involving vertical landmark detection, the detected vertical landmarks can help autonomous vehicles navigate in a physical environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] With reference to the accompanying drawings, the following describes in detail a prior art system and method for landmark position reconstruction in autonomous machine applications, wherein:

[0006] Figure 1 is a block diagram of an example landmark detector according to some embodiments of the present disclosure;

[0007] Figure 2A is an illustration of examples of sensor configurations and corresponding two-dimensional observations according to some embodiments of the present disclosure;

[0008] Figure 2B is an illustration of an example of another sensor configuration and corresponding two-dimensional observations according to some embodiments of the present disclosure;

[0009] Figure 2C is an illustration of an example of another sensor configuration and corresponding two-dimensional observations according to some embodiments of the present disclosure;

[0010] Figure 3 is an illustration of examples of image correction and lookup table generation according to some embodiments of the present disclosure;

[0011] Figure 4 is an illustration of an example of grid sampling for vertical landmark reconstruction according to some embodiments of the present disclosure;

[0012] Figure 5is an illustration of an example of pattern finding for vertical landmark reconstruction according to some embodiments of the present disclosure;

[0013] Figure 6 is an illustration of an example of seeding along a line orthogonal to the displacement of a landmark detector according to some embodiments of the present disclosure;

[0014] Figure 7 is a flow chart illustrating a method for vertical landmark reconstruction according to some embodiments of the present disclosure;

[0015] Figure 8 is a flow chart illustrating a method for directional landmark reconstruction according to some embodiments of the present disclosure;

[0016] Fig. 9 is a flow chart illustrating another method for directional landmark reconstruction according to some embodiments of the present disclosure;

[0017] Fig.10 are examples of possible camera locations and corresponding fields of view on a vehicle according to some embodiments of the present disclosure;

[0018] Fig.11 is a diagram of a system for transmitting detected landmark quantities to one or more vehicles and a cloud-based server according to some embodiments of the present disclosure; and

[0019] Fig.12 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure. Detailed Description

[0020] Systems and methods are disclosed herein that relate to the detection of three-dimensional (3D) directional landmarks (e.g., 3D vertical landmarks). The disclosure may be described with respect to a vehicle that facilitates landmark detection. Examples of suitable vehicles are described below with respect to Fig.10 and Fig.11 Detailed description. Furthermore, while the present disclosure may be described with respect to a vehicle that facilitates landmark detection, this is not necessarily the case. The methods and processes described herein may be implemented without a vehicle and / or in a configuration that includes additional or alternative structures, components, and / or architectures without departing from the scope of the present disclosure. For example, the present disclosure may be applicable to robotic applications, augmented reality (AR) applications, virtual reality (VR) applications, remote controlled machines or unmanned aerial vehicles (UAV) applications, and / or other applications where identifying vertical landmarks may aid in positioning, navigation, and / or other uses.

[0021] The present disclosure relates to techniques for identifying the locations of directional landmarks using 3D reconstruction. More specifically, embodiments described herein relate to methods and processes for marker capture, rectification, marker merging, 3D reconstruction, and / or pattern recognition. In some examples, marker merging can reduce 2D observations to corresponding 1D lookups compared to traditional methods using two-dimensional (2D) lookups, thereby reducing processing and storage requirements. In addition or alternatively, reconstruction of 3D directional landmarks (such as vertical landmarks) can be limited to 2D reconstructions, thereby avoiding the need to sample one of the dimensions of the 3D grid and reducing many of the corresponding calculations required compared to using a 3D grid.

[0022] As described herein, some conventional systems use direct triangulation to reconstruct 3D features from a set of two-dimensional (2D) observations and associated camera orientations. Direct triangulation can be used to estimate the 3D position of an observed 3D point using a direct linear transformation and selecting a scheme that minimizes algebraic and / or geometric errors over the entire set of observations. However, direct triangulation has many disadvantages. For example, direct triangulation requires data association across 2D observations (e.g., the trajectory of a 2D observation given a 3D point), but data association is often difficult to achieve in the presence of rapid and large motion. In addition, direct triangulation may be sensitive to noisy observations and camera orientations, may not be easily parallelizable, and in the presence of noisy observations, it is difficult to guarantee an upper limit on the computation.

[0023] Additionally, some conventional systems perform 3D reconstruction using volumetric reconstruction, which samples 3D space and finds landmarks of 3D objects in a set of 2D observations. This principle is used in many applications, such as spatial carving from photo coherence, shaping from silhouettes, and probabilistic reconstruction of moving objects via background subtraction. In general, volumetric reconstruction constructs a 3D probability distribution that reflects the joint probability of the presence of a 3D object at a particular 3D location given the presence of that object in a set of 2D observations. However, volumetric reconstruction requires solving a large number of independent problems, namely computing the joint probability at each point sampled in 3D space. The computational complexity of computing the 3D probability distribution is n d , where n is the number of samples per dimension and d is the number of dimensions (here, d = 3). In practice, if volume reconstruction is not heavily parallelized, it cannot be performed in sufficient time for real-time applications.

[0024] Compared to conventional systems, embodiments of the present disclosure relate to reconstructing 3D vertical landmarks or other 3D directional landmarks from a modified marker lookup table in a one-dimensional (1D) space. At a high level, a set of 2D observations of one or more 3D objects can be generated. For example, one or more sensors, such as cameras, lidars, etc., can be mounted on a moving vehicle and used to capture images of one or more fields of view (including vertical landmarks) of one or more sensors while the vehicle is traveling. In some cases, 2D observations (e.g., two-dimensional observations) can be part of 3D observations (e.g., two-dimensional or three-dimensional observations). Camera configuration information can be associated with 2D observations. Object detection (such as edge detection or line detection) can be used to generate indications of directional landmarks (e.g., parameterized, such as lines, polylines, segmentation masks, etc.). In an embodiment of detecting vertical landmarks, each observation can be corrected by applying a transformation to each observation, thereby pushing the vertical vanishing point to infinity. Alternatively, a transformation can be applied to an indication of a detected directional landmark (e.g., line, polyline, segmentation mask, etc.). Thus, for each frame of sensor data, a transformation may be applied based at least in part on the camera configuration information to generate a modified observation.

[0025] The modified observations may include vertical landmarks that have been transformed or translated so that the vertical landmarks extend vertically relative to the modified observations (e.g., extend along one or more columns of pixels of the modified observations). For each modified observation, a corresponding 1D lookup table may be generated to represent the presence of the detected vertical landmarks via a probability distribution that is a function of a single parameter (e.g., a normalized horizontal position in the image). By reducing each 2D observation to a 1D lookup table (e.g., by generating a modified frame), subsequent calculations may no longer need to consider the complete 2D observation (e.g., every pixel of the image, or more precisely, every row of pixels). As a result, significantly less data may need to be loaded, cached, and / or buffered than with conventional methods using 2D lookup tables, thereby saving significant processing and memory.

[0026] In some embodiments, 3D directional landmarks (such as vertical landmarks) can be reconstructed without calculating a complete 3D probability distribution. For example, it can be assumed that a directional landmark can be represented by its cross-section with a 2D plane. For example, in an embodiment of identifying a vertical landmark, the 2D position of the vertical landmark on the ground plane can be used as a representative of the 3D position (for example, because the vertical landmark extends along a known vertical direction, the vertical direction can be separated out (factor out)). Therefore, the 3D reconstruction may be limited to a 2D reconstruction. As a result, a 2D grid can be sampled in 3D space (along the ground plane), and for each 3D point, the probability of the vertical landmark at the point can be calculated based on a 1D lookup table. More specifically, for a given point on the grid, the position of the point can be back-projected into each 1D lookup table to calculate the probability that the vertical landmark is located at the position of the point - for a specified number of 1D lookups (e.g., the N nearest 1D lookups, the N nearest 1D lookups that meet certain baseline criteria, etc.). The probability that the vertical landmark is located at the point position can be specified as the joint probability of different probabilities of the vertical landmark being located at the position (e.g., one for each 1D lookup table). The result may be a 2D probability distribution reflecting the likelihood of a vertical landmark being located at each point on a 2D grid (e.g., along the ground plane). Simplifying a 3D reconstruction into a 2D reconstruction eliminates the need to sample one of the dimensions of the 3D grid, thereby reducing the corresponding amount of computation required compared to using a 3D grid (such as in traditional volume reconstruction).

[0027] From the 2D probability distribution, the location of the vertical landmarks can be determined by finding the distribution pattern. Any conventional technique can be used, such as clustering, pattern finding, optimization, and / or the like. In some examples, an iterative pattern-hill climbing algorithm (such as mean shift) can be used to identify the pattern. In general, to ensure that all patterns are climbing, there may be at least one seed in each basin of attraction in the probability distribution. In some embodiments, it can be assumed that the patterns are separated by a minimum distance d, (and a kernel with infinite support is used) to assist in identifying an upper limit on the number of seeds used, while ensuring that there is at least one seed in each point d. In addition, in the case of vertical landmark reconstruction, if the distance traveled by the vehicle in each 2D observation is less than d, all patterns can be identified by seeding along a line orthogonal to the vehicle displacement (such as along the ground plane). In practice, even if the distance is greater than d, the probability distribution may be smooth and anisotropic (e.g., elongated along the direction of travel) enough to make it possible for the seeds to converge to the corresponding pattern. Therefore, in some examples, the seeds can be initialized at intervals d along a line orthogonal to the vehicle displacement.

[0028] In some embodiments, for each new 2D observation (or a set of observations), a corresponding probability distribution can be estimated in world coordinates, and seeds can be generated to climb to the mode. In some embodiments, mean shift can be further applied to the mode from the previous 2D observation (or a set of observations). Therefore, multiple seeds may converge to the same mode, so that repeated seeds can be pruned. For example, a distance-based pruning scheme can be used to discard duplicates (e.g., within 1 meter). In some embodiments, the range of the reconstructed directional landmark along the direction of the landmark (e.g., the vertical height of the vertical landmark) can be restored by reprojecting the reconstructed directional landmark into the frame of sensor data and identifying the start and end points of the detected directional landmark by comparing with the projected landmark. In some examples, the mode of a persistent group can be maintained (e.g., across time). In such an example, if the detected mode does not yet exist in the persistent group, the mode can be added to the group, and a signal can be sent as a notification that a new vertical landmark is found at a specific location. The location of the vertical landmark can be embedded in the map and provided to the map provider for integration into a map or mapping application. Therefore, vertical landmarks can help autonomous vehicles navigate in the physical environment, and in particular, can help autonomous vehicles localize, thereby achieving more accurate and safer navigation.

[0029] Now refer to Figure 1 , Figure 1 is a block diagram of an example landmark detector 102 according to some embodiments of the present disclosure. It should be understood that this arrangement and other arrangements described herein are only described as examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) can be used to supplement or replace the arrangements and elements shown, and some elements can be omitted entirely. In addition, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, can be implemented with other components, and can be implemented in any appropriate combination and location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory.

[0030] At a high level, the landmark detector 102 can capture the markings of 3D directional landmarks ("directional landmarks") (e.g., 3D vertical landmarks ("vertical landmarks")) and can identify the locations of the directional landmarks from the markings. In general, the landmark detector 102 or a portion thereof can be implemented as a separate device, or can be incorporated with, mounted to (e.g., fixedly or removably) some machine or device, or otherwise associated with some machine or device. For example, the landmark detector 102 or a portion thereof can be mounted to, for example, the following description of the invention. Fig.10The example vehicle described is on or Fig.11 In some examples, the components, features, and / or functions of landmark detector 102 may be used. Fig.12 The present invention may be implemented with any or all of the features, components and / or functionality of the computing device 1200, which are described in more detail herein, or may include additional or alternative components.

[0031] exist Figure 1 In the illustrated embodiment, the landmark detector 100 includes a landmark capture component 110, a correction component 120, a landmark merging component 130, a grid sampler 140, a pattern recognition component 150, a spatial range component 160, and a map integration component 170. This configuration is for illustrative purposes only, and other configurations with similar or different functions may be implemented without departing from the scope of the present disclosure. For example, some components of the landmark detector 100 may be located on another vehicle or device, such as Fig.11 The components can communicate with each other via a network, such as Fig.11 Network 1120 in.

[0032] The marker capture component 110 may include one or more sensors 112. The one or more sensors 112 may include any type of sensor capable of capturing sensor data, including observations of 3D objects (such as, vertical landmarks, etc.). In some embodiments, the one or more sensors 112 may include one or more sensors capable of capturing, deriving, or otherwise generating sensor data representing a frame including 2D observations. For example, the one or more sensors 112 may include one or more cameras capable of generating images (e.g., 2D images), as described below with respect to Fig.10 Any or all of the cameras described. Additionally or alternatively, the one or more sensors 112 may include any combination of one or more global navigation satellite systems (GNSS), one or more radar sensors, one or more ultrasonic sensors, one or more lidar sensors, one or more inertial measurement unit (IMU) sensors, one or more microphones, one or more stereo cameras, one or more wide-angle cameras, one or more infrared cameras, one or more surround cameras, one or more long-range cameras and / or mid-range cameras, and / or other types of sensors. Sensor data from any of these sensors, either alone or in combination, may be used to capture, derive, or otherwise generate frames of 2D observations of 3D objects appearing in a corresponding field of view, such as vertical landmarks.

[0033] The tag capture component 110 may include a control component 114 that operates in communication with one or more sensors 112 to generate and store multiple frames of sensor data over time. The control component 114 may include circuitry and / or logic that controls one or more sensors 112 to capture sensor data and generate a series of frames (e.g., images) of sensor data used as 2D observations. For example, in an embodiment where one or more sensors 112 include a camera mounted on a vehicle, the control component 114 may control the camera to capture a series of images of one or more fields of view of the camera while the vehicle is traveling. Frames (e.g., images) represented by sensor data can be generated at specified times, intervals, on demand, when certain detected events occur, some combination thereof, or otherwise.

[0034] The tag capture component 110 can include a configuration component 116. The configuration component 116 can be configured to determine or otherwise access configuration data, including position and / or orientation data reflecting the position and / or orientation of one or more sensors 112 at a given point in time. The configuration data for the one or more sensors 112 can be determined or otherwise accessed in association with a frame in which the sensor data is generated (e.g., when the sensor data is captured, when the frame is generated, etc.), and the configuration data can be associated with the frame.

[0035] For example, the configuration component 116 may include a location component configured to determine or otherwise access location data reflecting the location of the one or more sensors 112. In embodiments where the one or more sensors 112 are mounted to a vehicle, the location of the vehicle may be determined and used as the location of the one or more sensors 112. The location data may be determined or otherwise accessed based on data generated from one or more sensors, such as one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types, and / or may be determined or otherwise accessed based on any type of positioning technology, such as a global navigation satellite system (GNSS), a cellular network, and / or other wireless positioning technology, or otherwise.

[0036] In addition or alternatively, the configuration component 116 may include a direction component that is configured to determine or otherwise access direction data reflecting the direction of one or more sensors 112 at a given point in time. The direction data can be determined or otherwise accessed based on data generated from one or more sensors (such as one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types). In addition or alternatively, the direction data can be predetermined, for example, based on a known direction. For example, a sensor mounted on a vehicle can be configured with a specific known direction relative to the vehicle. Therefore, the direction of the vehicle can be determined based on the direction of the vehicle and the direction of the one or more sensors 112 relative to the vehicle, as well as the direction of the one or more sensors 112.

[0037] In some embodiments, the configuration data may be reflected with respect to a world coordinate system. For example, the orientation data may be reflected as an orientation matrix (such as a rotation matrix) or some other appropriate representation. In some embodiments, the perspective projection matrix (such as a 4×3 camera projection matrix) may be calculated using configuration data (such as position data and / or orientation data) associated with a frame of sensor data (such as an image). The perspective projection matrix may be used to convert any 3D point in the world space into its position in a frame of reference (e.g., image space, 2D plane, etc.), which corresponds to a frame (e.g., image) represented by sensor data associated with a sensor 112 having a known position and orientation. For example, in an embodiment where one or more sensors 112 include a camera that generates an image, a 4×3 camera projection matrix may be used to convert a 3D point into its 2D position in the image space (e.g., its 2D position in the image). In general, the perspective projection matrix of one or more sensors 112 may be determined or otherwise accessed in association with the generated frame (e.g., when capturing sensor data, when generating a frame, etc.), and the perspective projection matrix may be associated with the frame.

[0038] In some embodiments, the correction component 120 can apply a transformation to correct one or more frames (e.g., each frame) represented by the sensor data to adjust the alignment of objects reflected in the one or more frames. For example, in order to facilitate the detection of directional landmarks (such as vertical landmarks) represented in the frames, the frames can be transformed to align the directional landmarks along a particular direction. Typically, taking an image of a vertical landmark as an example, the vertical landmark may include a utility pole, a sign pole, a straight tree, a light column, etc. If the camera that generates the image of the vertical landmark is pointed downward or upward, the vertical landmark may not appear vertical in the image. Figure 2A-2C Examples of different sensor (eg, camera) configurations and corresponding 2D observations (eg, images) are illustrated.

[0039] For example, in Figure 2A In the generated image 230A, the vertical landmark 220A is in the field of view of the camera 210A, and the camera 210A sees the vertical landmark 220A when looking downward (e.g., at an angle below the horizontal). Due to the downward-looking camera configuration pointing between the two vertical landmarks 240A (e.g., the camera is mounted on a vehicle and pointing in the direction of travel), the two vertical landmarks 240A may point outward in the generated image 230A. Using this camera configuration, the image 230A captured by the camera 210A may have a vertical vanishing point below the image. Figure 2B In the example, the camera 210B is looking horizontally at the vertical landmark 220B. When the horizontal camera configuration is pointed between the two vertical landmarks 220B, the two vertical landmarks 220B appear substantially vertically in the generated image 230C. With this configuration, the image 230B captured by the camera 210B may have a vertical vanishing point at infinity or near infinity. In contrast, in the example Figure 2C In the example, the camera 210C sees the vertical landmark 220C when looking upward (e.g., at an angle above the horizontal). When the upward-looking camera configuration is pointed between the two vertical landmarks 240C, the two vertical landmarks 240C may point inward in the generated image 230C. With this configuration, the image 230C captured by the camera 210C may have a vertical vanishing point above the image.

[0040] Generally speaking, in embodiments involving detecting directional landmarks, the correction component 120 may apply a transformation to correct a frame (e.g., each frame) represented by sensor data so that the directional landmark reflected in the frame is aligned along a specific direction of the frame. Taking an image of a vertical landmark as an example, the correction component 120 may apply a transformation to correct a specific image (e.g., each image) so that the vertical landmark in the image is aligned along the vertical direction of the image. In this example, each image may be corrected by transforming each image based at least in part on the corresponding configuration data of the camera so that the vertical vanishing point is pushed to infinity. In some embodiments, the transformation of the frame may be based on a transformation that adjusts the corresponding sensor direction (e.g., camera matrix) associated with the frame to a point in the horizontal direction, parallel to the ground plane, perpendicular to the direction of gravity, or some other reference direction. The resulting modified frame may reflect the transformed or translated vertical landmark so that the vertical landmark extends vertically relative to the modified frame (e.g., extends along one or more columns of pixels of the modified frame).

[0041] Figure 3 An example of image correction according to some embodiments of the present disclosure is shown. Figure 3In , camera frames 310A, 310B, and 310C reflect images of a series of vertical landmarks (e.g., a camera mounted on a vehicle and pointing in the direction of travel) captured over time. Figure 3 In the example, each camera frame 310A, 310B, and 310C may correspond to a camera pose 320A, 320B, and 320C, respectively. In the example, the correction component 120 may apply a transformation to correct the camera frames 310A, 310B, and 310C based on the camera poses 320A, 320B, and 320C, thereby generating corrected frames 330A, 330B, and 330C, respectively, with vertical landmarks aligned in the vertical direction of the corrected frames. Figure 3 As shown, the vertical landmarks appearing in camera frames 310A, 310B, and 310C are all pointing outward, while the same vertical landmarks appearing in modified frames 330A, 330B, and 330C are substantially vertical.

[0042] Additionally or alternatively, the correction component can identify a transformation that aligns a directional landmark reflected in the frame of sensor data along a particular orientation of the frame and apply the transformation to a corresponding indication (e.g., a line, a polyline, a segmentation mask, etc.) of the directional landmark detected in the frame.

[0043] Back to Figure 1 , the landmark detector 102 may include a label merging component 130. The label merging component 130 may collapse the labels of the directional landmarks into a 1D lookup table that is a function of a single parameter. For example, the label merging component 130 may merge the detected landmarks into a probability distribution that is a function of a single parameter (e.g., horizontal position in the image). Figure 1 In the illustrated embodiment, the tag merging component 130 includes a 1D lookup table generator 134 .

[0044] The object detector 132 can access a frame (e.g., an image), such as a modified frame (e.g., a modified image), and perform object detection to identify directional landmarks represented in the frame, such as vertical landmarks. The object detector 132 can use any suitable detection technology. For example, computer vision, image processing, machine learning, edge or line detection, some combination thereof, or other object detection can be used to perform object detection. In some embodiments, a neural network (e.g., a convolutional neural network (CNN)) can be trained to detect a specific type of landmark (e.g., a directional landmark, such as a vertical landmark). For any given frame (e.g., each modified frame), the object detector 132 can perform object detection to identify directional landmarks in the frame. The result can be an indication of a portion of the frame (e.g., a pixel, an area, a border, etc.), which includes a detected landmark (e.g., a binary indication), the possibility of a detected landmark (e.g., a probability), or other. Taking an image of a vertical landmark as an example, the object detector 132 can output a detection area of ​​the image in which a vertical landmark has been detected, the detection area includes the probability of a detected vertical landmark, the probability of each pixel, the probability of a pixel being part of a vertical landmark, or a combination thereof, or other. For example, when using a CNN, the result or output can be a segmentation mask that includes values ​​(e.g., probability values) associated with pixels of one or more images that may correspond to a target type. In such an example, the CNN can be trained to output a binary segmentation mask that identifies vertical landmarks (e.g., as a first class) and background (e.g., pixels not associated with vertical landmarks, as a second class) or can be trained to output a multi-class segmentation mask that identifies different types of vertical landmarks (such as tree classes, pole classes, etc.) and backgrounds. These are merely examples, and any suitable target detection technique can be applied. In addition, although some examples are described with respect to vertical landmarks, in some embodiments, any directional landmarks, landmark classes, or other desired targets can be detected.

[0045] The 1D lookup table generator 134 can access the signature of the target detected in the frame of sensor data (such as the output from the target detector 132) and simplify the signature into a probability distribution, which is a function of a single parameter (1D probability distribution). The presence of the target detected in the frame can be represented by a 1D lookup table, which returns the corresponding value of the probability distribution. More specifically, the single parameter of the 1D lookup table can correspond to a specific dimension of the frame, and the specific dimension can be perpendicular to the direction of the landmark aligned in the modified frame. Taking the vertical landmark in the modified image as an example, the 1D lookup table can reflect the probability that the vertical landmark is located at a specific horizontal position (e.g., a column of pixels) in the modified image. In an embodiment where the frame of sensor data includes two dimensions (such as an image), by simplifying each 2D frame into a 1D lookup table, the 1D lookup table can be stored and / or used in subsequent calculations instead of the 2D frame. Therefore, subsequent calculations no longer need to consider the entire 2D frame (e.g., each pixel of the image, or more specifically, each row of pixels of the image). Furthermore, detected landmarks can be identified from a 1D lookup table using a single parameter, rather than using a 2D lookup table with two parameters (e.g., the x and y values ​​of a given pixel). As a result, much less data needs to be loaded, cached, or buffered than with traditional methods using a 2D lookup table, thus saving significant processing and memory.

[0046] In some embodiments, for each rectified frame, a corresponding 1D lookup table may be generated from a probability distribution (a function of a single parameter) to represent the presence of detected landmarks (e.g., detected vertical landmarks). In general, vertical landmarks in the rectified image may be represented as lines of a single parameter family (e.g., parameterized by θ) because all lines representing vertical landmarks pass through a vanishing point in the vertical direction. Figure 2A-2C Different parameterizations of θ based on different vertical vanishing point locations are illustrated. For example, in Figure 2A and Figure 2C In , the line through vertical landmarks 240A and 240C passes through the corresponding vanishing points in images 230A and 230C, respectively. In these examples, θ represents the angular position relative to the vanishing point in the image. Figure 2B In , when the vertical vanishing point is at infinity, the parameter θ represents the horizontal position of the image 230B. In some embodiments, the parameterized representation can be normalized. Figure 2B In the parameterized example of , the position range on the frame can be normalized (e.g., [0, 1]). In addition or alternatively, the 1D probability distribution can be discretized into a 1D histogram to facilitate sampling. Therefore, in embodiments where the frame contains directional landmarks and their corresponding vanishing points are approximately at infinity (e.g., the modified frame), the directional landmarks can be parameterized on θ to represent their positions in the frame in a direction perpendicular to the directional landmarks.

[0047] The 1D lookup table generator 134 may generate a 1D probability distribution using any suitable technique. For example, a 1D probability distribution may be generated using kernel density estimation by placing a kernel with a specified variance at the detected vertical landmark positions and summing the kernels. Figure 3 Three example 1D lookup tables 340A, 340B, and 340C are illustrated according to some embodiments of the present disclosure. Figure 3 In the embodiment of the present invention, the modified frames 330A, 330B and 330C include vertical landmarks aligned with the vertical direction of the modified frames. In this example, 1D lookup tables 340A, 340B and 340C are parameterized based on the horizontal positions of the modified frames 330A, 330B and 330C, respectively, and the example kernels have been placed at the positions of the vertical landmarks detected in the modified frames 330A, 330B and 330C in the 1D lookup tables 340A, 340B and 340C. These parameterized representations may not vary with the rows of the image. Therefore, the positions of the vertical landmarks may be identified from other frames of the image or sensor data by performing a 1D lookup in a manner that does not vary with the rows in which the lookup is performed.

[0048] Simplifying some or all of the frames (e.g., images) represented by the sensor data into corresponding 1D lookup tables may save a significant amount of processing and memory. For example, in some embodiments, the 1D lookup table generator 134 may store a specified number of 1D lookup tables in a rolling buffer. As a non-limiting example, by storing the most recent 10 or 50 (or some other number) 1D lookup tables instead of the most recent 10 or 50 images, much less data may need to be stored. Similarly, when the location of a landmark is identified from the 1D lookup table, much less data may need to be loaded and / or cached. Therefore, converting the frames of sensor data so that all landmarks appear in one dimension (e.g., a cross-section) may help save a significant amount of processing and memory.

[0049] Back to Figure 1, the grid sampler 140 of the landmark detector 102 can reconstruct the directional landmarks based on a set of frames of sensor data and / or corresponding 1D lookup tables. More specifically, the grid sampler 140 can use a set of 1D lookup tables corresponding to a set of frames (e.g., the most recent N frames) to perform 3D reconstruction on the directional landmarks appearing in the frames. Typically, a frame of sensor data can reflect 2D observations of 3D directional landmarks. The position and / or probability distribution of the position of the 3D directional landmark can be identified in a 3D coordinate system (e.g., a 3D world space). For example, the world coordinate system can be defined with respect to absolute coordinates (e.g., longitude, latitude, and altitude), relative coordinates (e.g., defined with respect to the position of the landmark detector 102), some combination thereof, or in other ways. As described above, a frame of sensor data can be associated with corresponding configuration data and / or a corresponding perspective projection matrix of one or more sensors 112 used to generate the frame (e.g., stored associatively in a rolling buffer). Thus, the grid sampler 140 may perform 3D reconstruction by sampling a grid of points in a 3D coordinate system, converting the 3D points to corresponding locations in the frame (e.g., their 2D locations on the image), and accessing probabilities from a corresponding 1D lookup table to identify the probability that a landmark passes through a given 3D point.

[0050] In some embodiments, 3D directional landmarks such as vertical landmarks can be reconstructed without calculating a complete 3D probability distribution. More specifically, it can be assumed that the directional landmark can be represented by its cross-section having a 2D plane. For example, in an embodiment where a vertical landmark is identified, the 2D position of the vertical landmark on the ground plane can be considered to represent a 3D position (e.g., since the vertical landmark extends along a known vertical direction, the vertical direction can be separated). Therefore, in some embodiments, the grid sampler 140 can sample a grid of points from a plane in the 3D coordinate system (e.g., the ground plane) and calculate the probability of the landmark passing through each point. Therefore, the 3D reconstruction can be limited to a 2D reconstruction, avoiding sampling of one of the dimensions of the 3D grid, thereby reducing the corresponding amount of computation required compared to using a 3D grid - for example in traditional volume reconstruction.

[0051] For each sampling point on the grid (eg, each cell on the ground plane), the grid sampler 140 may use a 1D lookup table and corresponding configuration data and / or a perspective projection matrix to determine the probability that the directional landmark passes through the point. Figure 4 is an illustration of an example of grid sampling for vertical landmark reconstruction according to some embodiments of the present disclosure. Figure 4, the grid 410 may coincide with the ground plane of the 3D coordinate system. For each cell in the grid 410, the grid sampler 140 may determine the probability that the vertical landmark passes through the cell. For example, for a given cell in the grid 410, the position of the cell in 3D space may be back-projected into each 1D lookup table to identify the probability that the vertical landmark is located at the position of the cell for each of a specified number of 1D lookups (e.g., the N nearest 1D lookups). In some embodiments, the 1D lookup table may represent the position of the detected landmark as a probability. Therefore, the projection into a 1D lookup table may involve looking up and using the probability from the 1D lookup table. In embodiments where the target detection obtains a detection area (such as a straight line, a box, etc.), if the projection falls on the detection area within a specified tolerance range, the corresponding probability associated with the detection area may be specified. These and other techniques for determining probabilities may be implemented.

[0052] Continuing with the above example, the probability that a vertical landmark is at the location of a cell can be assigned a joint probability of different probabilities that the vertical landmark is at that location, one for each 1D lookup table. Taking cell 420 as an example, for each of a set of N corrected frames, a corresponding 1D lookup table can be used to determine the probability that the vertical landmark passes through cell 420, resulting in a set of N probabilities, one for each 1D lookup table. The probability that a vertical landmark passes through cell 420 can be assigned a joint probability of the N probabilities. In some embodiments, the joint probability can be modified to account for false negative detections by adding a bias γ to the individual probabilities, forcing them to be strictly positive. The result of grid sampling can be a 2D probability distribution that reflects the likelihood that a vertical landmark is located at each point on a 2D grid (e.g., along a ground plane).

[0053] In some embodiments, the probability calculation of each point in the grid can be performed in parallel across multiple processors, such as a graphics processing unit (GPU). For example, since the probability calculation of each point may be independent of the probability calculation of other points, the probability calculation of each point can be assigned to different GPUs or different threads of the same GPU to implement a per-grid sampling thread scheme. Therefore, a grid can be generated on the GPU, and the probability of each sample is solved in parallel. For example, for a grid of size 1000*100, 10∧5 problems can be formulated and solved simultaneously. In general, the probability calculation described herein is suitable for GPU acceleration because it has low memory bandwidth requirements. More specifically, in some embodiments, For each frame of sensor data, only a single 1D lookup table and a corresponding transformation matrix (such as a camera projection matrix) need to be uploaded. Therefore, these elements can be uploaded to a rolling buffer, which maintains the most recent N pairs of {1D lookup, conversion}. In some examples, data storage can be reduced to Nx (lookup resolution + camera pose) elements. This embodiment is for example purposes only, and other appropriate changes can be implemented within the scope of the present disclosure.

[0054] The pattern recognition component 150 of the landmark detector 102 ( Figure 1 ) has access to a grid of probabilities that forms a probability distribution indicating the locations of directional landmarks (like a 2D probability distribution), and the locations can be resolved by finding the pattern of the distribution. Figure 5 is an illustration of an example of pattern finding for vertical landmark reconstruction according to some embodiments of the present disclosure. Figure 5 , a 2D probability distribution 510 with modes 520A and 520B is described. Any conventional technique may be used to identify modes 520A and 520B, such as clustering, pattern finding, optimization, and / or the like. In some examples, an iterative pattern-climbing algorithm, such as mean shift, may be used to identify modes 520A and 520B. In general, any suitable pattern finding technique may be implemented.

[0055] In the example, to ensure that all patterns are climbing, the seed component 152 can initialize at least one seed in each attraction domain in the probability distribution. In some embodiments, it can be assumed that the patterns are separated by a minimum distance d, (and a kernel with infinite support is used) to assist in identifying an upper limit on the number of seeds used, while ensuring that there is at least one seed within d at each point. In addition, in the case of vertical landmark reconstruction, if the distance traveled in each frame of sensor data (e.g., a vehicle equipped with a landmark detector 102) is less than d, then seeding only along lines orthogonal to the displacement (e.g., along the ground plane) should identify all patterns. In practice, even if the distance is greater than d, the probability distribution may be smooth and anisotropic (e.g., elongated along the direction of travel) to be sufficient to make it possible for the seeds to converge to the corresponding pattern. Therefore, in some examples, the seed component 152 can initialize the seeds as an iterative pattern climbing algorithm with a spacing of d along lines orthogonal to the displacement of the landmark detector 102. Figure 6 is an illustration of an example of seeding along a line orthogonal to the displacement of a landmark detector according to some embodiments of the present disclosure. Figure 6 In , the landmark detectors travel along the x-direction, and the seeds 610 are initialized along the y-axis at a distance d from each other. This is just one example of seeding, and other variations can be implemented.

[0056] In some embodiments, for each new frame (or a group of frames) represented by the sensor data, a corresponding probability distribution (such as a 2D network) can be estimated, and seeds can be generated to climb to the pattern. In some embodiments, mean shift can be further applied to the pattern from the previous frame (or a group of frames). Therefore, multiple seeds can converge to the same pattern, so that repeated seeds can be pruned. Therefore, the de-duplication component 154 can implement a distance-based pruning scheme, such as by discarding duplicate data (e.g., within 1 meter or some other distance). In some examples, the pattern recognition component 150 can maintain or otherwise access a persistent group pattern (e.g., across time). In such an example, if the detected pattern does not yet exist in the persistent group, the pattern recognition component 150 can add the detected pattern to the group and can trigger the emission of a signal to serve as a notification of the discovery of a new landmark at a specific location.

[0057] In some embodiments, the spatial range component 160 ( Figure 1) can recover the extent of a reconstructed directional landmark along the direction of the landmark (e.g., the vertical height of a vertical landmark). For example, the reconstructed directional landmark can be represented in a 3D coordinate system (e.g., world space) as a line passing through the identified pattern. The spatial extent component 160 can recover the extent of a frame (e.g., an image) of sensor data from which the directional landmark was detected by reprojecting the reconstructed directional landmark (e.g., line) to a reference frame (e.g., image space, 2D plane, etc.). For example, for each frame Fi associated with a detected directional landmark (e.g., each frame in a set of N frames), the spatial extent component 160 can perform one or more of the following operations. The spatial extent component 160 can detect or otherwise identify the start point si and end point ei of the detected directional landmark appearing in the frame by comparison with the reprojected landmark. For each of the identified start points, the spatial extent component 160 can reconstruct a 3D ray passing through one or more sensors 112 (e.g., a camera) and the start point, identify the ray closest to the reconstructed landmark, and identify the corresponding point s on the reconstructed landmark closest to the ray. min For each of the identified endpoints, the spatial range component 160 can reconstruct a 3D ray that passes through one or more sensors 112 (eg, one or more cameras) and the endpoint, identify the ray that is closest to the reconstructed landmark, and identify a corresponding point on the reconstructed landmark that is closest to the ray. min By repeating this process for each frame Fi, the spatial range component 160 can generate a set of possible starting points {s1 min ,s2 min ,…,s N min} and a set of possible endpoints {e1 min ,e2 min ,…,e N min}. The spatial range component 160 can identify representative start and end points from the respective groups, for example, using K-means clustering, some other clustering technique, or other methods.

[0058] In some embodiments, map integration component 170 can embed or otherwise integrate the range of detected position and / or directional landmarks into a map. For example, in embodiments where vertical landmark locations are detected, these locations can be embedded into a map (e.g., a high-definition map), provided to a map provider for such embedding, uploaded to a repository of landmark locations, and / or the like. Thus, vertical landmarks can help autonomous vehicles navigate in a physical environment, and in particular can help autonomous vehicles perform localization, thereby achieving more accurate and safer navigation.

[0059] Now refer to Figure 7, each block of the method 700 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, a processor executing instructions stored in a memory can implement various functions. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a stand-alone application, a service, or a hosted service (stand-alone or in conjunction with another hosted service), or a plug-in to another product, etc. In addition, as an example, regarding Figure 1 The method 700 is described with reference to the landmark detector 102 of FIG. However, the method may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0060] Figure 7 700 for vertical landmark reconstruction according to some embodiments of the present disclosure. At block B702, the method 700 includes receiving a set of observations of vertical landmarks. For example, the landmark capture component 110 can generate a set of images and corresponding camera configuration information, and the landmark capture component 110 can provide the images and the corresponding configuration information to the correction component 120. In some embodiments, the object detector 132 can perform object detection on the image to generate an indication of a directional landmark (e.g., parameterized, such as a line, a polyline, a segmentation mask, etc.), and the indication of the directional landmark (e.g., a segmentation mask) and the corresponding configuration information can be provided to the correction component 120. The above are merely examples, and other examples can also be implemented within the scope of the present disclosure.

[0061] At block B704, method 700 includes revising the observations. For example, the correction component 120 can apply a transformation to the image based at least in part on the camera configuration information to generate a set of revised observations. The transformation can align at least one vertical landmark along a vertical direction of the transformed image. Additionally or alternatively, the transformation can be applied to an indication of a detected directional landmark (e.g., a line, a polyline, a segmentation mask, etc.) to generate a set of revised observations.

[0062] At block B706, method 700 includes generating a first probability distribution from each modified observation. For example, label merging component 130 may generate a first probability distribution for each transformed observation representing the probability that at least one vertical landmark is located in the column of the transformed observation. Each first probability distribution may be a function of a single parameter.

[0063] At block B708, method 700 includes calculating, for each point in the grid, a probability that the vertical landmark location is located at the point based on each first probability distribution. For example, grid sampler 140 may identify, for each point sampled from the ground plane, a probability that the vertical landmark is located at the point. The probability for each point may be based at least in part on the first probability distribution for each transformed observation.

[0064] At block B710, method 700 includes generating a second probability distribution using the probability for each point in the grid. For example, grid sampler 140 may generate a second probability distribution using the probability for each point sampled from the ground plane. The second probability distribution may include, for each point, a joint probability of a set of probabilities (one for each transformed observation) that the vertical landmark is located at that point.

[0065] At block B712, method 700 includes identifying a vertical landmark location based on the second probability distribution. For example, pattern recognition component 150 may identify the location of the vertical landmark by finding a pattern of the second probability distribution.

[0066] At block B714, method 700 includes performing one or more operations using the vertical landmark position. For example, map integration component 170 can embed or otherwise integrate the vertical landmark position into the map. Therefore, the vertical landmark can help the autonomous vehicle navigate in the physical environment, and in particular, can help the autonomous vehicle perform positioning, thereby achieving more accurate and safer navigation.

[0067] Now refer to Figure 8 , each block of the method 800 described herein comprises a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, a processor executing instructions stored in a memory can implement various functions. The method can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by a stand-alone application, a service or a hosted service (stand-alone or in combination with another hosted service), or a plug-in for another product, etc. In addition, as an example, Figure 1 The method 800 is described with reference to the landmark detector 102. However, the method may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0068] Figure 8800 for directional landmark reconstruction according to some embodiments of the present disclosure. At block B802, the method 800 includes correcting the observation to align the directional landmark along the direction of the corrected observation. For example, the observation can be a 3D space, 2D or 3D lidar data, a segmentation mask (whether in a 2D layout, sparse representation or other), or an image of some other representation, and the correction component 120 can correct the observation by transforming each observation to generate a corrected observation that aligns one or more directional landmarks along the direction of the corrected observation.

[0069] At block B804, method 800 includes generating a lookup table representing the position of the directional landmark along the direction of the modified observation. For example, the marker merging component 130 can generate a lookup table for each modified observation in the set of modified observations, which represents the position of one or more directional landmarks along the direction of the modified observation. In some embodiments, each lookup table can be a probability distribution that is a function of a single parameter.

[0070] At block B806, method 800 includes sampling a probability grid representing the location of directional landmarks at points in the grid based on a table lookup. For example, grid sampler 140 may generate a probability distribution by calculating, for each point in the grid sampled from the 3D space, a probability that one of the one or more directional landmarks is located at the point based at least in part on a table lookup for each corrected observation.

[0071] At block B808, the method 800 includes identifying locations of directional landmarks based on the probability grid.For example, the pattern recognition component 150 may identify the locations of one or more directional landmarks in the 3D space by finding patterns in the probability distribution.

[0072] Now refer to Fig. 9 , each block of the method 900 described herein comprises a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, a processor executing instructions stored in a memory can implement various functions. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a stand-alone application, a service or a hosted service (stand-alone or in combination with another hosted service), or a plug-in for another product, etc. In addition, as an example, Figure 1 The method 900 is described with reference to the landmark detector 102 of FIG. However, the method may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0073] Fig. 99 is a flow chart illustrating a method 900 for directional landmark reconstruction according to some embodiments of the present disclosure. At block B902, the method 900 includes correcting a set of 2D observations by aligning directional landmarks along the direction of the corrected 2D observations. For example, the correction component 120 may correct a set of 2D observations by transforming each 2D observation to generate a corrected 2D observation that aligns one or more directional landmarks along the direction of the corrected 2D observations.

[0074] At block B904, method 900 includes generating a 1D lookup table representing the locations of directional landmarks present in the modified 2D observations. For example, label merging component 130 may generate a 1D lookup table as a 1D probability distribution for each modified 2D observation in a set of 2D observations representing one or more directional landmarks present in the modified 2D observations.

[0075] At block B906, method 900 includes generating a probability distribution by calculating, for each point in the grid, a probability that one of the directional landmarks is located at the point based on a 1D table lookup. For example, grid sampler 140 may generate a probability distribution by calculating, for each point in the grid, a probability that one of the one or more directional landmarks is located at the point based on a 1D table lookup associated with a set of 2D observations.

[0076] At block B908, the method 900 includes identifying the location of the directional landmarks based on the probability distribution.For example, the pattern recognition component 150 can identify the location of one or more directional landmarks by finding a pattern in the probability distribution.

[0077] Although the embodiments described herein relate to the detection of vertical landmarks, in some embodiments, any directional landmarks may be detected (e.g., horizontal landmarks, landmarks occurring at an angle such as 45 degrees, etc.). Furthermore, although the embodiments described herein relate to the generation of a 2D grid representing the positions of vertical landmarks as a 2D probability distribution, in some embodiments, any suitable geometry of the distribution may be sampled (e.g., a 3D grid, some irregular 2D or 3D shape, etc.). Furthermore, although the embodiments described herein relate to the generation and use of a 1D lookup table, this need not be the case. More generally, some embodiments may perform dimensionality reduction on a multidimensional framework of sensor data to generate a lookup table of one or more dimensions of collapse markers. These and other variations will be considered in the present disclosure.

[0078] Example Vehicle

[0079] In some embodiments, Figure 1The landmark detector 102 in may be mounted to a moving vehicle and used to capture images of one or more fields of view (including vertical landmarks) of one or more sensors while the vehicle is traveling. Fig.10 1002 are examples of possible camera positions and corresponding fields of view on vehicle 1002, according to some embodiments of the present disclosure. The cameras and respective fields of view are simple examples and are not intended to be limiting. For example, one or more cameras may be used, additional and / or alternative cameras may be included, and / or cameras may be located at different locations on vehicle 1002. In general, vehicle 1002 may include a passenger vehicle, such as a car, truck, bus, and / or other type of vehicle that can accommodate one or more passengers. In some embodiments, the vehicle may be an airborne vehicle (e.g., an airplane, a helicopter, etc.), a marine vehicle (e.g., a ship, a submarine, etc.), a space or vehicle (e.g., a rocket, a satellite, etc.), an autonomous vehicle, a semi-autonomous vehicle, a non-autonomous vehicle, an indoor / outdoor robot of various sizes, or other.

[0080] In general, the landmark detector 102 and / or the vehicle 1002 may include any number of camera types, including one or more stereo cameras 1068, one or more wide-angle cameras 1070, one or more infrared cameras 1072, one or more surround cameras 1074, one or more remote and / or mid-range cameras 1098, and / or other types of cameras. The camera can be used to capture image data of the entire periphery of the vehicle 1002. The type of camera used depends on the embodiment and requirements of the vehicle 1002, and any combination of camera types can be used to provide the necessary coverage around the vehicle 1002. In addition, the number of cameras may vary depending on the embodiment. For example, the vehicle may include 6 cameras, 7 cameras, 10 cameras, 12 cameras, and / or another number of cameras. As an example but not a limitation, these cameras may support low voltage differential signaling (LVDS) and / or Gigabit Ethernet.

[0081] The camera type of the camera may include, but is not limited to, a digital camera that may be suitable for use with the components and / or systems of the vehicle 1002. One or more cameras may operate at an automotive safety integrity level (ASIL) B and / or another ASIL. According to an embodiment, the camera type may have any image capture rate, such as 60 frames per second (fps), 120 frames per second, 240 frames per second, etc. These cameras may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red colorless colorless colorless (RCCC) color filter array, a red colorless colorless blue (RCCB) color filter array, a red blue green colorless (RBGC) color filter array, a FoveonX3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a transparent pixel camera, such as a camera with an RCCC, RCCB, and / or RBGC color filter array, may be used in an effort to improve light sensitivity. Any suitable lens may be used, including lenses with narrow field of view lenses, wide field of view lenses, fisheye lenses, etc.

[0082] In some examples, one or more cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multifunctional single camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more cameras (e.g., all cameras) can record and provide image data (e.g., video) simultaneously.

[0083] One or more cameras may be mounted in a mounting assembly, such as a custom designed (3D printed) assembly, to reduce stray light and reflected light from the car (e.g., reflections from the dashboard reflected into the windshield mirrors) that may interfere with the camera's image data capture capabilities. For the wing mirror mounting assembly, the wing mirror assembly may be custom 3D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, the camera may be integrated into the wing mirror. For the side view cameras, one or more cameras may also be integrated into the four pillars at each corner of the cabin.

[0084] A camera whose field of view includes a portion of the environment in front of the vehicle 1002 (e.g., a front-facing camera) can be used for a surround view to help identify the path and obstacles facing ahead, as well as to assist in providing key information for generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers and / or control SOCs. A front-facing camera can be used to perform many of the same ADAS functions as lidar, including emergency braking, pedestrian detection, and obstacle avoidance. A front-facing camera can also be used for ADAS functions and systems, including lane departure warning ("LDW"), automatic cruise control ("ACC"), and / or other functions, such as traffic sign recognition.

[0085] A variety of cameras can be used in the front-facing configuration, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imager. Another example can be one or more wide-angle cameras 1070, which can be used to sense objects (such as pedestrians, road crossers, or bicycles) entering the field of view from the periphery. Although Fig.10 There is only one wide-angle camera in the vehicle 1002, but there may be any number of wide-angle cameras 1070 on the vehicle 1002. In addition, one or more remote cameras 1098 (e.g., a long-view stereo camera set) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. One or more remote cameras 1098 can also be used for object detection and classification, as well as basic object tracking.

[0086] One or more stereo cameras 1068 may also be included in the front configuration. The one or more stereo cameras 1068 may include an integrated control unit including a scalable processing unit that may provide programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative one or more stereo cameras 1068 may include one or more compact stereo vision sensors (which may include two camera lenses (one on each side)) and an image processing chip that may measure the distance between the vehicle and the target object and use the generated information (e.g., metadata) to activate automatic emergency braking and lane departure warning functions. In addition to or in lieu of those described herein, other types of one or more stereo cameras 1068 may also be used.

[0087] A camera (e.g., a side view camera) whose field of view includes a portion of the side environment of the vehicle 1002 can be used for a view of the surroundings, providing information for creating and updating an occupancy grid and generating a side collision warning. For example, one or more surround cameras 1074 can be positioned on the vehicle 1002. One or more surround cameras 1074 can include one or more wide-angle cameras 1070, one or more fisheye cameras, one or more 360-degree cameras, and / or cameras of the like. For example, four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use one or more three-way surround cameras 1074 (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., a front-facing camera) as a fourth surround view camera.

[0088] A camera whose field of view includes a portion of the environment behind the vehicle 1002 (e.g., a rear view camera) can be used to assist with parking, surround view, rear collision warning, and creating and updating occupancy networks. A wide variety of cameras can be used, including but not limited to cameras that can also be adapted to serve as one or more front-facing cameras (e.g., one or more long-range and / or mid-range cameras 1098, one or more stereo cameras 1068, one or more infrared cameras 1072, etc.).

[0089] Fig.11 is a system diagram of communications between one or more cloud-based servers and example vehicles 1110A through 1110N according to some embodiments of the present disclosure. Fig.11 The vehicle communication system shown may include one or more servers 1130, one or more networks 1120, and any number of vehicles 1110A to 1110N (which may correspond to Fig.10 vehicle 1002).

[0090] Any or all of the vehicles 1110A to 1110N may include one or more network interfaces, which may include one or more wireless antennas (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface may be used to enable wireless connections with the cloud (e.g., using one or more servers 1130 and / or other network devices), with other vehicles (e.g., one or more of the vehicles 1110A to 1110N), and / or with computing devices (e.g., a passenger's client device) via the Internet. In order to communicate with other vehicles, a direct link may be established between the two vehicles and / or an indirect link may be established (e.g., through a network and through the Internet). A vehicle-to-vehicle communication link may be used to provide a direct link. The vehicle-to-vehicle communication link may provide information about vehicles approaching one or more vehicles 1110A to 1110N (e.g., vehicles in front of, to the side of, and / or behind one or more vehicles 1110A to 1110N). This functionality may be part of a collaborative adaptive cruise control function.

[0091] The network interface may include a SOC that provides modem functionality and enables one or more controllers to communicate over a wireless network. The network interface may include a radio frequency front end for upconverting from baseband to radio frequency and downconverting from radio frequency to baseband. The frequency conversion may be performed by known processes or may be performed using a superheterodyne process. In some examples, the radio frequency front end functionality is provided by a separate chip. The network interface may include wireless functionality for communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0092] The network interface may use one or more wireless antennas and / or modems to communicate over one or more networks. For example, the network interface may be capable of communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. One or more wireless antennas may also enable communication between objects (e.g., vehicles, mobile devices, etc.) in the environment using one or more local area networks (e.g., Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc.) and / or one or more low power wide area networks (LPWANs) (e.g., LoRaWAN, SigFox, etc.).

[0093] Any or all of the vehicles 1110A to 1110N may further include one or more data stores, which may include off-chip (e.g., off-chip SOC) storage. The one or more data stores may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices that can store at least one bit of data.

[0094] Any or all vehicles 1110A to 1110N may further include one or more GNSS sensors. One or more GNSS sensors (e.g., GPS and / or assisted GPS sensors) may assist in mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors may be used, including, for example, but not limited to, GPS using a USB connector with an Ethernet to serial (RS-232) bridge.

[0095] One or more servers 1130 may include multiple GPUs, PCIe switches, and / or CPUs. The GPUs, CPUs, and / or PCIe switches may be interconnected using high-speed interconnects, such as, but not limited to, NVLink interfaces and / or PCIe connections developed by NVIDIA. In some examples, the GPUs are connected via NVLink and / or NVSwitch SoCs, and the GPUs and PCIe switches are connected via PCIe interconnects. According to an embodiment, each of the multiple servers may include any number of GPUs, CPUs, and / or PCIe switches.

[0096] One or more servers 1130 may receive the range of the detected position and / or direction landmarks, the map of the range of the detected position and / or direction landmarks embedded or otherwise integrated therein, or other related information from any or all of the vehicles 1110A to 1110N via one or more networks 1120. One or more servers 1130 may send the range of the detected position and / or direction landmarks, the map of the range of the detected position and / or direction landmarks embedded or otherwise integrated therein, or other related information to any or all of the vehicles 1110A to 1110N via one or more networks 1120, and / or send the above information to one or more map providers via one or more networks 1120. In addition or alternatively, any or all of the vehicles 1110A to 1110N may send the range of the detected position and / or direction landmarks, the map of the range of the detected position and / or direction landmarks embedded or otherwise integrated therein, or other related information to one or more map providers via network 1120. These variations and other variations are all considered within the present disclosure.

[0097] Example computing device

[0098] Fig.12 1 is a block diagram of an example computing device 1200 suitable for implementing some examples of the present disclosure. The computing device 1200 may include a bus 1202 that directly or indirectly couples the following devices: a memory 1204, one or more central processing units (CPUs) 1206, one or more graphics processing units (GPUs) 1208, a communication interface 1210, an input / output (I / O) port 1212, an input / output component 1214, a power supply 1216, and one or more presentation components 1218 (e.g., one or more displays).

[0099] Although Fig.12 The various boxes in are shown as being connected via bus 1202, but this is not intended to be limiting and is only for clarity. For example, in some embodiments, presentation component 1218 (e.g., a display device) can be considered as I / O component 1214 (e.g., if the display is a touch screen). As another example, CPU 1206 and / or GPU 1208 may include memory (e.g., memory 1204 may represent a storage device other than the memory of GPU 1208, CPU 1206, and / or other components). In other words, Fig.12 The computing devices described herein are illustrative only. No distinction is made between categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “gaming console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, all of which are considered Fig.12 within the range of computing devices.

[0100] The bus 1202 may represent one or more buses, such as an address bus, a data bus, a control bus, or a combination thereof. The bus 1202 may include one or more bus types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or other types of buses.

[0101] Memory 1204 may include any of a variety of computer-readable media. Computer-readable media may be any available media accessible to computing device 1200. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0102] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storing information (such as computer readable instructions, data structures, program modules, and / or other data types). For example, memory 1204 may store computer readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computer device 1200. Computer storage media, as used herein, does not include signals.

[0103] Communication media may contain computer readable instructions, data structures, program modules and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery media. The term "modulated data signal" may refer to a signal having one or more characteristic groups or varying in a manner with respect to encoding information in the signal. As an example and not limitation, communication media may include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency, infrared media and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.

[0104] One or more CPUs 1206 may be configured to execute computer-readable instructions to control one or more components of the computing device 1200 to perform one or more methods and / or processes described herein. One or more CPUs 1206 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) that are capable of processing a large number of software threads simultaneously. One or more CPUs 1206 may include any type of processor, and may also include different types of processors, depending on the type of computing device 1200 implemented (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 1200, the processor may be an ARM processor implemented using reduced instruction set computing (RISC), or may be an x86 processor implemented using complex instruction set computing (CISC). The computing device 1200 may also include one or more CPUs 1206 in addition to one or more microprocessors or supplementary coprocessors (such as mathematical coprocessors).

[0105] The computing device 1200 may render graphics (such as 3D graphics) using one or more GPUs 1208. The one or more GPUs 1208 may include hundreds or thousands of cores that are capable of processing hundreds or thousands of software threads simultaneously. The one or more GPUs 1208 may generate pixel data for an output image in response to a rendering command (e.g., rendering a command received from one or more CPUs 1206 via a host interface. The one or more GPUs 1208 may include graphics memory, such as display memory, for storing pixel data. The display memory may be included as part of the memory 1204. The one or more GPUs 1208 may include two or more GPUs running in parallel (e.g., by linking). When combined together, each GPU 1208 may generate pixel data for different parts of an output image or different output images (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0106] In examples where computing device 1200 does not include one or more GPUs 1208 , one or more CPUs 1206 may be used to render graphics.

[0107] The communication interface 1210 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1200 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communications). The communication interface 1210 may include components and functionality that enable communication over any of a number of different networks, such as a wireless network (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), a wired network (e.g., communicating via Ethernet), a low-power wide area network (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0108] I / O ports 1212 can logically couple computing device 1200 to other devices, including I / O components 1214, one or more presentation components 1218, and / or other components, some of which can be built into (e.g., integrated into) computing device 1200. Illustrative I / O components 1214 include microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. I / O components 1214 can provide a natural user interface (NUI) that processes air gestures, sounds, or other body inputs generated by a user. In some cases, the input can be transmitted to an appropriate network element for further processing. NUI can implement speech recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 1200 (described in more detail later). Computing device 1200 can include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and a combination of these technologies, for gesture detection and recognition. In addition, computing device 1200 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) to enable detection of motion. In some examples, the output of the accelerometer or gyroscope may be used by computing device 1200 to render immersive augmented reality or virtual reality.

[0109] The power supply 1216 may include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 1216 may provide power to the computing device 1200 to enable the components of the computing device 1200 to operate.

[0110] One or more presentation components 1218 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. One or more presentation components 1218 may receive data from other components (e.g., one or more GPUs 1208, one or more CPUs 1206, etc.) and output data (e.g., as images, video, sound, etc.).

[0111] The present disclosure may be described in the general context of computer code or machine-usable instructions that can be executed by a computer or other machine (such as a personal digital assistant or other handheld device), the instructions including computer executable instructions, such as program modules. Typically, program modules include routines, programs, objects, components, data structures, etc., which refer to codes that perform specific tasks or implement specific abstract data types. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices connected via a communication network.

[0112] As used herein, the statement "and / or" about two or more elements should be interpreted as meaning only one element or a combination of multiple elements. For example, "element A, element B and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B and C. In addition, "at least one of element A or element B" may include at least one element A, at least one element B, or at least one element A and at least one element B. Further, "at least one of element A and element B" may include at least one element A, at least one element B, or at least one element A and at least one element B.

[0113] To satisfy statutory requirements, the subject matter of the present disclosure is described in detail herein. However, the description itself is not intended to limit the scope of the present disclosure. Rather, the inventors contemplate that the claimed subject matter may also be embodied in other ways to include different steps or combinations of steps similar to those described in this document, together with other existing or future technologies. In addition, although the terms "step" and "box" are used herein to refer to different elements of the method used, these terms should not be interpreted as implying any particular order between the various steps disclosed herein unless the order of the various steps is explicitly described.

Claims

1. A method comprising: receiving camera configuration information of a camera and a set of images generated using the camera; generating a set of transformed observations by applying a transformation to data derived from at least one image in the set of images based at least in part on the camera configuration information, the transformation corresponding to alignment of at least one landmark along a direction; generating, for each transformed observation in the set of transformed observations, a first probability distribution representing a probability that the at least one landmark is located in a column of the transformed observation; identifying, for each point sampled from a ground plane, a probability that the at least one landmark is located at the point based at least in part on the first probability distribution for each of the transformed observations; generating a second probability distribution using the probability of each of the points sampled from the ground plane; as well as A location of the at least one landmark is identified based at least in part on the second probability distribution.

2. The method of claim 1, further comprising uploading the first probability distribution for each transformed observation to a rolling buffer, the rolling buffer maintaining corresponding probability distributions associated with a specified number of most recently captured images. 3 . The method of claim 1 , wherein the camera is associated with a vehicle and the set of images is generated while the vehicle is moving.

4. The method of claim 1 , wherein generating the first probability distribution for each transformed observation comprises: identifying the at least one landmark in a single dimension of the transformed observation; as well as The first probability distribution is generated by placing a kernel at the location of the at least one landmark in the single dimension and summing the kernel.

5. The method of claim 1, further comprising discretizing the first probability distribution of each transformed observation into a one-dimensional histogram. 6 . The method of claim 1 , further comprising generating the second probability distribution on a GPU by solving in parallel the probability that the at least one landmark is located at each point sampled from the ground plane.

7. The method according to claim 1, wherein: The identifying the position of the at least one landmark includes identifying a pattern in the second probability distribution by seeding at specified intervals along a line orthogonal to the displacement of the camera.

8. A method comprising: modifying the set of observations to generate a set of modified observations by transforming, for each observation in the set of observations of the 3D space, the observations to generate modified observations, the modification aligning one or more directional landmarks along a direction of the modified observations; generating, for each of the set of corrected observations, a lookup table representing the position of the one or more directional landmarks along the direction of the corrected observation; generating a probability distribution by calculating, for each point in a grid sampled from the 3D space, a probability that one of the one or more directional landmarks is located at the point based at least in part on a table lookup of each of the modified observations; as well as Positions of the one or more directional landmarks in the 3D space are identified based on the probability distribution.

9. The method of claim 8, further comprising uploading the lookup table associated with the set of observations to a rolling buffer, the rolling buffer maintaining a set of lookup tables associated with a specified number of most recently captured observations.

10. The method of claim 8, wherein generating the lookup table representing the position of one or more directional landmarks along the corrected observed direction comprises: detecting the one or more directional landmarks in a dimension orthogonal to the direction of the corrected observation; as well as The lookup table is generated as a ID probability distribution by placing kernels at the detected positions of one or more directional landmarks in the dimension and summing the kernels.

11. The method of claim 8, further comprising discretizing each lookup table into a one-dimensional histogram.

12. The method of claim 8, wherein for each point in the grid, calculating the probability that the one or more directional landmarks are located at the point comprises: back-projecting the point into each look-up table using corresponding sensor configuration information for the look-up table; generating a set of probabilities by calculating, for each lookup table associated with the set of observations, a probability that the point coincides with a directional landmark represented by the lookup table; and The probability that one of the directional landmarks is located at the point is assigned as the joint probability of the set of probabilities.

13. The method of claim 8, further comprising generating the probability distribution on a GPU by solving in parallel the probability that one of the one or more directional landmarks is located at each point in a grid.

14. The method of claim 8, wherein identifying the locations of the one or more directional landmarks comprises: Patterns in the probability distribution are identified by seeding at specified intervals along lines orthogonal to the displacements of the sensors associated with the set of observations.

15. A method comprising: Modifying a set of two-dimensional (2D) observations by transforming each 2D observation in the set of 2D observations to generate a modified 2D observation, the modification aligning one or more directional landmarks along a direction of the modified 2D observation; generating, for each modified 2D observation in the set of 2D observations, a one-dimensional (1D) lookup table representing locations where one or more directional landmarks appear in the modified 2D observation as a 1D probability distribution; generating a probability distribution by computing, for each point in the grid, a probability that one of the one or more directional landmarks is located at the point based on the 1D lookup table associated with the set of 2D observations; and The locations of the one or more directional landmarks are identified based on the probability distribution.

16. The method of claim 15, further comprising maintaining in a rolling buffer a set of ID lookup tables associated with a specified number of most recently captured 2D observations.

17. The method according to claim 15, wherein: The set of 2D observations is generated using one or more sensors mounted on the vehicle.

18. The method of claim 15, further comprising discretizing the 1D probability distribution corresponding to each 1D lookup table into a 1D histogram.

19. The method of claim 15, further comprising generating the probability distribution on a GPU by solving in parallel the probability that one of the one or more directional landmarks is located at each point in a grid.

20. The method of claim 15, wherein identifying the locations of the one or more directional landmarks comprises: Patterns in the probability distribution are identified by seeding at specified intervals along a line orthogonal to the displacements of the sensor associated with the set of 2D observations.

Images