Global motion modeling of automobile image data

Abstract

Certain aspects of the present disclosure provide techniques for global motion modeling. Embodiments include receiving a first image and a second image from a camera attached to a moving object. Embodiments include identifying pixels in a first image. Embodiments include determining, based on one or more parameters associated with the camera, a vector representing a range of positions in which a real-world point corresponding to the pixel is likely to be found in the second image, wherein the parameters associated with the camera include: a first parameter related to a position of the camera relative to a ground surface; a second parameter related to the movement of the moving object; and a third parameter related to the orientation of the camera relative to the ground surface. Embodiments include using the vector to determine a location of the real-world point in the second image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 17 / 929,242 filed on September 1, 2022, which is hereby incorporated by reference into this document. Background Art

[0003] Aspects of the present disclosure relate to object detection and tracking based on image data captured from a non-stationary source, such as a car.

[0004] The estimation of optical flow between two video frames is a fundamental component of video processing. Optical flow refers to the pattern of apparent motion of physical components (e.g., objects, surfaces, and edges) in a visual scene caused by the relative motion between the observer and the scene. Optical flow can be used in various applications such as frame rate conversion in displays, video compression, assisted video scene understanding, and other types of applications. The motion of pixels is usually caused by the motion of objects in the scene or the motion of the camera recording the scene. The latter is called global motion, while the former is called local motion. Understanding the motion caused by camera movement is an important part of estimating optical flow. Global motion estimation can improve the robustness of the entire estimation process and reduce the complexity when accurately estimating the model.

[0005] The automotive use case is characterized by significant high-speed motion and often involves the use of wide-angle lenses to generate views around the vehicle. Many conventional global motion models have limited effectiveness when dealing with the unique aspects of video captured from cameras on moving vehicles. In addition to the limitations of the motion model, the estimation of the parameters of the model may be unreliable, such as due to low light, weather conditions, shadows, limited textures, noise, etc. Therefore, pure image-based motion analysis may be unreliable in the automotive context. Recent efforts to apply machine learning to global motion problems in automotive contexts have shown promise, but significant complexity is generated when various camera orientations and lens distortions (each requiring a different model) must be considered. Another challenge in automotive applications comes from limited ground truth training data, because methods for recording high-accuracy, high-resolution motion information from real outdoor scenes are limited.

[0006]

[0006] Therefore, there is a need for improved techniques for understanding image data captured from a mobile source, such as a vehicle. Summary of the invention

[0007] Certain aspects provide a method comprising: receiving a first image and a second image from a camera attached to a moving object; identifying a pixel in the first image; determining a vector representing a range of locations where a real-world point corresponding to the pixel may be found in the second image based on one or more parameters associated with the camera, wherein the parameters associated with the camera include: a first parameter related to the position of the camera relative to a ground surface; and a second parameter related to the motion of the moving object; and using the vector to determine the location of the real-world point in the second image.

[0008] Other aspects provide: a processing system configured to perform the aforementioned methods and those described herein; a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods and those described herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the aforementioned methods and those further described herein; and a processing system comprising components for performing the aforementioned methods and those further described herein.

[0009] The following description and associated drawings set forth in detail certain illustrative features of the one or more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings depict certain aspects of the one or more aspects and therefore are not to be considered limiting of the scope of the present disclosure.

[0011] Figure 1A Describes aspects of the camera that are relevant to global motion modeling.

[0012] Figure 1B -D depicts example images captured by a camera relevant to global motion modeling.

[0013] Figure 2 Depicted are examples of graphs related to global motion modeling.

[0014] FIG. 3A to FIG. 3D Depicts an example related to computing optical flow.

[0015] 4A to 4D An example flow field is depicted.

[0016] Figure 5A and Figure 5B Depicted is an example relating to a vehicle moving between points on a circle when a camera located in front of the vehicle has a parallel displacement.

[0017] Figure 6 Depicted are examples of graphs related to global motion modeling.

[0018] Figure 7 An example fusion operation is depicted.

[0019] Figure 8 An example of a cost penalty is depicted.

[0020] Fig. 9 An example related to searching for candidates is depicted.

[0021] Fig.10 An example related to warping is depicted.

[0022] FIG. 11A to FIG. 11E is an illustration of performing a one-dimensional search at a pixel location in a particular image according to the techniques described herein.

[0023] Fig.12 Depicted are examples of operations related to global motion modeling.

[0024] Fig.13 Depicted are devices adapted to perform operations for the techniques disclosed herein (such as Fig.12 An example of a processing system that performs the operations depicted and described herein.

[0025] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one aspect may be beneficially incorporated in other aspects without further recitation. DETAILED DESCRIPTION

[0026] Aspects of the present disclosure provide apparatus, methods, processing systems, and non-transitory computer-readable media for global motion modeling.

[0027] There are multiple challenges associated with efficiently generating optical flow in video sequences captured from moving objects (such as automotive applications) where the cameras have different orientations and positions as well as various types of wide-angle lens distortions. Efficiency issues can lead to computational constraints for conventional methods as well as limiting the network size for neural network-based methods. One challenge is how to exploit the constraints of the automotive environment while providing support for fast motion and various intrinsic and non-intrinsic camera capabilities within the bounds of the computational needs of optical flow calculations.

[0028] Embodiments of the present invention relate to the use of motion models developed for unique aspects of automotive applications, as described in more detail below with respect to the accompanying drawings. For example, the model described herein includes limited degrees of freedom of motion, support for various camera orientations and lens distortion characteristics, inertial measurement units accurately describing the potential usability of vehicles and therefore camera motion under challenging lighting and imaging conditions. The motion model described herein can be used to derive a dense global motion field. The field can be used to enhance or assist optical flow calculations in several ways. The first method is fusion, in which a dense global motion field is combined with an alternative optical flow estimate by a classical method or a machine learning-based method to provide an improved optical flow field result. The second method is to use a global motion field to provide search candidates to a dense flow estimation algorithm, which results in reduced complexity, such as by specifying a search center. The third method is to pre-warp the image to remove the effects of global motion before applying an alternative motion estimation method. This may be particularly useful for reducing the complexity of machine learning-based methods because the camera orientation and motion of static objects caused by camera movement are removed before the operation of the machine learning method. The fourth method is to guide the search along a one-dimensional or substantially one-dimensional search area, rather than using a more complex two-dimensional search area. For example, constrained search or modified penalty terms can be used to account for known one-dimensional motion of static world points.

[0029] In a specific example, a pixel corresponding to a real-world point is identified in a first image captured by a camera associated with a moving vehicle, and the position of the real-world point in a second image captured by the camera after the first image is determined using the techniques described herein. According to certain embodiments, a known parameter associated with the camera is used to constrain the search for a specific point in the second image. For example, the height of the camera from the ground can be used to project a plane onto the first image at a height associated with the real-world point, and a vector of the real-world point can be determined based on the plane. For example, the vector can be positioned along the plane and can represent a one-dimensional direction in which the real-world point is expected to travel in a subsequent image. The magnitude of the vector can be based on the speed at which the vehicle is moving (e.g., determined based on an accelerometer associated with the vehicle). The vector can be used to constrain the search for pixels corresponding to the real-world point in the second image, such as by limiting the search to pixels that fall along or close to the vector, by penalizing candidate pixels that do not fall along or close to the vector, using the vector as a starting point for the search, comparing the results of alternative search techniques with the vector as a confidence check, and the like.

[0030] In addition, various techniques can be used to account for distortion in the image caused by the parameters of the camera, such as lens distortion from a wide angle lens. For example, one or more formulas can be applied based on the known parameters of the camera to reduce or remove distortion from the image before analysis.

[0031] Embodiments of the present disclosure overcome many challenges associated with conventional techniques. For example, by intelligently focusing the search for points in a series of images from a moving camera based on the expected movement vector of the point, the techniques described herein reduce the amount of computing resources required to perform the search and produce more accurate results. Even under poor environmental and / or lighting conditions, embodiments of the present disclosure may allow flow motion to be determined by intelligent prediction of pixel movement. By utilizing known information about the camera and its movement, the techniques described herein improve the efficiency and accuracy of the flow motion model based on readily available data. For example, by projecting a plane into an image based on the known height of the camera from the ground and generating a vector for a given point based on the plane and the speed at which the camera is moving, the search space can be reduced or focused on a subset of pixels that are most likely to correspond to a given point. In addition, by applying one or more formulas to account for radial distortion produced by a wide-angle lens, the techniques described herein allow flow motion to be accurately determined even for distorted images produced by irregular camera lenses.

[0032] Motion Model

[0033] Figure 1A Camera 110 is depicted at height 104 from plane 102 corresponding to the ground and pointing in direction 106 .

[0034] In vehicle applications, the translational motion of the camera may be significant and may dominate the flow field. General translational motion is considered, rather than assuming that the translation is zero or that the depth of the scene is constant, as required by the projection model to give a global constant. In addition, when deriving the motion model, the height 104 above the ground plane 102 is considered fixed, rather than assuming that the depth in the scene is constant.

[0035] The model assumes that the camera 110 is attached to the vehicle at a fixed pitch and yaw orientation angle relative to the vehicle. The vehicle moves in a plane along a 1-D curve. Note that the minimum turning radius and maximum yaw rate at high speeds constrain the motion. The yaw of the camera 110 is fixed relative to the vehicle, but can change relative to the real world as the yaw of the vehicle changes.

[0036] The form of the motion model in the pixel domain at the pixel of row r and column c may have the following form shown in Equation 1.

[0037] The pixel-domain motion model is in the form of Equation 1:

[0038]

[0039] The image width in pixels is W and the number of horizontal rows is r horiz, and the focal length of the camera in pixels is f. The horizontal line describes the pitch of the camera. For example, if the camera is level, the horizontal line will correspond to the middle of the image.

[0040] The values ​​of the parameters α and β depend on the vehicle's kinematic dynamics, combining motion and external camera orientation. Three-dimensional (3D) world motion is described with only two degrees of freedom, velocity and yaw rate. Considering the projection under different camera pitch and yaw orientations and the spatial variations caused by the projection of the spatial distribution of motion is particularly complex.

[0041] Figure 1B illustrates an image 100B captured by a camera such as camera 110 in a first orientation (eg, a center view), and Figure 1C An image 100C captured by a camera, such as camera 110 , in a second orientation (eg, a left side view) is illustrated.

[0042] exist Figure 1B , the front view pixels below the horizon move downward (e.g., in a series of consecutive images as the camera moves), as shown by arrow 118. The pixels on the right side of the image move to the right, as shown by arrow 120. The pixels on the left side of the image move to the left, as shown by arrow 116.

[0043] exist Figure 1C , illustrating an image captured by a left-side view camera, the pixels on the left side generally move upward (e.g., in a series of consecutive images as the camera moves), as shown by arrow 132. The pixels on the right side of the image move primarily to the left, as shown by arrow 130.

[0044] Given the known properties of the projective transformation that maps lines to lines, the direction of the pixel-domain flow is independent of the height above the ground or the velocity of the camera. This property can be used to simplify a two-dimensional (2D) search to a one-dimensional (1D) or essentially 1D search along the flow direction.

[0045] Figure 1D An image 100C captured by a camera such as camera 110 is illustrated (e.g., from Figure 1C same orientation as image 100C).

[0046] Explain lens distortion

[0047] For an image captured by a camera, the direction from the optical axis of the camera corresponds to the distance from the center of the image. This can be modified slightly in cases where the optical axis does not correspond exactly to the pixel at the center of the image. For example, for a pinhole camera, the relationship is given by a tangent function that relates the camera focal length, the distance from the center, and the angle from the optical axis.

[0048]

[0049] The focal length in pixels can be determined from the camera's viewpoint:

[0050]

[0051] For wide-angle cameras that produce images with radial distortion, the viewing angle θ is still related to the radial distance r, but a more general relationship can be applied. For example, a polynomial can describe the ratio between viewing angle and radius.

[0052] For three different pinhole cameras with two different field of view (FOV) parameters and a third wide-angle camera that can capture images greater than 90 degrees (e.g., "looking behind"), Figure 2 The graph 200 shown in compares the relationship between a first parameter represented by axis 210 comprising an angle in degrees from the optical axis and a second parameter represented by axis 220 comprising a distance rho from the center of the image.

[0053] An analytical solution that supports various radial lens distortions is not particularly useful. Instead, it is useful to use a lens distortion model to compute optical flow to transform between the warped image and spatial coordinates.

[0054] Figure 3A is an example 300A of initialization of a method for converting between a warped image and spatial coordinates. Figure 3A , the orientation of the camera 320 is calculated and defined Three vectors of is the camera normal, point vertically downward, and Points are along the horizontal direction. Vector The collection is then rotated according to the orientation of the camera 320, giving the vector

[0055] The radial distortion model is used to define the relationship between rho and the angle θ from the optical axis and the relationship between cos(θ) and rho: RhoToTheta and CosThetaToRho.

[0056] Figure 3B , Figure 3C and Figure 3D is used to calculate the translation displacement of camera 320 at height h 300B, 300C and 300D illustrate the process of optical flow at an image point (r0, c0) caused by moving on a vehicle with a yaw change Δθ.

[0057] Calculate the translation displacement of the camera due to height h The optical flow at the image point (r0, c0) caused by the vehicle moving with a yaw change Δθ may include the following steps.

[0058] The first step may be to convert the image pixel position (r0, c0) to a 3D space point in the driving plane using the back-projected line intersection with the plane of height zero like Figure 3B shown in . Specifically, Figure 3B This example shows the conversion of camera pixel (r0, c0) to 3D point The inverse projection of .

[0059] The second step can be based on the displacement of the vehicle Apply motion in 3D to move points Move to point like Figure 3C shown. Specifically, Figure 3C The motion of a 3D point due to a displacement vector is illustrated.

[0060] The third step can be based on arrive The vehicle yaw changes Δθ to rotate the camera base.

[0061] The fourth step can be to use the rotation basis The projection will Transform to a different pixel location (r1, c1), such as Figure 3D shown. Specifically, Figure 3D Example of using the slave point Projection of the rotated basis to pixel (r1, c1).

[0062] The fifth step may be to calculate the pixel-domain optical flow as the difference of pixel positions (r1, c1)-(r0, c0).

[0063] The specific steps may include the following steps.

[0064] Given the motion parameters velocity and yaw rate, use the camera frame rate to calculate the displacement due to the change in velocity and orientation Δθ according to yaw_rate and the time between frames (i.e. 1 / frame rate) Base Rotate to the new orientation due to this additional rotation of the camera The optical flow at a pixel in an image is calculated by the following steps.

[0065] Given a pixel row and column (r0, c0), calculate the distance rho in pixels from the center of the image and the corresponding angle α representing the pixel (r0, c0) in polar coordinates (rho, α), where the origin is at the center of the image. Use the radial distortion model to calculate the angle θ from the camera optical axis.axis ,θ.

[0066] A line in space is constructed based on the position of the camera, and points in a direction given by the angle θ with the optical axis and the polar angle α.

[0067]

[0068] The location where this line intersects the plane at the camera height is then determined to determine the constant k. If k < 0, the point is above the horizon and k is set to a large value, such as 10,000.

[0069] The point in the drive plane corresponding to a pixel is given by: Move the point in 3D space by the displacement calculated from the motion parameters to give

[0070] Using Rotation Basis Project the modified point back to the image pixel. Specifically, at the origin and the modified point Calculate between Normalized vector. The angle between the line and the camera optical axis is calculated by the inner product of the camera normal. and The cosine of is calculated. The specific step also involves calculating the modified value of rho1 from this angle via CosThetaToRho. The line and the rotation of the camera vector are also calculated and The polar angle between and These steps also include calculating the coordinates of the image point relative to the origin at the center of the image, u1=rho1*cos(α1) and v1=rho1*sin(α1). The modified rows and columns r1, c1 are also calculated by adding an offset to change the origin to the top left of the image instead of the middle. Finally, the optical flow is calculated as the difference (r1, c1)-(r0, c0).

[0071] Basic flow implementation

[0072] In general, the computation of the flow model described herein involves using the camera and motion parameters to determine the optical flow at each pixel of the image. The computational approximation set forth below involves computing the base flow corresponding to each degree of freedom of motion (there are two degrees of freedom of motion in the context of the car, driving / speed and turning / yaw rate). Given the base flow of two parameters speed and yaw_rate, the motion from point A to point C can be determined by changing the first parameter speed to get an intermediate point B and then changing the second parameter yaw_rate to move to the final point C. The relationship between these three points and the corresponding flows is given by the following equation:

[0073]

[0074] For continuous flow, the rotational flow due to yaw_rate is roughly constant:

[0075]

[0076] For a pinhole camera, the lines are preserved and the result is that the flow at the pixel is proportional to the parameter size.

[0077] For lenses with radial distortion, this property may not hold exactly, but only approximately.

[0078]

[0079] Combining these relations, the flow can be approximated as:

[0080]

[0081] Therefore, the following equation can be used to determine the total flow based on the basis and parameters:

[0082]

[0083] In the above relationship, the total flow at pixel position A is given by a weighted combination of two elementary flows with weights depending on the motion parameters speed and yaw_rate. In this formula, the elementary flows need to be calculated only once for a video sequence.

[0084] Figure 4A , Figure 4B , Figure 4C and Figure 4D Example basic flow fields 400A, 400B, 400C, and 400D are depicted. 4A to 4D Each of depicts an example of using base flow images to generate a general flow.

[0085] exist Figure 4A In , the vehicle hardly turns (e.g., the basic flow corresponds only to the speed), while in Figure 4B In , the vehicle performs a right turn (eg, the base flow corresponds only to the turn). In each case, the total flow is calculated by the weighted sum of the previously shown base flows.

[0086] exist Figure 4C In , the base flow corresponds to a velocity of 9.522 m / s and a yaw rate of 0.04 degrees / s. Figure 4D The base flow corresponds to a velocity of 4.775 m / s and a yaw rate of -27.72 degrees / s.

[0087] A pure "rotation-only" second basis function is not achievable by a vehicle with a positive minimum turning radius. This basis flow can be replaced by an achievable flow if desired. When estimating motion parameters, a minimum turning radius constraint can be imposed.

[0088] Using the base flow field, a general flow field for a given velocity and yaw rate is calculated with low complexity, including two multiplications and one addition for each component of each pixel. Therefore, a total of four multiplications and two additions are required for each image pixel. This method requires pre-computation of the base flow image and storage to maintain the base flow image. The flow can be calculated at a reduced resolution and then upsampled as needed by the application, such as to reduce storage and computation requirements.

[0089] Two basis functions can be used since the motion has primarily two degrees of freedom. The method can be generalized to have three or more basis flows corresponding to the additional motions studied.

[0090] For clarity, these examples are shown without radial distortion, but the same techniques can be used when computing the base flow image based on radial distortion.

[0091] Determine model parameters

[0092] There may be situations where the camera intrinsic and extrinsic parameters are known, but the dynamic motion of the camera is unknown or not used. In some cases, an inertial measurement unit (IMU) can provide motion information. In analytical models, velocity and yaw rate determine the motion parameters analytically. Model calculations that support distorted lenses rely on displacements rather than direct velocity and yaw rate values. In calculating these offsets, velocity and yaw rate can be used to define a circular path with constant speed. The camera frame rate defines the time interval and position, and angular displacement can be calculated.

[0093] Some embodiments assume that the camera is located at the center of the vehicle's motion. In practice, the camera may be slightly translated from the origin for the vehicle (e.g., a side mirror or the like).

[0094] Figure 5A is an illustration 500A of a vehicle 510 moving between points on a circle 502 when a camera located in front of the vehicle has a parallel displacement. In addition, the vehicle rotates, which can be applied after the translation when analyzing the motion path. This rotation does not move the camera located at the center of rotation of the vehicle, but introduces an offset due to the rotating vehicle and the camera having the offset The additional translation can be calculated as the difference between the rotation relative to the camera position and the camera position. In some embodiments, the motion, speed, yaw rate and camera offset can be expressed relative to the standard ISO 8855, such as Figure 5B As depicted in .

[0095] For example, Figure 5B 500B is an illustration of ISO 8855 coordinates for a vehicle 510. Utilizing ISO 8855 coordinates may allow technology to be generalized for collaboration between different vendors / organizations.

[0096] The reliance on IMU data can be eliminated while still using the underlying motion model. For example, this can involve estimating the motion parameters of the current frame based on results from local flow calculations or previous frames. Due to the inertia of the vehicle, the motion parameters will be similar between frames. Additionally, the estimates can be filtered in time to remove outliers and prevent rapid changes.

[0097] The camera parameters are known and are used to define the motion model. However, there are two unknown factors: velocity and rotation rate. Thus, the estimation process computes two base images as described above. A spatial region of the images is selected and samples of the motion vector components are collected. A typical use involves a rectangular region in front of the car, for a forward-facing camera. The samples used for parameter estimation can be limited by a confidence metric so that only high confidence values ​​(e.g., values ​​for which the confidence metric exceeds a threshold) are used in the estimation process.

[0098] The collected samples at pixel position p and the flow vector with two components can be combined into a linear least squares problem. Ideally, the flow at each pixel will be related to the base flow at the same pixel position through a linear relationship. This can be organized into a linear estimation problem by the following equation, where the subscripts h and v represent the horizontal and vertical components of the measured flow F and the base flow Basis0Basis1. The vector p corresponds to the point where the sample was collected. The variable x0 corresponds to speed and x1 corresponds to yaw_rate. Therefore, the estimation problem can be formulated as follows:

[0099]

[0100] This is the form of a known matrix inversion problem.

[0101]

[0102] Given and M, for The well-known solution MSE solution is:

[0103]

[0104] The minimum MSE estimator is determined by selecting the samples to be used, the sample values, and the underlying flow values ​​at the sample points.

[0105] Estimating motion parameters from local pixel values

[0106] The motion parameters of the motion model are estimated based on comparing pixel values ​​rather than relying on previous flow calculations. The input to the estimation includes the current and previous images I n and two basis streams Basis0 and Basis1, as described above. These steps may include selecting a spatial region R for estimating motion parameters and selecting a range of possible motion parameters, i.e., (speed, yaw rate )∈[S0,S1]×[Y0,Y1]. This may be restricted to valid combinations in some embodiments to reduce computation. The motion parameter vector and the underlying flow can be used to calculate the pixel position as explained below The motion parameter vector at The induced flow is determined by the formula.

[0107]

[0108] For motion parameters For each motion vector , the step may include calculating a cost function by summing the error between the value of the previous frame at p resulting from the flow defined by the parameters and the value of the current frame at that pixel in the region R, as explained below for calculating the cost function due to the motion parameter vector The formula for the cost incurred is shown below.

[0109]

[0110] The steps may also include selecting the motion parameter vector giving the minimum cost.The motion parameter vector may optionally be restricted to a valid range as described above.

[0111] Estimating motion parameters from local pixel values

[0112] Several physical constraints can be used to limit the valid region of motion parameters. The following are examples of additional constraints that can be imposed on the initial estimates: minimum turning radius; maximum acceleration due to turning; maximum speed (e.g., different forward and backward); finite difference from previous speed / yaw rate values; etc.

[0113] It is possible that the determined parameter values ​​are not feasible because one or more known constraints are violated. For example, real-world vehicles typically have a minimum turning radius and therefore cannot have a positive yaw rate and zero speed. Therefore, the yaw rate will be limited at low speeds. Similarly, at high speeds, fast turns will give unacceptable acceleration. Therefore, the yaw rate will be limited as speed increases. These constraints can be imposed by processing the direct MSE parameter estimates to constrain them to a feasible range.

[0114] As an example of a constraint, consider the minimum turning radius, which is the tightest circle a vehicle can turn. The motion becomes:

[0115]

[0116] The maximum acceleration due to turning can be limited. For motion in a circle of radius R with speed S, the acceleration is S^2 / R. If an upper limit is given, a relationship can be derived for speed when turning on a circle determined by the yaw rate.

[0117]

[0118] Identify two families of constraints on velocity:

[0119]

[0120] These constraints are Figure 6 , which illustrates a valid set of speed and yaw rate combinations. When estimating motion parameters, these constraints can be applied to improve the estimate or limit the search range. Given a value of speed and yaw_rate, the constraint is imposed by mapping a pair (speed, yaw_rate) outside the constraint to the same speed but smaller |yaw_rate| that satisfies the inequality.

[0121] Additionally, the parameters do not change abruptly over time. A temporal median filter can be used to remove outliers, while a smoothing filter can be used to limit the rate of change. In the case of a parameter search, the search can be centered on the parameters from the previous frame with a limited search range.

[0122] Information from other cameras may also be used. In typical operation, the forward-facing camera may be used to estimate motion parameters. These parameters may be used when processing other camera views. Additional cameras such as rear-view cameras may be used to help form parameter estimates, such as when the forward-facing estimate is invalid or has low confidence. An example of this is when the vehicle is stopped but pedestrians and local motion are such that the estimated flow would suggest invalid motion when relying only on the forward-facing camera.

[0123] The motion model described herein is valuable in providing global motion estimation suitable for the automotive domain. A description of the model and parameter estimation is discussed above. In the following sections, certain applications of the model and parameter estimation are discussed.

[0124] Optical flow fusion

[0125] The fusion application uses the global motion field from the motion model to modify the separate local flow estimation algorithms. In many cases, such as low light, shadows, small textures, etc., local motion estimation may not be accurate, and global motion flow will not be accurate when local motion is present, but may be fairly accurate even in low light when using the IMU to determine motion parameters. Figure 7 An overview of this operation is shown in .

[0126] Figure 7 An example fusion operation is illustrated.

[0127] At step 702, camera motion is estimated. At step 704, a motion field is generated.

[0128] At step 712, an image pair is determined, and at step 714, a local flow is estimated based on the image pair.

[0129] At step 706, motion fusion is performed based on the local flows and the motion field.

[0130] At step 720, pixel motion is determined based on motion fusion.

[0131] Various fusion methods can be used, and only some examples are described herein. The motion model assumes that the ground altitude is zero. For objects with different altitudes, the local flows will have the same direction but different magnitudes, and this fact can be exploited when designing the fusion algorithm.

[0132] For fusion, it can be assumed that local flow and motion model flow are defined at each pixel. A cost function is defined for the flow vectors, and the lower cost vector is selected for fusion. In this discussion, different methods will differ in how the cost function is defined.

[0133]

[0134] For values ​​of N, such as N=3, N=8, for a given vector v, the block SAD at each point p is defined as:

[0135]

[0136] Additive Penalty

[0137] Figure 8is an illustration 800 of an example of a cost penalty. The cost of a flow vector is the sum of the SAD and a penalty term. For local motion, the penalty term is based on the directional difference between the motion model (p) and the measured vector v. For vectors in the same direction, which may differ in magnitude from the motion model, the penalty term may be negative for local motion. In this case, local motion in the same direction as the motion model is preferred. Local motion that is not in the same direction as the motion model has a penalty, but may still be selected in the fusion if the local motion gives enough SAD cost savings to offset the penalty.

[0138] Multiplicative weighted terms

[0139] The cost of a flow vector is the SAD scaled by a weight term. The weight term is based on the motion model and the measured vector The directional difference between them. For vectors in the same direction, which may differ in magnitude from the motion model, the weight term can be less than one for local motion. In this case, local motion in the same direction as the motion model is preferred. Local motion that is not in the same direction as the motion model has a weight term greater than one. Fusion will choose the local motion and give enough SAD cost savings to offset the weight.

[0140]

[0141] Search Candidates

[0142] The motion model process is used to provide candidates to the local search, rather than providing alternatives as in the fusion approach.

[0143] Fig. 9 An example related to searching for candidates is illustrated. In this use case, the motion model generates flow used during local flow estimation, thereby giving candidates from global motion to consider during the search. For example, at step 902, camera motion is estimated, and at step 904, a motion field is generated. At step 912, image pairs are determined, and at step 914, local flow is estimated based on the image pairs and the motion field (e.g., based on candidates determined using the motion field) to determine pixel motion at step 920.

[0144] Candidates can be provided at the highest level of the multi-resolution search. In the case of fast motion, the global motion field may have large motion vectors. The use of global candidates avoids the need to significantly increase the search range used in local flow estimation to account for these long motion vectors. The global motion field is smooth in space and time because the structure of the model provides spatial smoothness and the limited changes in motion parameters provide temporal smoothness. This smooth reference can help local flows to be smoother. In addition, the motion model flow can be used to define the direction of motion, thereby simplifying the 2D search to a 1D search or essentially a 1D search in the indicated direction. This is due to the observation that the pixel flow direction of a static object is fixed, although the amplitude may vary depending on the height of the static object.

[0145] Modified cost function

[0146] In addition to providing search candidates, the cost function used to evaluate motion vector candidates during the search can be modified to include a scaling factor or penalty term that depends on the global motion vector. This is similar to the cost function used in the fusion of optical flow. This weight or penalty will bias the decision towards the global flow direction, while still allowing the local flow to deviate from the global flow if sufficient cost reduction is achieved. The previous penalty term discussed in fusion can be added to the SAD cost used during the local motion search to bias the search towards the flow direction determined by the global motion model flow:

[0147]

[0148] Pre-warping

[0149] The purpose of pre-warping is to remove known camera motion from the frame sequence prior to local flow estimation. This allows local flow estimation to operate with low complexity relative to the search range in traditional methods or the network size in neural network based methods. In the case of neural networks, this is helpful when multiple camera orientations are supported, as the known camera orientations and global motion can be removed, thereby reducing or avoiding the need to learn a range of possible camera operating modes. An additional benefit is a reduction in the range of required training data and conditions.

[0150] Fig.10 An example related to warping is illustrated. For example, at step 1002, camera motion is estimated, and at step 1004, a motion field is generated. At step 1012, image pairs are determined, and at step 1014, images are warped based on the motion field. At step 1016, local flows are estimated based on the warped images, and at step 1018, total flows are calculated based on the local flows and the motion field. At step 1020, pixel motion is determined based on the total flow.

[0151] In operation, a dense flow from the current frame to the previous frame will be generated. Using this dense global flow, the previous frame can be warped as a reference frame in which global motion is compensated. Local motion will operate on a frame pair consisting of the warped previous frame and the current frame. The total motion from the previous frame to the current frame will then be a combination of the global motion (before warping) and the estimated local motion (warped before the current frame).

[0152] Search Guide

[0153] The global motion field is caused by two types of motion of the camera, rotation and translation. The pixel flow caused by rotation depends on the camera properties but is independent of the scene content, while the translation component depends on the camera properties and motion but also depends on the content of the scene. As in the above model, the pixel flow caused by translation can be generated by assuming the properties of the scene (such as a fixed constant depth or a constant height). Although the pixel flow caused by translation depends on the scene content, at each pixel, the direction of this flow is independent of the scene content. For a pinhole camera, this corresponds to the known epipolar line in computer vision. When the camera introduces distortion, the epipolar line corresponds to a 1D curve composed of motions corresponding to different height / depth values ​​of pixels in the scene. These observations can be used to simplify the general 2D search for optical flow vectors at a point to a 1D search that starts at the flow caused by rotation and then searches along a 1D line or curve defined by translation. The following figure illustrates a 1D search at a pixel position.

[0154] Fig.11A , Fig. 11B , Fig. 11C , Fig.11D and Fig.11E 1100A, 1100B, 1100C, 1100D, and 1100E are examples of performing a ID search at a pixel location in a particular image according to the techniques described herein.

[0155] FIG. 11A to FIG. 11E The 1D search in each of includes both the direction determined by the camera translation and the offset due to the camera rotation. Fig.11A and Fig. 11B 2 shows two camera views facing different directions on a vehicle moving forward. Fig.11A Depicted is the 1D search direction at each pixel for the left rear-view camera with the vehicle moving forward. Fig. 11B Depicted is the 1D search direction at each pixel of the forward-looking camera for the case where the vehicle is moving forward. The direction of the 1D search is indicated by the line segment overlay. Since the vehicle is moving forward, there is no offset due to rotation, as seen in the next example.

[0156] exist Fig. 11C and Fig.11D An example with the same camera position but with the vehicle turning right is shown in . Fig. 11C Depicted is the 1D search direction at each pixel of the left rearview mirror camera in the case of a vehicle turning right. Fig.11D Depicted is the 1D search direction at each pixel of the forward-looking camera for the case where the vehicle turns right. The 1D search direction is still indicated by the overlapping line segments, but it can be seen that there is an additional offset due to the rotation.

[0157] Certain embodiments involve computing the basis rotation (dyaw) and translation (dZ and dX) flow fields. Given the values ​​of the motion parameters, an offset and direction for a 1D optical flow search can be defined at each pixel location. The offset is determined by scaling the basis rotation flow by the yaw rate parameter. The direction of the 1D search at pixel p is determined by the relative weights of dZ and dX due to camera translation. A 1D search for pixel matches between the current frame and the previous frame can be performed in a direction determined by a line passing through the offset.

[0158] In the case of distorted camera footage, the search can still be limited to a 1D set, but this set can be curved rather than lying perfectly on a straight line, as used to generate FIG. 11A to FIG. 11E The example pinhole camera model shown in .

[0159] In addition to the 1D guided search direction defined for each pixel location, a search range can also be defined to further reduce the number of points tested during the search. Fig.11E The use of a search range in addition to a 1D search direction is indicated in , where the search range is indicated in addition to the direction for the front-looking camera. The small segment shows the 1D search direction, and the length corresponds to the search range.

[0160] A search that is completely constrained by offset and direction will generally not capture pixel values ​​corresponding to moving objects in the scene. If the motion of the object is a translation parallel to the translation of the camera, it will be located on the 1D search curve. In some embodiments, guidance can be used to modify the penalty term rather than completely restricting the search. The search points can be located primarily along the 1D search curve, but may have some additional points off the curve to test for local motion. The penalty term will prefer the direction of motion, but still allow moving objects to be tested.

[0161] Implementation

[0162] The various components of the present invention are used to support improving optical flow. The model can additionally be used to improve other uses of motion information, such as detection of moving objects, where the motion model information is fed to a machine learning algorithm to determine objects that are in motion compared to a static background that has a flow of pixels due to a moving camera.

[0163] A model of the pixel flow caused by camera motion is determined. This can be based on analytical analysis and / or extensions to support various lens distortions. The model defines the pixel flow resulting from camera rotation and / or translation. The model describes the spatial structure of the motion field via a small set of parameters describing camera intrinsic, camera extrinsic, and vehicle motion information. By computing the flow field corresponding to basic parameter changes (such as rotation or translation), an efficient implementation is possible. Model determination and basic calculations do not need to be performed for each camera frame, as a single initialization can be used.

[0164] Next, the parameters of the motion model can be determined from external sensors (such as inertial measurement units, steering wheel angle sensors, etc.), and / or can be calculated by fitting the model to selected flow data or pixel data. This fitting may be limited by practical considerations. Using the defined model and the determined model parameters, the global flow can be used to provide improved optical flow estimation. These may involve fusion when the model output is compared with the results of another optical flow estimation algorithm and the two are fused to provide improved results. In other embodiments, the model output is used to provide guidance to the local optical flow search method by specifying a set of candidate and / or modified cost functions. The cost function can be modified to take into account deviations from the direction of the model flow. An additional example is to modify the image by pre-warping before using the algorithm to remove the global motion captured by the model, thereby allowing later processing to focus on unknown local motion. Parameter determination will usually be performed by video frame, although due to the inertia of the vehicle, the extent to which the motion parameters can change rapidly is limited. Therefore, in some cases, parameter estimation can be performed at a reduced rate.

[0165] The model and parameters are used to determine the optical flow field in any method, such as fusion, candidate search, pre-warping and / or guided search as described above.

[0166] Example Method

[0167] Fig.12 Depicted are example operations 1200 related to global motion modeling according to embodiments of the present disclosure. In some examples, these operations are performed by a system including a processor that executes a set of codes to control functional elements of a device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Typically, these operations are performed according to the methods and processes described according to various aspects of the present disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in conjunction with other operations. In some aspects, Fig.13 The processing system 1300 may perform the operation 1200 .

[0168] Operations 1200 begin at step 1205 where a first image and a second image are received from a camera attached to a moving object.

[0169] Operations 1200 continue at step 1210 where pixels in the first image are identified.

[0170] Operation 1200 continues at step 1215, where a vector representing a range of locations where a real-world point corresponding to the pixel may be found in the second image is determined based on one or more parameters associated with the camera, wherein the parameters associated with the camera include: a first parameter related to the position of the camera relative to the ground surface; a second parameter related to the motion of the moving object; and a third parameter related to the orientation of the camera relative to the ground surface. In some embodiments, determining the vector is based on projecting a plane parallel to the ground surface onto the first image based on the distance of the camera from the ground surface and the orientation of the camera relative to the ground surface, wherein the vector is located along the plane. In certain embodiments, determining the vector is based on applying a function to account for radial distortion of the first image caused by known lens distortion of the camera. For example, the function can be based on a distance relative to an optical axis of the camera.

[0171] Operation 1200 continues at step 1220, where the position of the real-world point in the second image is determined using the vector. In some embodiments, using the vector to determine the position of the real-world point in the second image includes using the vector to constrain a search for the real-world point in the second image. In certain embodiments, using the vector to determine the position of the real-world point in the second image includes penalizing candidate points in the second image that are inconsistent with the vector.

[0172] In some embodiments, using the vector to determine the position of the real-world point in the second image is based on applying a function to account for a corresponding radial distortion of the second image caused by a known lens distortion of the camera.

[0173] Some embodiments further include determining the flow field associated with the real world point based on a weighted sum of a plurality of base flow fields corresponding to each of a plurality of degrees of freedom of motion. Some embodiments further include determining a weight used in the weighted sum based on a vector.

[0174] Example Processing System

[0175] Fig.13 Depicted are devices including those operable, configured, or adapted to perform operations for the techniques disclosed herein (such as Fig.12 An example of a processing system 1300 with various components of the operations depicted and described.

[0176] Processing system 1300 includes a central processing unit (CPU) 1305, which in some examples may be a multi-core CPU 1305. Instructions executed at CPU 1305 may be loaded, for example, from a program memory 1360 associated with CPU 1305, or may be loaded from a memory 1360 partition.

[0177] The processing system 1300 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 1310, a digital signal processor (DSP) 1315, a neural processing unit (NPU) 1320, a multimedia processing unit 1325, and wireless connectivity 1330 components.

[0178] The NPU 1320 is, for example, a dedicated circuit that is typically configured to implement all necessary control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, etc. The NPU 1320 is sometimes alternatively referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), or a vision processing unit (VPU).

[0179] NPU 1320 may be configured, for example, to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other tasks. In some examples, multiple NPUs 1320 may be instantiated on a single chip such as a system on a chip (SoC), while in other examples, multiple NPUs may be part of a dedicated machine learning accelerator device.

[0180] NPU 1320 can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPU 1320 capable of performing both training and inference, these two tasks can generally still be performed independently.

[0181] The NPU 1320 designed to accelerate training is generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing data set (usually labeled or tagged), iterating on the data set, and then adjusting model parameters 1384 (such as weights and biases) to improve model performance. Generally speaking, optimization based on error prediction involves passing back through the layers of the model and determining the gradient to reduce the prediction error.

[0182] An NPU 1320 designed to accelerate inference is generally configured to operate on a complete model. Thus, such an NPU 1320 can be configured to input a new piece of data and quickly process it through a trained model to generate a model output (e.g., inference).

[0183] In some aspects, NPU 1320 may be implemented as part of one or more of CPU 1305 , GPU 1310 , and / or DSP 1315 .

[0184] NPU 1320 is a microprocessor dedicated to accelerating machine learning algorithms. For example, NPU 1320 can operate predictive models such as artificial neural networks (ANN) or random forests (RF). In some cases, NPU 1320 is designed in a way that makes it unsuitable for general-purpose computing such as performed by CPU 1305. Additionally or alternatively, software support for NPU 1320 may not be developed for general-purpose computing.

[0185] ANN is a hardware or software component including multiple connection nodes (i.e., artificial neurons), which loosely correspond to neurons in the human brain. Each connection or edge sends a signal from one node to another (such as a physical synapse in the brain). When a node receives a signal, it processes the signal and then sends the processed signal to other connection nodes. In some cases, the signal between nodes includes real numbers, and the output of each node is calculated by a function of the sum of its inputs. Each node and edge is associated with one or more node weights that determine how to process and send signals. During the training process, these weights are adjusted to improve the accuracy of the results (i.e., by minimizing the loss function that corresponds to the difference between the current result and the target result in some way). The weight of the edge increases or decreases the intensity of the signal sent between the nodes. In some cases, the node has a threshold below which no signal is sent at all. In some examples, the nodes are clustered into layers. Different layers perform different transformations on their inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signal passes through certain layers many times.

[0186] Convolutional neural networks (CNNs) are a class of neural networks commonly used in computer vision or image classification systems. In some cases, CNNs can enable processing of digital images with minimal preprocessing. CNNs can be characterized by using convolutional (or cross-correlation) hidden layers. These layers apply convolution operations to the input before signaling the results to the next layer. Each convolution node can process data of a finite field (i.e., receiving field) of the input. During the forward pass of the CNN, the filter at each layer can be convolved across the input volume, thereby calculating the dot product between the filter and the input. During the training process, the filters can be modified so that they activate when they detect specific features within the input.

[0187] Supervised learning is one of the three basic machine learning paradigms in addition to unsupervised learning and reinforcement learning. Supervised learning is a machine learning technique based on learning a function that maps input to output based on example input-output pairs. Supervised learning generates a function for predicting labeled data based on labeled training data consisting of a set of training examples. In some cases, each instance is a pair consisting of an input object (usually a vector) and a desired output value (i.e., a single value, or an output vector). The supervised learning algorithm analyzes the training data and generates an inference function that can be used to map new examples. In some cases, learning generates a function that correctly determines the class label of an unseen instance. In other words, the learning algorithm generalizes from the training data to unseen examples.

[0188] The term "loss function" refers to a function that affects how a machine learning model is trained in a supervised learning model. Specifically, during each training iteration, the output of the model is compared to the known annotation information in the training data. The loss function provides a value for how close the predicted annotation data is to the actual annotation data. After calculating the loss function, the parameters of the model are updated accordingly, and a new set of predictions are made during the next iteration.

[0189] In some aspects, the wireless connectivity 1330 components may include, for example, subcomponents for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity 1330 processing components are also connected to one or more antennas 1335.

[0190] The processing system 1300 may also include one or more sensor processing units associated with any manner of sensors, one or more image signal processors (ISP 1345) associated with any manner of image sensors, and / or a navigation 1350 processor, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0191] The processing system 1300 may also include one or more input and / or output devices, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0192] In some examples, one or more of the processors of processing system 1300 may be based on the ARM or RISC-V instruction set.

[0193] The processing system 1300 also includes a memory 1360, which represents one or more static and / or dynamic memories, such as a dynamic random access memory 1360, a flash-based static memory 1360, etc. In this example, the memory 1360 includes computer-executable components that can be executed by one or more of the aforementioned components of the processing system 1300.

[0194] Examples of memory 1360 include random access memory (RAM), read-only memory (ROM), or hard disk. Examples of memory 1360 include solid-state memory and hard disk drive. In some examples, memory 1360 is used to store computer-readable, computer-executable software containing instructions, which, when executed, cause the processor to perform various functions described herein. In some cases, among other things, memory 1360 includes a basic input / output system (BIOS), which controls basic hardware or software operations, such as interactions with peripheral components or devices. In some cases, a memory controller operates a memory cell. For example, a memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells in memory 1360 store information in the form of logical states.

[0195] Specifically, in this example, memory 1360 includes a location identification component that can perform operations related to global motion modeling described herein, such as Fig.12 The operations described in .

[0196] Generally, the processing system 1300 and / or its components may be configured to perform the methods described herein.

[0197] It is noted that in other aspects, aspects of the processing system 1300 may be omitted, such as where the processing system 1300 is a server computer, etc. For example, in other aspects, the multimedia component 1325, wireless connectivity 1330, sensor 1340, ISP 1345, and / or navigation 1350 components may be omitted. Furthermore, aspects of the processing system 1300 may be distributed.

[0198] Notice, Fig.13 This is merely one example, and alternative processing systems 1300 having more, fewer, and / or different components may be used in other examples.

[0199] In one aspect, processing system 1300 includes CPU 1305 , GPU 1310 , DSP 1315 , NPU 1320 , multimedia 1325 , wireless connectivity 1330 , antenna 1335 , sensor 1340 , ISP 1345 , navigation 1350 , input / output 1355 , and memory 1360 .

[0200] In some aspects, the sensor 1340 may include an optical instrument (e.g., an image sensor, a camera, etc.) for recording or capturing an image, which may be stored locally, sent to another location, etc. For example, an image sensor may capture visual information using one or more photosensitive elements that may tune sensitivity for the visible spectrum of electromagnetic radiation. The resolution of such visual information may be measured in pixels, where each pixel may relate to an independent piece of captured information. In some cases, each pixel may therefore correspond to, for example, a component of a two-dimensional (2D) Fourier transform of an image. Computational methods may use pixel information to reconstruct an image captured by the device. In a camera, an image sensor may convert light incident on a camera lens into an analog or digital signal. The electronic device may then display the image on a display panel based on the digital signal. Image sensors are typically mounted on electronic devices such as smart phones, tablet personal computers (PCs), laptop PCs, and wearable devices.

[0201] In some aspects, sensor 1340 may include a direct depth sensing sensor, such as, for example, radar, LIDAR, and other depth sensing sensors, as described herein.

[0202] Input / output 1355 (e.g., I / O controller) can manage input and output signals for device. Input / output 1355 can also manage peripheral devices that are not integrated into the device. In some cases, input / output 1355 can represent physical connection or port to external peripheral devices. In some cases, input / output 1355 can utilize operating system. In other cases, input / output 1355 can represent modem, keyboard, mouse, touch screen or similar device or interact with it. In some cases, input / output 1355 can be implemented as a part of processor (e.g., CPU1305). In some cases, users can interact with device via input / output 1355 or via hardware components controlled by input / output 1355.

[0203] In some examples, such as where processing system 1300 is a server computer, certain aspects, such as wireless connectivity component 1330, antenna 1335, multimedia component 1325, navigation component 1350, and sensor 1340 may be omitted.

[0204] It is worth noting that Fig.13 These are examples only, and many other examples and configurations of processing system 1300 are possible, including implementations involving multiple computing devices.

[0205] Sample Clauses

[0206] Specific implementation examples are described in the following numbered clauses:

[0207] Clause 1: A method comprising: receiving a first image and a second image from a camera attached to a moving object; identifying a pixel in the first image; determining a vector representing a range of locations where a real-world point corresponding to the pixel may be found in the second image based on one or more parameters associated with the camera, wherein the parameters associated with the camera include: a first parameter related to the position of the camera relative to a ground surface; a second parameter related to the motion of the moving object; and a third parameter related to the orientation of the camera relative to the ground surface; and using the vector to determine the position of the real-world point in the second image.

[0208] Clause 2: The method of clause 1, wherein determining the position of the real-world point in the second image using the vector comprises using the vector to constrain a search for the real-world point in the second image.

[0209] Clause 3: A method according to any one of clauses 1 to 2, wherein the vector is determined based on projecting a plane parallel to the ground surface onto the first image based on the distance of the camera from the ground surface and the orientation of the camera relative to the ground surface, wherein the vector is positioned along the plane.

[0210] Clause 4: A method according to any of clauses 1 to 3, wherein determining the position of the real-world point in the second image using the vector comprises penalizing candidate points in the second image that are inconsistent with the vector.

[0211] Clause 5: The method of any of clauses 1 to 4, wherein determining the vector is based on applying a function to account for radial distortion of the first image caused by known lens distortion of the camera.

[0212] Clause 6: The method of clause 5, wherein determining the position of the real-world point in the second image is based on applying the function to account for a corresponding radial distortion of the second image caused by the known lens distortion of the camera.

[0213] Clause 7: A method according to any of clauses 5 to 6, wherein the function is based on a distance relative to an optical axis of the camera.

[0214] Clause 8: The method of any one of clauses 1 to 7, further comprising determining a flow field associated with the real-world point based on a weighted sum of a plurality of base flow fields corresponding to each of a plurality of degrees of freedom of movement.

[0215] Clause 9: The method of clause 8, further comprising determining a weight to be used in the weighted sum based on the vector.

[0216] Item 10: A processing system comprising: a memory comprising computer-executable instructions; one or more processors configured to execute the computer-executable instructions and cause the processing system to: receive a first image and a second image from a camera attached to a moving object; identify a pixel in the first image; determine a vector representing a range of positions where a real-world point corresponding to the pixel may be found in the second image based on one or more parameters associated with the camera, wherein the parameters associated with the camera include: a first parameter related to the position of the camera relative to a ground surface; a second parameter related to the motion of the moving object; and a third parameter related to the orientation of the camera relative to the ground surface; and use the vector to determine the position of the real-world point in the second image.

[0217] Clause 11: The system of clause 10, wherein determining the position of the real-world point in the second image using the vector comprises using the vector to constrain a search for the real-world point in the second image.

[0218] Clause 12: A system according to any one of clauses 10 to 11, wherein determining the vector is based on projecting a plane parallel to the ground surface onto the first image based on the distance of the camera from the ground surface and the orientation of the camera relative to the ground surface, wherein the vector is positioned along the plane.

[0219] Clause 13: The system of any of clauses 10 to 12, wherein determining the position of the real-world point in the second image using the vector comprises penalizing candidate points in the second image that are inconsistent with the vector.

[0220] Clause 14: The system of any of clauses 10 to 13, wherein determining the vector is based on applying a function to account for radial distortion of the first image caused by known lens distortion of the camera.

[0221] Clause 15: The system of clause 14, wherein determining the position of the real-world point in the second image is based on applying the function to account for a corresponding radial distortion of the second image caused by the known lens distortion of the camera.

[0222] Clause 16: A system as described in any of clauses 14 to 15, wherein the function is based on a distance relative to an optical axis of the camera.

[0223] Clause 17: A system according to any one of clauses 10 to 16, wherein the one or more processors are further configured to determine the flow field associated with the real-world point based on a weighted sum of multiple base flow fields corresponding to each of multiple degrees of freedom of movement.

[0224] Clause 18: The system of clause 17, wherein the one or more processors are further configured to determine a weight to use in the weighted sum based on the vector.

[0225] Item 19: A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to: receive a first image and a second image from a camera attached to a moving object; identify a pixel in the first image; determine a vector representing a range of locations where a real-world point corresponding to the pixel may be found in the second image based on one or more parameters associated with the camera, wherein the parameters associated with the camera include: a first parameter related to the position of the camera relative to a ground surface; a second parameter related to the motion of the moving object; and a third parameter related to the orientation of the camera relative to the ground surface; and use the vector to determine the position of the real-world point in the second image.

[0226] Clause 20: The non-transitory computer-readable medium of Clause 19, wherein determining the position of the real-world point in the second image using the vector comprises using the vector to constrain a search for the real-world point in the second image.

[0227] Clause 21: A non-transitory computer-readable medium according to any one of clauses 19 to 20, wherein the vector is determined based on projecting a plane parallel to the ground surface onto the first image based on the distance of the camera from the ground surface and the orientation of the camera relative to the ground surface, wherein the vector is positioned along the plane.

[0228] Clause 22: The non-transitory computer readable medium of any one of clauses 19 to 21, wherein determining the position of the real-world point in the second image using the vector comprises penalizing candidate points in the second image that are inconsistent with the vector.

[0229] Clause 23: The non-transitory computer readable medium of Clause 22, wherein determining the vector is based on applying a function to account for radial distortion of the first image caused by known lens distortion of the camera.

[0230] Clause 24: A non-transitory computer-readable medium as described in any of clauses 22 to 23, wherein determining the position of the real-world point in the second image is based on applying the function to account for a corresponding radial distortion of the second image caused by the known lens distortion of the camera.

[0231] Clause 25: The non-transitory computer readable medium of any one of clauses 22 to 24, wherein the function is based on a distance relative to an optical axis of the camera.

[0232] Clause 26: A non-transitory computer-readable medium according to any one of clauses 19 to 25, wherein the instructions, when executed by the one or more processors, further cause the computing system to determine a flow field associated with the real-world point based on a weighted sum of multiple base flow fields corresponding to each of multiple degrees of freedom of motion.

[0233] Clause 27: The non-transitory computer readable medium of clause 26, wherein the instructions, when executed by the one or more processors, further cause the computing system to determine a weight to use in the weighted sum based on the vector.

[0234] Clause 28: An apparatus comprising: a component for receiving a first image and a second image from a camera attached to a mobile object; a component for identifying a pixel in the first image; a component for determining a vector representing a range of positions where a real-world point corresponding to the pixel may be found in the second image based on one or more parameters associated with the camera, wherein the parameters associated with the camera include: a first parameter related to the position of the camera relative to a ground surface; a second parameter related to the movement of the mobile object; and a third parameter related to the orientation of the camera relative to the ground surface; and a component for determining the position of the real-world point in the second image using the vector.

[0235] Clause 29: The apparatus of clause 28, wherein determining the position of the real-world point in the second image using the vector comprises using the vector to constrain a search for the real-world point in the second image.

[0236] Clause 30: An apparatus according to any one of clauses 28 to 29, wherein determining the vector is based on projecting a plane parallel to the ground surface onto the first image based on the distance of the camera from the ground surface and the orientation of the camera relative to the ground surface, wherein the vector is positioned along the plane.

[0237] Additional considerations

[0238] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not limited to the scope, applicability or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, without departing from the scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Various examples may omit, replace or add various processes or components as appropriate. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted or combined. In addition, the features described with respect to some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functionality, or structures and functionality other than the various aspects of the present disclosure set forth herein or different from the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the present invention.

[0239] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0240] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items (including single members). For example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0241] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, and the like. Additionally, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Additionally, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0242] The method disclosed herein includes one or more steps or actions for implementing the method. The steps and / or actions of the method can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. In addition, the various operations of the method described above can be performed by any suitable component capable of performing the corresponding function. The component may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Typically, in the case of operations illustrated in the accompanying drawings, those operations may have corresponding corresponding components with similar numbers plus functional components.

[0243] The following claims are not intended to be limited to the various aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, reference to an element in the singular form is not intended to mean "one and only one", but "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. Any claim element should not be interpreted according to the provisions of 35 U.S.C. § 112 (f), unless the element is explicitly recorded using the phrase "a component for...", or in the case of a method claim, the element is recorded using the phrase "a step with...". All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known later to a person of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, regardless of whether such disclosure is explicitly stated in the claims, anything disclosed herein is not intended to be disclosed to the public.

Claims

1. A method, comprising: receiving a first image and a second image from a camera attached to the moving object; identifying pixels in the first image; determining a vector representing a range of locations where a real-world point corresponding to the pixel may be found in the second image based on one or more parameters associated with the camera, wherein the parameters associated with the camera include: a first parameter related to the position of the camera relative to the ground surface; a second parameter related to the motion of the mobile object; and a third parameter related to the orientation of the camera relative to the ground surface; and The position of the real-world point in the second image is determined using the vector. 2 . The method of claim 1 , wherein using the vector to determine the position of the real-world point in the second image comprises using the vector to constrain a search for the real-world point in the second image.

3. The method of claim 1 , wherein determining the vector is based on projecting a plane parallel to the ground surface onto the first image based on the distance of the camera from the ground surface and the orientation of the camera relative to the ground surface, wherein the vector is positioned along the plane. 4 . The method of claim 1 , wherein using the vector to determine the position of the real-world point in the second image comprises penalizing candidate points in the second image that are inconsistent with the vector. The method of claim 1 , wherein determining the vector is based on applying a function to account for radial distortion of the first image caused by known lens distortion of the camera. 6 . The method of claim 5 , wherein determining the position of the real-world point in the second image is based on applying the function to account for a corresponding radial distortion of the second image caused by the known lens distortion of the camera. The method of claim 5 , wherein the function is based on a distance relative to an optical axis of the camera.

8. The method of claim 1, further comprising determining a flow field associated with the real-world point based on a weighted sum of a plurality of base flow fields corresponding to each of a plurality of degrees of freedom of motion.

9. The method of claim 8, further comprising determining a weight to be used in the weighted sum based on the vector.

10. A processing system, comprising: a memory including computer executable instructions; one or more processors configured to execute the computer-executable instructions and cause the processing system to: receiving a first image and a second image from a camera attached to the moving object; identifying pixels in the first image; determining a vector representing a range of locations where a real-world point corresponding to the pixel may be found in the second image based on one or more parameters associated with the camera, wherein the parameters associated with the camera include: a first parameter related to the position of the camera relative to the ground surface; a second parameter related to the motion of the mobile object; and a third parameter related to the orientation of the camera relative to the ground surface; and The position of the real-world point in the second image is determined using the vector. 11 . The system of claim 10 , wherein using the vector to determine the position of the real-world point in the second image comprises using the vector to constrain a search for the real-world point in the second image.

12. The system of claim 10, wherein determining the vector is based on projecting a plane parallel to the ground surface onto the first image based on the distance of the camera from the ground surface and the orientation of the camera relative to the ground surface, wherein the vector is located along the plane.

13. The system of claim 10, wherein using the vector to determine the position of the real-world point in the second image comprises penalizing candidate points in the second image that are inconsistent with the vector.

14. The system of claim 10, wherein determining the vector is based on applying a function to account for radial distortion of the first image caused by known lens distortion of the camera.

15. The system of claim 14, wherein determining the position of the real-world point in the second image is based on applying the function to account for a corresponding radial distortion of the second image caused by the known lens distortion of the camera.

16. The system of claim 14, wherein the function is based on a distance relative to an optical axis of the camera.

17. The system of claim 10, wherein the one or more processors are further configured to determine a flow field associated with the real-world point based on a weighted sum of a plurality of base flow fields corresponding to each of a plurality of degrees of freedom of motion.

18. The system of claim 17, wherein the one or more processors are further configured to determine a weight to use in the weighted sum based on the vector.

19. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to: receiving a first image and a second image from a camera attached to the moving object; identifying pixels in the first image; determining a vector representing a range of locations where a real-world point corresponding to the pixel may be found in the second image based on one or more parameters associated with the camera, wherein the parameters associated with the camera include: a first parameter related to the position of the camera relative to the ground surface; a second parameter related to the motion of the mobile object; and a third parameter related to the orientation of the camera relative to the ground surface; as well as The position of the real-world point in the second image is determined using the vector.

20. The non-transitory computer readable medium of claim 19, wherein determining the position of the real-world point in the second image using the vector comprises using the vector to constrain a search for the real-world point in the second image.

21. The non-transitory computer-readable medium of claim 19, wherein determining the vector is based on projecting a plane parallel to the ground surface onto the first image based on a distance of the camera from the ground surface and the orientation of the camera relative to the ground surface, wherein the vector is positioned along the plane.

22. The non-transitory computer readable medium of claim 19, wherein determining the position of the real-world point in the second image using the vector comprises penalizing candidate points in the second image that are inconsistent with the vector.

23. The non-transitory computer readable medium of claim 22, wherein determining the vector is based on applying a function to account for radial distortion of the first image caused by known lens distortion of the camera.

24. The non-transitory computer readable medium of claim 22, wherein determining the position of the real-world point in the second image is based on applying the function to account for a corresponding radial distortion of the second image caused by the known lens distortion of the camera.

25. The non-transitory computer readable medium of claim 22, wherein the function is based on a distance relative to an optical axis of the camera.

26. The non-transitory computer-readable medium of claim 19, wherein the instructions, when executed by the one or more processors, further cause the computing system to determine a flow field associated with the real-world point based on a weighted sum of a plurality of base flow fields corresponding to each of a plurality of degrees of freedom of motion.

27. The non-transitory computer readable medium of claim 26, wherein the instructions, when executed by the one or more processors, further cause the computing system to determine a weight used in the weighted sum based on the vector.

28. An apparatus, comprising: means for receiving a first image and a second image from a camera attached to a moving object; means for identifying pixels in said first image; means for determining a vector representing a range of locations where a real-world point corresponding to the pixel may be found in the second image based on one or more parameters associated with the camera, wherein the parameters associated with the camera include: a first parameter related to the position of the camera relative to the ground surface; a second parameter related to the motion of the mobile object; and a third parameter related to the orientation of the camera relative to the ground surface; and Means for determining a position of the real-world point in the second image using the vector.

29. The apparatus of claim 28, wherein using the vector to determine the position of the real-world point in the second image comprises using the vector to constrain a search for the real-world point in the second image.

30. The apparatus of claim 28, wherein determining the vector is based on projecting a plane parallel to the ground surface onto the first image based on the distance of the camera from the ground surface and the orientation of the camera relative to the ground surface, wherein the vector is located along the plane.

Images