Monocular camera system for a vehicle for estimating the depth of an object

Abstract

A monocular camera system (10) for a vehicle (12), the monocular camera system (10) comprising: a front mono camera (30) positioned at a front part of the vehicle (12); a side mono camera (32) positioned along one side of the vehicle (12); and one or more controllers (20) in electronic communication with the front mono camera (30) and the side mono camera (32), the one or more controllers (20) executing instructions to: collect image data captured by the front mono camera (30) and the side mono camera (32), the image data including an object to be assessed; to determine an ideal deceleration distance that the vehicle (12) travels between a current time step and a previous time step while two asynchronous camera frames are recorded by the front mono camera (30) and the side mono camera (32); determine a number of delayed frames captured by either the front mono camera (30) or the side mono camera (32) between the current time step and the previous time step based on the ideal delay distance; determine a direction of travel of the vehicle (12), wherein the direction of travel of the vehicle (12) indicates which mono camera is selected to provide a previous camera frame captured in the previous time step; perform a feature match to identify pixel positions in a camera frame captured by the front mono camera (30) and a camera frame captured by the side mono camera (32) that correspond to the same three-dimensional coordinates of the object to be assessed; and perform a triangulation of the pixel positions detected by the front monocamera (30) and the side monocamera (32) in order to determine the three-dimensional coordinates of the object to be assessed.

Description

IntroductionThe present invention relates to a monocular camera system for a vehicle, in which the monocular camera system performs a depth estimation of an object.For general background information, reference is made here to US 2012 / 0 170 812 A1.An autonomous vehicle performs various tasks such as perception, localization, mapping, path planning, decision making, and motion control, without limitation. For example, an autonomous vehicle may include perception sensors such as one or more cameras to capture image data related to the environment around the vehicle. The image data collected by the cameras may be used in various active systems that are part of the vehicle. In a specific example, the image data collected by the cameras may be used for curb detection and localization. However, it may be difficult to estimate the height and depth of an object based on a monocular camera system. That is, a monocular camera system may not be as robust when a three-dimensional view of the world is calculated from a planar two-dimensional image received from a single camera, compared to a stereo vision camera that includes multiple cameras with overlapping fields of view.One approach to alleviate the problems experienced in estimating an object's height and depth based on a monocular camera system may include an additional near-field depth sensor, such as a LiDAR, an ultrasonic sensor, or a short-range radar, as part of the monocular camera system. However, the additional depth sensor introduces unnecessary complexity into the monocular camera system.Thus, although autonomous vehicle camera systems fulfil their intended purpose, there is a need for an improved approach to estimating the depth and height of an object based on a monocular camera system.SummaryAccording to the present invention, there is provided a monocular camera system for a vehicle, comprising a front monocamera positioned at a front portion of the vehicle, a side monocamera positioned along a side of the vehicle, and one or more controllers in electronic communication with the front monocamera and the side monocamera. The one or more controllers execute instructions to collect image data captured by the front mono camera and the side mono camera, the image data including an object to be evaluated. The controllers determine an ideal lag distance (ideal lag distance) that the vehicle travels between a current time step and a previous time step while two asynchronous camera frames are captured by the front mono camera and the side mono camera. The controllers determine a number of delayed frames captured by either the front mono camera or the side mono camera between the current time step and the previous time step based on the ideal delay distance. The controllers determine a direction of travel of the vehicle, wherein the direction of travel of the vehicle indicates which monocamera is selected to provide a previous camera frame captured in the previous time step. The controllers perform feature matching to identify pixel positions in a camera frame captured by the front mono camera and a camera frame captured by the side mono camera corresponding to the same three-dimensional coordinates of the object to be evaluated. The controllers triangulate the pixel positions detected by the front mono camera and the side mono camera to determine the three-dimensional coordinates of the object to be evaluated.In one aspect, the one or more controllers execute instructions to, in response to determining that the vehicle is driving in the forward direction, select the previous camera frame in the image data captured by the front mono camera in the previous time step and the camera frame captured by the side mono camera in the current time step.In another aspect, the one or more controllers execute instructions to, in response to determining that the vehicle is driving in the rearward direction, select the previous camera frame in the image data captured by the side mono camera in the previous time step and the camera frame captured by the front mono camera in the current time step.In yet another aspect, the number of delayed frames indicates how many frames are counted back in the image data captured by either the front mono camera or the side mono camera between the current time step and the previous time step.In one aspect, the number of delayed frames is determined based on: where v l is a longitudinal speed of the vehicle, s represents the ideal delay distance, FPS represents frames per second of an object camera, and floor( ) represents an operator returning the integer portion of a floating point number.In another aspect, determining the ideal delay distance includes resolving for a range of overlap between a front field of view captured by the front monocamera and a side field of view captured by the side monocamera.In yet another aspect, a steering angle of the vehicle is zero, and the range of overlap is determined by: where A represents the range of overlap, L 1 represents a equations of lines defining the boundaries of a lateral field of view of the side monocamera, L 2 represents a equations of lines defining the boundaries of a front field of view of the front monocamera, x r represents an x position of the side monocamera, y r represents a y position of the side monocamera, l e is an upper limit of integration, x represents a lateral direction, and y represents a longitudinal direction.In one aspect, the equation of lines defining the limits of the lateral field of view of the side monocamera is solved as follows: where FOV r represents the lateral field of view for the side monocamera and ψ r is a yaw angle for the side monocamera.In another aspect, the equation of lines defining the boundaries of the front field of view of the front monocamera is solved as follows: where FOV f represents the front field of view for the front monocamera and ψ f is a yaw angle for the front monocamera.In yet another aspect, determining the ideal delay path comprises:solving a convex optimization problem by plotting a relationship between a convex cost function and the ideal delay distance, wherein the relationship between the convex cost function and the ideal delay distance is represented by a parabola; andselecting a local minimum of the parabola as an ideal delay path.In one aspect, the convex cost function is expressed as: wherein the convex cost function is s represents the ideal delay distance, and η represents a scalar.In another aspect, a steering angle of the vehicle is a non-zero value, and the range of overlap is determined by: where A represents the range of overlap, L 1 represents a equations of lines defining the boundaries of a lateral field of view of the side monocamera, L 2 represents an equations of lines defining the boundaries of a front field of view of the front monocamera, x r' represents an x position of the side monocamera when turning, y r' represents a y position of the side monocamera when turning, l e is an upper limit of integration, and x represents a lateral direction.In yet another aspect, a steering angle of the vehicle is a non-zero value and the range of overlap is determined by: where R represents a turning radius of the vehicle, θ represents the steering angle of the vehicle, x r represents an x position of the side monocamera, and y r represents a y position of the side monocamera.In one aspect, the equation of lines defining the boundaries of the front field of view of the front monocamera is solved as follows: where FOV r represents the lateral field of view for the side monocamera, ψ r is a yaw angle for the side monocamera, and y represents the longitudinal direction.In another aspect, the equation of lines defining the limits of the lateral field of view of the side monocamera is solved as follows: where FOV f represents the front field of view for the front monocamera and ψ f is a yaw angle for the front monocamera.Furthermore, a method for determining three-dimensional coordinates of an object by means of a monocular camera system for a vehicle is described. The method includes collecting, by one or more controllers, image data captured by a front mono camera and a side mono camera, the image data including an object to be evaluated. The method includes determining an ideal delay distance that the vehicle travels between a current time step and a previous time step while capturing two asynchronous camera frames from the front mono camera and the side mono camera. The method also includes determining a number of delayed frames captured by either the front mono camera or the side mono camera between the current time step and the previous time step based on the ideal delay distance. The method further comprises determining a direction of travel of the vehicle, wherein the direction of travel of the vehicle indicates which monocamera is selected to provide a previous camera frame captured in the previous time step. The method also includes performing feature matching to identify pixel positions in a camera frame captured by the front mono camera and a camera frame captured by the side mono camera that correspond to the same three-dimensional coordinates of the object to be evaluated. Finally, the method includes performing triangulation of the pixel positions captured by the front monocamera and the side monocamera to determine the three-dimensional coordinates of the object to be evaluated.In another aspect, the method includes, in response to determining that the vehicle is driving in the forward direction, selecting the previous camera frame in the image data captured by the front mono camera in the previous time step and the camera frame captured by the side mono camera in the current time step.In yet another aspect, the method includes, in response to determining that the vehicle is travelling in the rearward direction, selecting the previous camera frame in the image data captured by the side mono camera in the previous time step and the camera frame captured by the front mono camera in the current time step.In one aspect, the method further includes resolving, after a region of overlap between a front field of view captured by the front monocamera and a side field of view captured by the side monocamera. The method includes solving a convex optimization problem by plotting a relationship between a convex cost function and the ideal delay distance, wherein the relationship between the convex cost function and the ideal delay distance is represented by a parabola. The method also includes selecting a local minimum of the parabola as an ideal delay path.In another aspect, a monocular camera system for a vehicle is disclosed, and includes a front mono camera positioned at a front portion of the vehicle, a side mono camera positioned along a side of the vehicle, and one or more controllers in electronic communication with the front mono camera and the side mono camera. The one or more controllers execute instructions to collect image data captured by the front mono camera and the side mono camera, the image data including an object to be evaluated. The controllers determine an ideal delay distance that the vehicle travels between a current time step and a previous time step while two asynchronous camera frames are captured by the front mono camera and the side mono camera. Determining the ideal delay distance includes resolving, after a range of overlap between a front field of view captured by the front mono camera and a side field of view captured by the side mono camera, resolving a convex optimization problem by plotting a relationship between a convex cost function and the ideal delay distance, wherein the relationship between the convex cost function and the ideal delay distance is represented by a parabola, and selecting a local minimum of the parabola as the ideal delay distance. The controllers determine a number of delayed frames captured by either the front mono camera or the side mono camera between the current time step and the previous time step based on the ideal delay distance. The controllers determine a direction of travel of the vehicle, where the direction of travel of the vehicle indicates which monocamera is selected to provide a previous camera frame captured in the previous time step. The controllers perform feature matching to identify pixel positions in a camera frame captured by the front mono camera and a camera frame captured by the side mono camera corresponding to the same three-dimensional coordinates of the object to be evaluated. The controllers triangulate the pixel positions detected by the front mono camera and the side mono camera to determine the three-dimensional coordinates of the object to be evaluated.Further areas of applicability will become apparent from the description provided herein. It should be understood that the specification and specific examples are for illustrative purposes only.Brief Description of the DrawingsThe drawings described herein are for illustrative purposes only. FIG. 1 is a schematic diagram of a vehicle including the disclosed monocular camera system including one or more controllers in electronic communication with a front camera and a side camera, according to an example embodiment; FIG. 2 is a block diagram of the one or more controllers shown in FIG. 1, according to an example embodiment; FIG. 3 is an exemplary diagram of the forward-driving vehicle illustrated in FIG. 1 according to an exemplary embodiment; FIG. 4 is a graph illustrating a relationship between a convex cost function and an ideal deceleration distance when a steering angle of the vehicle is zero according to an exemplary embodiment; FIG. 5 is an exemplary diagram of the vehicle traveling in the forward direction with a steering angle illustrated in FIG. 1 according to an exemplary embodiment; FIG. 6 is a graph illustrating a relationship between a convex cost function and the ideal deceleration distance as the steering angle of the vehicle increases according to an exemplary embodiment; and FIG. 7 is a process flow diagram illustrating a method for determining the three-dimensional coordinates of an object using the monocular camera system according to an example embodiment.Detailed DescriptionThe following description is merely exemplary in nature and is not intended to limit the present invention, application, and uses.Referring to FIG. 1, an example monocular camera system 10 for depth estimation of an object is illustrated in FIG. 1. The monocular camera system 10 is part of a vehicle 12. it should be understood that the vehicle 12 may be any type of vehicle, such as, but not limited to, a sedan, a truck, a sport utility vehicle (SPORT UTILITY VEHICLE), a van, or a recreational vehicle. In one embodiment, the monocular camera system 10 is part of an automated driving system (ADS) or an advanced driver assistance system (ADAS) for assisting the driver in steering, braking, and / or accelerating; however, the monocular camera system 10 may also be used as part of a manually controlled vehicle. The monocular camera system 10 includes one or more controllers 20 in electronic communication with a plurality of perception sensors 22 that collect perceptions that indicate an environment 14 around the vehicle 12. The perception sensors 22 include a plurality of mono cameras 30, 32, an inertial measurement unit 34 (IMU), and a global positioning system 36 (GPS), however, it should be appreciated that additional or different perception sensors may also be used.In the example as shown in FIG. 1, the one or more controllers 20 are in electronic communication with a front monocamera 30 and a side monocamera 32, wherein the front monocamera 30 is positioned at a front portion 40 of the vehicle 12 to detect a front field of view FOV f and the side monocamera 32 is positioned along a right side 42 of the vehicle 12 to detect a side field of view FOV r. Although in FIG. 1, the side monocamera 32 is positioned along the right side 42 of the vehicle 12, it should be appreciated that the images are exemplary only and, in another embodiment, the side monocamera 32 is instead positioned along a left side 44 of the vehicle 12. FIG. 1 illustrates a negligible amount of overlap between the front field of view FOV f and the right field of view FOV r, when the vehicle 12 is stationary, wherein the negligible amount of overlap may be disregarded and the mono cameras 30, 32 are considered non-overlapping fields of view. As will be explained below, the monocular camera system 10 generates a temporary overlap of the front field of view FOV f, which is captured in a first time step, and the right field of view FOV r, which is captured in a second time step, which can be effectively used to perform stereo camera depth estimation.FIG. 2 is a block diagram of the one or more controllers 20 shown in FIG. 1, the one or more controllers 20 include an ideal delay line module 50, a frame module 52, a flag module 54, a front monocamera module 56, a side monocamera module 58, a feature matching module 60, and a triangulation module 62. Referring to both FIGS. 1 and 2, the ideal delay line module 50 of the one or more controllers 20 collects image data captured by the front monocamera 30 and the side monocamera 32, the image data including one or more objects that are evaluated for depth. In a non-limiting embodiment, the one or more objects include, for example, a curb that is part of a scene surrounding the vehicle 12.The ideal delay distance module 50 of the one or more controllers 20 determines an ideal lag distance (s) that the vehicle 12 travels between a current time step t and a previous time step t-δT while two asynchronous camera frames are captured by the front mono camera 30 and the side mono camera 32. The previous time step t-δT is measured between the current time step t and a change in time δT. If the vehicle 12 is traveling in the forward direction, as illustrated in FIG. 3, a front camera frame is captured at the previous time step t-δT, and a side camera frame is captured at the current time step t. If the vehicle 12 is traveling in the rearward direction, the front camera frame is captured at the current time step t, and the side camera frame is captured at the previous time step t-δT.Finally, if the vehicle 12 turns, then the front camera frame is captured in the previous time step t- δT, and the side camera frame is captured in the current time step t.FIG. 3 is an example diagram of the vehicle 12 traveling in the forward direction without turning, wherein the ideal deceleration distance module 50 of the one or more controllers 20 (FIG. 2 ) resolves after an area A of overlap between the front field of view FOV f, captured by the front monocamera 30, and the side field of view FOV r, captured by the side monocamera 32. It should be appreciated that the ideal delay distance s is optimized to maximize the overlap area A between the two fields of view captured by the two mono cameras 30, 32 while simultaneously minimizing the amount of delay between the current time step t and the previous time step t-δT.In the example as illustrated in FIG. 3, a steering angle θ of the vehicle 12 is zero. FIG. 5 is an illustration of the vehicle turning, where the steering angle θ is a non-zero value, which will be described below. Referring to FIGS. 2 and 3, the overlap region A is determined by integrating a difference between a straight line equation L 1, which defines the boundaries of the lateral field of view FOV r and a straight line equation L 2, which defines the boundaries of the front field of view FOV f and is expressed in Equation 1 as follows: where x r represents an x position of the side monocamera 32, y r represents a y position of the side monocamera 32, l e is the upper limit of the integration, cte represents a constant, x is the lateral direction, and y is the longitudinal direction. In this example, the x position x r of the page monocamera 32 remains constant. The upper limit l e of the integration is based on a range judged for the depth estimation. In an example where the monocular camera system 10 is used for curb detection, the upper limit l e for integration is about 5 meters. The straight line equations L1and L2are expressed in equations 2 and 3 as follows: where ψ f is the yaw angle of the front monocamera 30, and ψ r is the yaw angle for the side monocamera 32.Once resolved after the overlap region, the ideal delay distance module 50 of the one or more controllers 20 (FIG. 2 ) determines the ideal delay distance s by solving a convex optimization problem. The convex optimization problem is based on a convex cost function J, wherein the convex cost function minimizes the ideal delay distance s while simultaneously maximizing the overlap region A. FIG. 4 is a graph showing a relationship between the convex cost function J and the ideal delay distance s. The convex optimization is solved by plotting the relationship between the convex cost function J (along the y-axis) and the ideal delay distance s (along the x-axis), the relationship between the convex cost function J and the ideal delay distance s is represented by a parabola, and the ideal delay distance s is selected as the local minimum 70 of the parabola. The convex cost function is a function of the overlap region A, the ideal delay distance s and a scalar η and is expressed in equation 4 as follows: wherein the scalar η is selected to balance the relationship between the overlap region A and the ideal delay distance s.In the embodiment as illustrated in FIGS. 3 and 4, the vehicle 12 is steered straight, with the steering angle θ being zero. However, if the steering angle θ is a value other than zero, the ideal deceleration distance s decreases as the steering angle θ increases. FIG. 5 is an exemplary diagram of the vehicle 12 traveling with the steering angle θ in the forward direction. the overlap area A is determined by integrating a difference between the straight line equation L 1, which defines the boundaries of the lateral field of view FOV r and the straight line equation L 2, which defines the boundaries of the front field of view FOV f and is expressed in Equation 5 as follows: where x r' represents an x position of the side monocamera 32 while the vehicle 12 is turning, and y r' represents a y position of the side monocamera 32 while the vehicle is turning. The x-position x r' and the y-position y r' are expressed in equations 6 and 7 as follows: where R represents a turning radius of the vehicle 12, x r represents an x-position of the side monocamera 32, and y r represents a y-position of the side monocamera 32. The straight line equations L1 and L2 are expressed in equations 8 and 9 as follows:Once resolved after the overlap region A, the ideal delay distance module 50 of the one or more controllers 20 (FIG. 2 ) solves for the ideal delay distance s by solving a convex optimization problem based on a convex cost function that takes into account the steering angle θ. FIG. 6 is a graph illustrating the relationship between the convex cost function J|θ that takes the steering angle θ into account and the ideal deceleration distance s as a value of the steering angle θ, where an arrow 72 indicates an increasing value of the steering angle θ. As the steering angle θ increases, the ideal delay distance s, which is represented by the local minima 70 of the parabola of each convex cost function J|θ, decreases. It is also appreciated that as the value of the steering angle θ increases, the overall cost function also increases. This is because with steering angle θ (i.e., the vehicle 12 turns at a sharper angle), the overlap area A decreases.Referring to FIG. 2, when the ideal delay line module 50 of the one or more controllers 20 resolves after the ideal delay line s, the frame module 52 determines a number of camera frames between the current time step t and the previous time step t-δT based on the ideal delay line s, the number of frames indicating how many camera frames there are between the current time step t and the previous time step t-δT. The number of camera frames is determined based on Equation 10, which is: where v l is a longitudinal speed of the vehicle 12, s represents the ideal delay distance, FPS represents the frames per second of an object camera, and floor( ) represents an operator that returns the integer portion of a floating point number such that the number of frames is an integer.The flag module 54 of the one or more controllers 20 receives vehicle position information 78 as input. The flag module 54 of the one or more controllers 20 determines a heading direction of the vehicle 12 based on the vehicle position information 78 As explained below, the heading direction of the vehicle 12 indicates which monocamera 30, 32 is selected to provide a previous camera frame captured at the previous time step t- δT. The flag module 54 also updates an extrinsic matrix of either the front monocamera 30 or the side monocamera 32 based on the direction of travel of the vehicle 12, wherein updating the extrinsic matrix indicates that a previous camera frame is to be used in the image data captured in the previous time step t- δT when determining the three-dimensional coordinates of an object. Specifically, if the vehicle 12 is moving in the forward direction, the extrinsic matrix for the front monocamera 30 is updated. If the vehicle 12 is travelling in the rearward direction, then the extrinsic matrix for the side monocamera 32 is updated.The flag module 54 of the one or more controllers 20 may then send a flag 80 to either the front mono camera module 56 or the side mono camera module 58, depending on the direction of travel of the vehicle 12. In response to determining that the vehicle 12 is traveling in the forward direction, the flag module 54 sends the flag 80 to the front mono camera module 56, and the front mono camera module 56 selects the previous camera frame in the image data captured by the front mono camera 30 in the previous time step t- δT. The page mono camera module 58 selects the camera frame captured by the page mono camera 32 in the current time step t. In response to determining that the vehicle 12 is traveling in the rearward direction, the flag module 54 sends the flag 80 to the side monocamera module 58, and the side monocamera module 58 selects the previous camera frame in the image data captured by the side monocamera 32 in the previous time step t-δT. The front mono camera module 56 selects the camera frame captured by the front mono camera 30 in the current time step t.The feature matching module 60 of the one or more controllers 20 then performs feature matching to identify pixel positions in the camera frame captured by the front mono camera 30 and the camera frame captured by the side mono camera 32 that correspond to the same three-dimensional coordinates of an object to be evaluated. Specifically, u fi, v fi represent the horizontal and vertical pixel positions, respectively, of a corresponding matched feature captured by the front monocamera 30, and u ri, v ri represent the vertical and horizontal pixel positions of a corresponding matched feature captured by the side monocamera 32.The triangulation module 62 then triangulates the vertical and horizontal pixel positions captured by the front monocamera 30 and the side monocamera 32 to determine the three-dimensional coordinates of the object to be evaluated, the depth of the object being indicated by the three-dimensional coordinates.FIG. 7 is an exemplary process flow diagram illustrating a method 200 for determining the three-dimensional coordinates of an object by the monocular camera system 10. Referring generally to FIGS. 1-7, the method 200 may begin at block 202. In block 202, the ideal delay distance module 50 of the one or more controllers 20 collects image data captured by the front monocamera 30 and the side monocamera 32, the image data including an object to be evaluated. The method 200 may then proceed to block 204.In block 204, the ideal delay distance module 50 of the one or more controllers 20 determines the ideal delay distance s that the vehicle 12 travels between a current time step t and a previous time step t-δT while two asynchronous camera frames are captured by the front mono camera 30 and the side mono camera 32. The method 200 may then proceed to block 206.In block 206, the frame module 52 determines the number of delayed frames captured by either the front mono camera 30 or the side mono camera 32 between the current time step t and the previous time step t-δT based on the ideal delay distance s. Method 200 may then proceed to block 208.In block 208, the flag module 54 of the one or more controllers 20 determines the heading of the vehicle 12, where the heading of the vehicle 12 indicates which monocamera 30, 32 is selected to provide the previous camera frame captured in the previous time step t- δT. The method 200 may then proceed to decision block 210.At decision block 210, in response to determining that the vehicle is travelling in the forward direction, the method proceeds to block 212. In block 212, the flag module 54 selects the previous camera frame in the image data captured by the front mono camera 30 in the previous time step t-δT and the camera frame captured by the side mono camera in the current time step t. Method 200 may then proceed to block 216. Referring to decision block 210, in response to determining that the vehicle 12 is travelling in the rearward direction, the method proceeds to block 214. In block 214, the flag module 54 selects the previous camera frame in the image data captured by the side monocamera 32 in the previous time step t- δT and the camera frame captured by the front monocamera 30 in the current time step t. Method 200 may then proceed to block 216.In block 216, the feature matching module 60 of the one or more controllers 20 performs feature matching to identify pixel positions in the camera frame captured by the front mono camera 30 and the camera frame captured by the side mono camera 32 that correspond to the same three-dimensional coordinates of the object to be evaluated. Method 200 may then proceed to block 218.At block 218, the triangulation module 62 of the one or more controllers 20 triangulate the pixel positions captured by the front monocamera 30 and the side monocamera 32 to determine the three-dimensional coordinates of the object to be evaluated. The method 200 may then be ended.As can be seen from the figures, the disclosed monocular camera system offers various technical effects and advantages. Specifically, the monocular camera system uses current and temporary camera frames captured by two monocamers with non-overlapping fields of view to provide stereo camera depth estimation. The monocular camera system determines an ideal delay distance that the vehicle travels while two asynchronous camera frames are captured by the front mono camera and the side mono camera, wherein the ideal delay distance is optimized to maximize the range of overlap between the two fields of view captured by the two mono cameras while simultaneously minimizing an amount of delay between the current time step and the previous time step. It will be appreciated that the monocular camera system may estimate the depth of objects without using additional near range depth sensors such as LiDAR, ultrasonic sensors, or short range radars.The controllers may refer to or be part of an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor (shared, dedicated, or group) that executes code, or a combination of some or all of the above elements, such as a system-on-chip. In addition, the controllers may be based on a microprocessor, such as a computer having at least one processor, a memory (RAM and / or ROM), and associated input and output buses. The processor may operate under the control of an operating system residing in memory. The operating system may manage the computing resources such that computer program code embodied as one or more computer software applications, such as an application residing in memory, may include instructions executed by the processor. In an alternative embodiment, the processor may execute the application directly, in which case the operating system may be omitted.LegendIn the drawing figures, N represents No and Y represents Yes.

Claims

A monocular camera system (10) for a vehicle (12), the monocular camera system (10) comprising: a front monocamera (30) positioned at a front portion of the vehicle (12); a side monocamera (32) positioned along a side of the vehicle (12); and one or more controllers (20) in electronic communication with the front monocamera (30) and the side monocamera (32), the one or more controllers (20) executing instructions to: collect image data captured by the front monocamera (30) and the side monocamera (32), the image data including an object to be evaluated; determining an ideal delay distance that the vehicle (12) travels between a current time step and a previous time step while capturing two asynchronous camera frames from the front mono camera (30) and the side mono camera (32); determining a number of delayed frames captured from either the front mono camera (30) or the side mono camera (32) between the current time step and the previous time step based on the ideal delay distance; determining a heading direction of the vehicle (12), wherein the heading direction of the vehicle (12) indicates which mono camera is selected to provide a previous camera frame captured in the previous time step; performing feature matching to identify pixel positions in a camera frame captured by the front mono camera (30) and a camera frame captured by the side mono camera (32) corresponding to the same three-dimensional coordinates of the object to be evaluated; and perform triangulation of the pixel positions captured by the front mono camera (30) and the side mono camera (32) to determine the three-dimensional coordinates of the object to be evaluated.The monocular camera system (10) of claim 1, wherein the one or more controllers (20) execute instructions to: in response to determining that the vehicle (12) is driving in the forward direction, select the previous camera frame in the image data captured by the front mono camera (30) in the previous time step and the camera frame captured by the side mono camera (32) in the current time step.The monocular camera system (10) of claim 1, wherein the one or more controllers (20) execute instructions to: in response to determining that the vehicle (12) is travelling in the rearward direction, select the previous camera frame in the image data captured by the side monocamera (32) in the previous time step and the camera frame captured by the front monocamera (30) in the current time step.The monocular camera system (10) of claim 1, wherein the number of delayed frames indicates how many frames are counted back in the image data captured by either the front mono camera (30) or the side mono camera (32) between the current time step and the previous time step.The monocular camera system (10) of claim 1, wherein the number of delayed frames is determined based on: camera frames ≅ f l o o r ( | s v l × F P S | ) where v l is a longitudinal speed of the vehicle (12), s represents the ideal delay distance, FPS represents frames per second of an object camera, and floor() represents an operator returning the integer portion of a floating point number.The monocular camera system (10) of claim 1, wherein determining the ideal delay distance comprises: resolving, after a region of overlap, a front field of view captured by the front monocamera (30) and a lateral field of view captured by the side monocamera (32).The monocular camera system (10) of claim 6, wherein a steering angle of the vehicle (12) is zero and the area of overlap is determined by: A ∼ following following following ∼ x r l e ( L 1 ( x r, y r ) - L 2 ) d x ⇒ x r ∼ c t e A ∼ following following following following following x r l e ( L 1 ( y r ) - L 2 ) d x where A represents the area of overlap, L 1 represents a line equation defining the boundaries of a lateral field of view of the side monocamera (32), L 2 represents a line equation defining the boundaries of a front field of view of the front monocamera (30), x r represents an x position of the side monocamera (32), y r represents a y position of the side monocamera (32), l e is an upper limit of integration, cte represents a constant, x represents a lateral direction, and y represents the longitudinal direction.The monocular camera system (10) of claim 7, wherein the equation of lines defining the lateral field of view boundaries of the side monocamera (32) is solved as follows: L 1 = { tan ( F O V r 2 + ψ r ) ( x - x r ) + y r, f a l l s y ≥ y r - tan ( F O V r 2 + ψ r ) ( x - x r ) + y r, f a l l s y < y r where FOV r represents the lateral field of view for the side monocamera (32), and ψ r is a yaw angle for the side monocamera (32).The monocular camera system (10) of claim 7, wherein the equation of lines defining the boundaries of the front field of view of the front monocamera (30) is solved as follows: L 2 = { tan ( π 2 - F O V f 2 + ψ f ) x, f a l l s x ≥ 0 - tan ( π 2 - F O V f 2 + ψ f ) x, f a l l s x < 0 where FOV f represents the front field of view for the front monocamera (30), and ψ f is a yaw angle for the front monocamera (30).The monocular camera system (10) of claim 6, wherein determining the ideal delay distance comprises: solving a convex optimization problem by graphically representing a relationship between a convex cost function and the ideal delay distance, wherein the relationship between the convex cost function and the ideal delay distance is represented by a parabola; and selecting a local minimum of the parabola as the ideal delay distance.

Images