Camera misalignment detection system for vehicle

Abstract

A camera misalignment detection system that detects a misalignment condition of one or more cameras of a vehicle includes one or more controllers that determine a set of matching pixel pairs, determine a feature matching ratio based on the set of matching pixel pairs, and calculate an alignment angular difference for the one or more cameras. In response to determining that each of the feature match ratio, the point ratio in the essential matrix, and the alignment angle difference exceeds a respective threshold, the controller adds the alignment angle difference to a queue that includes a sequence of historical alignment angle difference values. The controller performs statistical filtering to determine a total number of historical alignment angle differences as inner points in the queue, and determines a misalignment state of the one or more cameras based on the total number of historical alignment angle differences as inner points in the queue.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a camera misalignment detection system that detects a misalignment condition of one or more cameras of a vehicle. BACKGROUND

[0002] A vehicle can utilize various types of perception sensors to collect perception data about the surrounding environment. One particular type of perception sensor that is commonly employed is a camera, which collects image data about the surrounding environment of the vehicle. The image data collected by the camera can be used for various vehicle systems, such as but not limited to, an autonomous driving system (ADS), an advanced driver assistance system (ADAS), and an automatic parking assist. However, when the camera is misaligned, problems can arise for vehicle systems that rely on the image data collected by the camera.

[0003] There are several camera misalignment detection algorithms currently available for detecting camera misalignment, however, some of these existing camera misalignment detection algorithms can have drawbacks. For example, one existing camera misalignment detection algorithm that uses only motion vectors can trigger false positive events when subjected to limited visibility conditions, such as nighttime driving and inclement weather, such as rainy or snowy days.

[0004] Accordingly, while current camera misalignment algorithms achieve their intended purpose, there is a need in the art for an improved method for detecting camera misalignment that maintains stability when subjected to limited visibility conditions. SUMMARY

[0005] According to aspects, a camera misalignment detection system for detecting a misalignment condition of one or more cameras of a vehicle is disclosed. The camera misalignment detection system includes one or more controllers in electronic communication with the one or more cameras to receive two image frames from the one or more cameras. The one or more controllers include one or more processors that execute instructions to determine a set of matching pixel pairs between the two image frames, where each matching pixel pair of the set of matching pixel pairs indicates a movement of a particular pixel between the two image frames. The one or more controllers determine a feature match ratio based on the set of matching pixel pairs, where the feature match ratio represents a ratio of a number of matching pixel pairs between the two image frames that are predicted to be within a threshold distance and are classified as inliers to a total number of matching pixel pairs between the two image frames. The one or more controllers compute an alignment angle difference of the one or more cameras based on an essential matrix of the one or more cameras and the set of matching pixel pairs between the two image frames. The one or more controllers compare the feature match ratio, the essential matrix inlier ratio, and the alignment angle difference to respective thresholds. In response to determining that each of the feature match ratio, the essential matrix inlier ratio, and the alignment angle difference exceeds the respective threshold, the one or more controllers determine that a potential misalignment condition of the one or more cameras exists and add the alignment angle difference to a queue that includes a sequence of historical alignment angle difference values. The one or more controllers perform statistical filtering to determine a total number of historical alignment angle difference values in the queue that are inliers. The one or more controllers determine a misalignment state of the one or more cameras based on the total number of historical alignment angle difference values in the queue that are inliers.

[0006] In another aspect, in response to determining the misalignment condition of the one or more cameras, the one or more controllers send a disable signal to one or more systems within the vehicle that rely on image data collected by the one or more cameras.

[0007] In yet another aspect, the alignment angle difference represents a motion difference between an image frame captured at a current timestamp by the one or more cameras and a historical motion of the one or more cameras.

[0008] In one aspect, the one or more processors of the one or more controllers execute instructions to evaluate historical results of pose recovery between two camera coordinate systems to determine a convergence condition, each of the two camera coordinate systems corresponding to one of the two image frames over a period of time, where the historical motion of the one or more cameras is represented by the historical results of pose recovery between the two camera coordinate systems, and where the potential misalignment condition of the one or more cameras is further determined based on the convergence condition.

[0009] In another aspect, a convergence condition is determined by confirming that a history of pose recovery between two camera coordinate systems includes a threshold sample size of image frames, each of the two camera coordinate systems corresponding to one of the two image frames.

[0010] In yet another aspect, the statistical filtering for determining the total number of historical alignment angle differences in the queue that are inliers includes a bandwidth-based loss function that evaluates all historical alignment angle differences in the queue to identify an ideal bandwidth.

[0011] In one aspect, the bandwidth-based loss function is expressed as:

[0012]

[0013] where b * represents the ideal bandwidth, b represents the bandwidth corresponding to each historical alignment angle difference in the queue, represents the historical alignment angle difference, i represents the index of one of the historical alignment angle differences in the queue, and j is different from the current historical alignment angle difference. Another historical index of the alignment angle difference.

[0014] On the other hand, statistical filtering for determining the total number of historical alignment angle differences in a queue that are inliers includes an angle-based loss function that evaluates all historical alignment angle differences in the queue to identify a dominant alignment angle difference, where the dominant alignment angle difference indicates the highest core density of all historical alignment angle differences in the queue.

[0015] In yet another aspect, the angle-based loss function is expressed as:

[0016]

[0017] in Indicates the main alignment angle difference, b * represents the ideal bandwidth, represents the historical alignment angle difference value, and i represents the index of one of the historical alignment angle difference values ​​in the queue.

[0018] In one aspect, a feature match ratio is determined by predicting a position in a subsequent image frame of a particular pixel of each matching pixel pair that is part of the set of matching pixel pairs based on the movement of the particular pixel indicated by the matching pixel pairs between the two image frames, the motion of the vehicle, the camera-to-vehicle alignment parameters, and the position of the feature represented by the particular pixel in three-dimensional space in the real world.

[0019] In another aspect, the location of a particular pixel in a subsequent image frame is represented by a two-dimensional probability distribution.

[0020] In another aspect, the feature match ratio is determined by determining a distance between the two-dimensional probability distribution representing the location of the particular pixel in the subsequent image frame and an actual location of the particular pixel indicated by each matching pixel pair of the subsequent image frame, wherein the actual location of the particular pixel is: part of the set of matching pixel pairs, and the feature match ratio is determined by comparing the distance between the two-dimensional probability distribution and the actual location of the particular pixel for each matching pixel pair that is part of the set of matching pixel pairs to a threshold distance value.

[0021] In one aspect, the distance between the two-dimensional probability distribution and the actual location of the particular pixel indicated by the subsequent image frame is represented by a Mahalanobis distance.

[0022] In another aspect, the two-dimensional probability distribution is a two-dimensional Gaussian distribution.

[0023] In yet another aspect, a method for detecting a misalignment condition of one or more cameras that are part of a vehicle is provided. The method includes determining, by one or more controllers in electronic communication with the one or more cameras, a set of matching pixel pairs between two image frames received by the one or more cameras, wherein each matching pixel pair in the set of matching pixel pairs indicates movement of a particular pixel between the two image frames. The method includes determining, by the one or more controllers, a feature match ratio based on the set of matching pixel pairs, wherein the feature match ratio represents a ratio of a number of matching pixel pairs between the two image frames that are predicted to be within a threshold distance value and classified as inliers to a total number of matching pixel pairs between the two image frames. The method includes calculating, by the one or more controllers, an alignment angle difference for the one or more cameras based on an essential matrix for the one or more cameras and the set of matching pixel pairs between the two image frames. The method includes comparing, by the one or more controllers, the feature match ratio, the essential matrix inlier ratio, and the alignment angle difference to respective threshold values. In response to determining that the feature match ratio, the essential matrix inlier ratio, and the alignment angle difference all exceed the respective threshold values, the method includes determining that a potential misalignment state of the one or more cameras exists and adding the alignment angle difference to a queue that includes a sequence of historical alignment angle difference values. The method includes performing, by the one or more controllers, statistical filtering to determine a total number of historical alignment angle difference values in the queue that are inliers. Finally, the method includes determining, by the one or more controllers, a misalignment state of the one or more cameras based on the total number of historical alignment angle difference values in the queue that are inliers.

[0024] In another aspect, in response to determining a misalignment condition of the one or more cameras, the method includes transmitting, by the one or more controllers, a disable signal to one or more systems within the vehicle that rely on image data collected by the one or more cameras.

[0025] In yet another aspect, a camera misalignment detection system for detecting a misalignment condition of one or more cameras of a vehicle is disclosed. The camera misalignment detection system includes one or more controllers in electronic communication with the one or more cameras to receive two image frames from the one or more cameras. The one or more controllers include one or more processors that execute instructions to determine a set of matching pixel pairs between the two image frames, where each matching pixel pair of the set of matching pixel pairs indicates a movement of a particular pixel between the two image frames. The one or more controllers determine a feature match ratio based on the set of matching pixel pairs, where the feature match ratio represents a ratio of a number of matching pixel pairs between the two image frames that are predicted to be within a threshold distance value and classified as inliers to a total number of matching pixel pairs between the two image frames. The one or more controllers compute an alignment angle difference of the one or more cameras based on an essential matrix of the one or more cameras and the set of matching pixel pairs between the two image frames, where the alignment angle difference represents a motion difference of the one or more cameras between an image frame captured at a current timestamp and a historical motion of the one or more cameras. The one or more controllers evaluate historical results of pose recovery between two camera coordinate systems each corresponding to one of the two image frames over a time period to determine a convergence condition, where the historical motion of the one or more cameras is represented by the historical results of pose recovery between the two camera coordinate systems each corresponding to one of the two image frames. The one or more controllers compare the feature match ratio, the essential matrix inlier ratio, and the alignment angle difference value to respective thresholds. In response to determining that each of the feature match ratio, the essential matrix inlier ratio, and the alignment angle difference value exceeds the respective threshold and the convergence condition exists, the one or more controllers determine that a potential misalignment condition of the one or more cameras exists, and add the alignment angle difference value to a queue that includes a sequence of historical alignment angle difference values. The one or more controllers perform statistical filtering to determine a total number of historical alignment angle difference values in the queue that are inliers. The one or more controllers determine a misalignment status of the one or more cameras based on the total number of historical alignment angle difference values in the queue that are inliers. In response to determining the misalignment status of the one or more cameras, the one or more controllers send a disable signal to one or more systems within the vehicle that rely on image data collected by the one or more cameras.

[0026] In another aspect, the statistical filtering to determine the total number of historical alignment angle difference values in the queue includes evaluating all historical alignment angle difference values in the queue to identify an ideal bandwidth based on a bandwidth-based loss function.

[0027] In yet another aspect, the statistical filtering for determining a total number of historical alignment angle differences in the queue that are inliers includes an angle-based loss function that evaluates all of the historical alignment angle differences in the queue to identify a dominant alignment angle difference, where the dominant alignment angle difference indicates a highest core density of all of the historical alignment angle differences in the queue.

[0028] based on a movement of the particular pixel between the two image frames indicated by the matching pixel pair, a motion of the vehicle, a camera-to-vehicle alignment parameter, and a location of a feature represented in three-dimensional space in the real world by the particular pixel, to predict a location of the particular pixel of each matching pixel pair that is part of the set of matching pixel pairs in a subsequent image frame, thereby determining a feature matching ratio.

[0029] Other applicational areas will become apparent from the description provided herein. It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0030] The drawings described herein are for purposes of illustration only and are not intended to limit the scope of the present disclosure in any way.

[0031] Figure 1 is a schematic diagram of a disclosed system camera misalignment detection system for a vehicle in accordance with one example embodiment, the system camera misalignment detection system including one or more controllers in electronic communication with one or more cameras;

[0032] Figure 2 is a block diagram illustrating a software architecture of the one or more controllers shown; Figure 1

[0033] Figure 3 is an illustration of an example subsequent image frame captured by the one or more cameras in accordance with one example embodiment, the example subsequent image frame including several exemplary two-dimensional probability distributions; and

[0034] Figure 4 is a process flow diagram illustrating a method for determining a misalignment condition of one or more cameras of a vehicle in accordance with one example embodiment. DETAILED DESCRIPTION

[0035] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

[0036] REFERENCE Figure 1 ​An exemplary camera misalignment detection system 10 for a vehicle 12 is shown. The camera misalignment detection system 10 includes one or more controllers 20 in electronic communication with one or more cameras 22 and one or more notification devices 24. The one or more cameras 22 are exterior-facing cameras that capture image data representative of the environment surrounding the vehicle 12. In Figure 1 In the non-limiting embodiment shown, the one or more cameras 22 include a forward-facing camera 22A, a rear-facing camera 22B, and two side-facing cameras 22C, however, it should be appreciated that fewer or additional cameras can also be included as part of the vehicle 12. Furthermore, although the vehicle 12 is shown as a sedan, the vehicle 12 can be any other type of automobile, such as a truck, sport utility vehicle, van, or recreational vehicle. Moreover, the vehicle 12 is not limited to an automobile and can be any type of rigid body that propels itself, such as an unmanned aerial vehicle (UAV), an airplane, or a water-borne conveyance such as a boat. Figure 1

[0037] As described below, the camera misalignment detection system 10 detects a misalignment condition of at least one of the one or more cameras 22 that are part of the vehicle 12. In response to detecting the misalignment of the at least one camera 22, the one or more controllers 20 instruct the one or more notification devices 24 to notify a user of the vehicle 12 about the misalignment condition of the one or more cameras 22. In Figure 1 In the non-limiting embodiment shown, the notification devices include a speaker 26 that generates an audio notification and a display 28 that generates a visual notification. However, it should be appreciated that other types of notification devices can also be used, such as a haptic device. The notification can instruct the user to visit an authorized dealership or other service center to check the misalignment condition of the one or more cameras 22.

[0038] The one or more controllers 20 also send a disable signal to one or more controllers 30 that are part of one or more systems within the vehicle 12 that rely on image data collected by the one or more cameras 22 in response to detecting the misalignment condition, where the disable signal disables or limits certain features. Some examples of systems that rely on image data collected by the one or more cameras 22 include, but are not limited to, an autonomous driving system (ADS), an advanced driver assistance system (ADAS), and an automatic parking assist.

[0039] Figure 2 is a block diagram showing a software architecture for the one or more controllers 20. The one or more controllers 20 include a feature prediction and matching module 40, a feature matching evaluation module 42, an essential matrix module 44, a pose recovery module 46, a visual filter module 50, a statistical filter module 52, and a decision module 54.

[0040] ​The feature prediction and matching module 40 of the one or more controllers 20 receives two image frames from the one or more cameras 22 of the vehicle 12, where the two image frames each include a unique timestamp and capture common features in the environment surrounding the vehicle 12. The two image frames need not be consecutive, however, both image frames include the same objects located in the environment surrounding the vehicle 12.

[0041] The feature prediction and matching module 40 of the one or more controllers 20 performs feature matching to determine a set of matching pixel pairs between the two image frames, where each matching pixel pair that is part of the set of matching pixel pairs indicates movement of a particular pixel between the two image frames. The particular pixel represents a point location of a feature located in the environment surrounding the vehicle 12. The feature can be, for example, an identifiable point on an object located in the environment, a road surface of a roadway, another vehicle, or vegetation such as a tree or bush. Each matching pixel pair represents two locations of the particular pixel as the pixel moves between the two image frames, and the set of matching pixel pairs represents all matching features included within the two image frames.

[0042] The feature matching evaluation module 42 receives the set of matching pixel pairs between the two image frames from the feature prediction and matching module 40 of the one or more controllers 20 and determines a feature matching ratio r1 based on the set of matching pixel pairs. The feature matching evaluation module 42 of the one or more controllers 20 determines the feature matching ratio r1 by first predicting a location of the particular pixel in a subsequent image frame that is part of the set of matching pixel pairs based on movement of the particular pixel indicated by the matching pixel pairs between the two image frames, motion of the vehicle 12, camera-to-vehicle alignment parameters, and a location of the feature represented by the particular pixel in the real world in three-dimensional space. The location of the particular pixel in the subsequent image frame is represented by a two-dimensional probability distribution 70, as shown in Figure 3 .

[0043] Figure 3 An exemplary subsequent image frame 72 captured by the one or more cameras 22 of the vehicle 12 Figure 1 ) is shown, which includes several exemplary two-dimensional probability distributions 70. In the exemplary embodiment shown in Figure 3 each two-dimensional probability distribution 70 is a two-dimensional Gaussian distribution that includes a probability threshold represented by an ellipse 74. The ellipse 74 includes a boundary 76, where an area A enclosed by the boundary 76 of the ellipse 74 represents an area in which the location of the particular pixel is within the probability threshold. Although Figure 3 each two-dimensional probability distribution 70 is shown as a Gaussian distribution, it should be understood that other types of statistical distributions can also be used.

[0044] Reference is made to Figure 2 and Figure 3, the feature matching evaluation module 42 of the one or more controllers 20 determines a distance between the two-dimensional probability distribution 70 and the actual location of the particular pixel, the actual location of the particular pixel being indicated by the subsequent image frame of each matching pixel pair that is part of the set of matching pixel pairs. The distance between the two-dimensional probability distribution 70 and the actual location of the particular pixel indicated by the subsequent image frame is represented by a Mahalanobis distance.

[0045] The feature matching evaluation module 42 of the one or more controllers 20 determines the feature matching ratio r1 by comparing the distance between the two-dimensional probability distribution 70 and the actual location of the particular pixel of each matching pixel pair that is part of the set of matching pixel pairs to a threshold distance value, and classifying the matching pixel pair as an inlier if the distance is less than the threshold distance value. The threshold distance value is determined based on a target accuracy of the one or more cameras 22, and can vary based on the particular application. In embodiments, the threshold distance value is determined based on empirical data collected during testing. The feature matching ratio r1 represents the matching pixel pairs between the two image frames that are predicted to be within the threshold distance value, and are classified as inliers relative to the total number of matching pixel pairs between the two image frames, and is represented in Equation 1 as:

[0046]

[0047] where d M represents the distance between the two-dimensional probability distribution 70 and the actual location of the particular pixel of each matching pixel pair that is part of the set of matching pixel pairs, ||{}|| represents a distance based on the distance d M The total number of matching pixel pairs between the two image frames is counted, and θ m represents the threshold distance value.

[0048] With reference to Figure 2 , the essential matrix module 44 receives the set of matching pixel pairs between the two image frames from the feature prediction and matching module 40 of the one or more controllers 20, and determines an essential matrix E of the one or more cameras 22 based on the set of matching pixel pairs between the two image frames. The essential matrix E indicates a movement of the one or more cameras 22 between the unique timestamps of the two image frames. In addition to the essential matrix E, the essential matrix module 44 of the one or more controllers 20 determines an essential matrix inlier ratio r2 based on the set of matching pixel pairs between the two image frames. The essential matrix inlier ratio r2 represents a number of matching pixel pairs that satisfy an epipolar geometry condition. The epipolar geometry condition evaluates a distance between a point and an epipolar line projected by a corresponding matching point in the remaining image frames of the two image frames, where the epipolar geometry condition is satisfied when the epipolar distance is less than a predetermined epipolar threshold distance.

[0049] The pose recovery module 46 receives the essential matrix E of the one or more cameras 22 from the essential matrix module 44 of the one or more controllers 20 and determines an alignment angle difference A of the one or more cameras 22 based on the set of matching pixel pairs between the two image frames and the essential matrix E of the one or more cameras 22. The alignment angle difference A represents a motion difference of the one or more cameras 22 between the image frames captured at the current timestamp and the historical motion of the one or more cameras 22. The pose recovery module 46 of the one or more controllers 20 determines the motion of the one or more cameras 22 at the current timestamp by performing a camera pose recovery algorithm to recover the motion of the one or more cameras 22 between the unique timestamps of the two image frames based on the set of matching pixel pairs between the two image frames and the essential matrix E of the one or more cameras 22. The motion of the one or more cameras 22 at the current timestamp is represented by a relative camera rotation and translation and includes six degrees of freedom.

[0050] The historical motion of the one or more cameras 22 is represented by the historical results of the pose recovery between two camera coordinate systems, each corresponding to one of the two image frames collected over a period of time. In an embodiment, the period of time lasts for at least about 30 seconds. It should be appreciated that the pose recovery module 46 of the one or more controllers 20 can perform various data operations, such as noise removal and averaging of data representing the cumulative motion of the camera 22, to ensure that the historical results of the pose recovery between two camera coordinate systems each corresponding to one of the two image frames are reliable and stable.

[0051] The pose recovery module 46 evaluates the historical results of the pose recovery between two camera coordinate systems each corresponding to one of the two image frames to determine a convergence condition. Specifically, determining the convergence condition includes confirming that the historical results of the pose recovery include a threshold sample size of image frames. The threshold sample size is selected to ensure that the historical results of the pose recovery are reliable and stable. In one embodiment, the threshold sample size is about 1000 image frames. In response to determining that the historical results of the pose recovery include the threshold sample size, the pose recovery module 46 then evaluates the historical results of the pose recovery to confirm that a value of a motion difference between a most recent image frame and the historical motion representing all image frames that are part of the historical results of the pose recovery decreases over the period of time. The value of the motion difference decreasing over time can be determined based on a cumulative average method or a moving window average method. In response to confirming that the historical results of the pose recovery include the threshold sample size and that the value of the motion difference between the most recent image frame and the historical motion representing all image frames that are part of the historical results of the pose recovery decreases over the period of time, the pose recovery module 46 determines that the convergence condition exists.

[0052] With continued reference to Figure 2The visual filtering module 50 of one or more controllers 20 receives the feature matching ratio r1′ from the feature matching evaluation module 42, the intrinsic matrix interior point ratio r2′ from the intrinsic matrix module 44, the alignment angle difference A′ from the posture recovery module 46, and the convergence condition from the posture recovery module 46. Figure 4 As explained in the flowchart shown, the visual filtering module 50 of the one or more controllers 20 compares the feature matching ratio r1, the intrinsic matrix inlier ratio r2, and the alignment angle difference A to respective thresholds. In response to determining that the feature matching ratio r1, the intrinsic matrix inlier ratio r2, and the alignment angle difference A all exceed their respective thresholds, and in response to the posture recovery module 46 determining that a convergence condition exists, the visual filtering module 50 of the one or more controllers 20 then determines that a potential misalignment condition exists for the one or more cameras 22 and adds the alignment angle difference A to a queue comprising a sequence of historical alignment angle difference values ​​based on the historical motion of the one or more cameras 22. The statistical filtering module 52 of the one or more controllers 20 then performs statistical filtering to determine the total number of historical alignment angle difference values ​​in the queue that are inliers, and determines the misalignment state of the one or more cameras 22 based on the total number of historical alignment angle difference values ​​in the queue that are inliers.

[0053] Figure 4 is a process flow diagram illustrating a method 400 for determining a misalignment condition of one or more cameras 22 based on a feature matching ratio r1, an intrinsic matrix interior point ratio r2, an alignment angle difference A, and a convergence condition. Figures 1-4 , method 400 may begin at block 402. In block 402, the visual filtering module 50 of the one or more controllers 20 compares the alignment angle difference A to an angle difference threshold. In response to determining that the alignment angle difference A is less than or equal to the angle difference threshold, method 400 terminates. The angle difference threshold is selected to indicate a camera misalignment condition and is determined based on the target accuracy of the one or more cameras 22 and varies based on the specific application.

[0054] In response to determining that the alignment angle difference A is greater than the angle difference threshold, method 400 proceeds to block 404. In block 404, visual filter module 50 of one or more controllers 20 compares intrinsic matrix inlier ratio r2 to the intrinsic matrix inlier ratio threshold. The intrinsic matrix inlier ratio threshold is selected based on the number of matching pixel pairs between the two image frames that satisfy the epipolar geometry condition. In response to determining that the intrinsic matrix inlier ratio r2 is less than or equal to the intrinsic matrix inlier ratio threshold, method 400 terminates.

[0055] In response to determining that the epipolar matrix inlier ratio r2 is greater than the epipolar matrix inlier ratio threshold, the method 400 proceeds to block 406. In block 406, the vision filtering module 50 of the one or more controllers 20 compares the feature match ratio r1 to a threshold feature match ratio value. The threshold feature match ratio value is selected based on a number of matching pixel pairs between two image frames predicted to be within a threshold distance value that are classified as inliers when the one or more cameras 22 are not misaligned. In response to determining that the feature match ratio r1 is less than or equal to the threshold feature match ratio value, the method 400 terminates.

[0056] In response to determining that the feature match ratio r1 is greater than the threshold feature match ratio value, the method 400 proceeds to block 408. In block 408, in response to the vision filtering module 50 receiving an indication from the pose recovery module 46 that the convergence condition exists, the method 400 can then proceed to block 410. Otherwise, the method 400 terminates.

[0057] In block 410, the vision filtering module 50 of the one or more controllers 20 determines that a potential misalignment condition of the one or more cameras 22 exists and adds the alignment angle difference A to a queue that includes a sequence of historical alignment angle difference values based on historical motion of the one or more cameras 22. It should be appreciated that the queue maintains a predetermined number of historical alignment angle difference values. The predetermined number of historical alignment angle difference values is selected to provide stable results when the statistical filtering module 52 performs statistical filtering to determine a number of inlier alignment angle differences within the queue. The method 400 can then proceed to block 412.

[0058] In blocks 412 and 414, the statistical filtering module 52 of the one or more controllers 20 performs statistical filtering to determine a total number of historical alignment angle difference values in the queue that are inliers. In the described embodiment, kernel density estimation (KDE) is used to determine inliers, however, it should be appreciated that other statistical filtering methods can also be used, such as a fixed distribution model based on existing knowledge, or a classifier based on supervised learning.

[0059] Referring to block 412, the statistical filtering module 52 of the one or more controllers 20 evaluates all of the historical alignment angle difference values in the queue based on a bandwidth-based loss function to identify an ideal bandwidth. The ideal bandwidth maximizes the bandwidth-based loss function and also maximizes the probability of identifying a principal angle difference when all of the historical alignment angle difference values in the queue are evaluated, which is described in block 414. The bandwidth-based loss function is represented in Equation 2 as:

[0060]

[0061] where b * represents the ideal bandwidth, b represents a bandwidth corresponding to each historical alignment angle difference value in the queue, represents a history alignment angle difference value, i represents an index of one history alignment angle difference value in the queue, and j represents an index of another history alignment angle difference value other than the current history alignment angle difference Method 400 can then proceed to block 414.

[0062] In block 414, the statistical filtering module 52 of the one or more controllers 20 evaluates all of the history alignment angle difference values in the queue to identify a primary alignment angle difference value based on an angle-based loss function, where the primary alignment angle difference value indicates the highest core density of all of the history alignment angle difference values in the queue and maximizes the angle-based loss function. The angle-based loss function is represented in Equation 3 as:

[0063]

[0064] where represents the primary alignment angle difference value. Method 400 can then proceed to block 416.

[0065] In block 416, the statistical filtering module 52 of the one or more controllers 20 calculates a total number of history alignment angle difference values that are within a predetermined range of the primary alignment angle difference value, where the history alignment difference values that fall within the predetermined range of the primary alignment angle difference value are inliers. The predetermined range is based on a target accuracy of the one or more cameras 22 and can vary based on the particular application. The statistical filtering module 52 of the one or more controllers 20 can then calculate an inlier ratio r3, which represents the total number of history alignment angle difference values in the queue that are inliers, to the total number of history alignment angle difference values in the queue. Method 400 can then proceed to block 418.

[0066] In block 418, the decision module 54 of the one or more controllers 20 compares the inlier ratio r3 to a threshold inlier. The threshold inlier 3 is selected to ensure that a sufficient number of history alignment angle difference values in the queue are classified as inliers and that subsequent misalignment conditions of the one or more cameras 22 are not intermittent or exist in only a small number of image frames due to conditions other than camera misalignment. In response to determining that the inlier ratio r3 is less than or equal to the threshold inlier 3, the decision module 54 determines that the misalignment condition of the one or more cameras 22 has not occurred and method 400 terminates.

[0067] In response to determining that the inlier r3 is greater than the threshold inlier 0 3, the decision module 54 determines that an out-of- alignment condition of the one or more cameras 22 has occurred, and the method 400 can proceed to block 420. In block 420, the decision module 54 of the one or more controllers 20 instructs the one or more notification devices 24 to notify a user of the vehicle 12 about the out-of- alignment condition of the one or more cameras 22. The decision module 54 of the one or more controllers 20 also sends a disable signal to the one or more controllers 30 that are part of one or more systems within the vehicle 12 that rely on image data collected by the one or more cameras 22. The method 400 can then terminate.

[0068] With general reference to the figures, the disclosed camera misalignment detection system has various technical effects and benefits. Specifically, the disclosed camera misalignment detection system relies on feature matching between two image frames, an essential matrix of the one or more cameras, an angle of alignment difference of the one or more cameras, a convergence condition, and statistical filtering to determine an out-of- alignment condition with the one or more cameras of the vehicle. It should be appreciated that the disclosed method for detecting misalignment results in improved accuracy and can trigger fewer false positive misalignment conditions compared to existing camera misalignment detection systems, especially when the vehicle is experiencing limited visibility conditions (e.g., nighttime driving and inclement weather).

[0069] A controller can refer to an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor executing code (shared, dedicated, or group), or a combination of some or all of the above, or as a portion of it, such as in a system on a chip. Additionally, a controller can be microprocessor-based, such as a computer having at least one processor, memory (RAM and / or ROM), and associated input and output buses. The processor can operate under the control of an operating system that resides in the memory. The operating system can manage computer resources so that computer program code embodied as one or more computer software applications (e.g., applications that reside in the memory) can have instructions executed by the processor. In alternative embodiments, the processor can execute the applications directly, in which case an operating system can be omitted.

[0070] The description of the present disclosure is merely exemplary in nature and variations that do not depart from the essence of the present disclosure are intended to be within the scope of the present disclosure. Such variations are not to be regarded as a departure from the spirit and scope of the present disclosure.

Claims

1. A camera misalignment detection system for detecting a misalignment condition of one or more cameras of a vehicle, the camera misalignment detection system comprising: one or more controllers in electronic communication with the one or more cameras to receive two image frames from the one or more cameras, the one or more controllers comprising one or more processors that execute instructions to: determine a set of matching pixel pairs between the two image frames, wherein each matching pixel pair of the set of matching pixel pairs indicates movement of a particular pixel between the two image frames; determine a feature match ratio based on the set of matching pixel pairs, wherein the feature match ratio represents a ratio of a number of matching pixel pairs between the two image frames that are predicted to be within a threshold distance value and classified as inliers, to a total number of matching pixel pairs between the two image frames; compute an alignment angle difference of the one or more cameras based on an essential matrix of the one or more cameras and the set of matching pixel pairs between the two image frames; compare the feature match ratio, the essential matrix inlier ratio, and the alignment angle difference to respective thresholds; in response to determining that each of the feature match ratio, the essential matrix inlier ratio, and the alignment angle difference exceeds the respective threshold, determine that there is a potential misalignment condition of the one or more cameras and add the alignment angle difference to a queue comprising a sequence of historical alignment angle difference values; perform statistical filtering to determine a total number of historical alignment angle difference values in the queue that are inliers; and determine the misalignment condition of the one or more cameras based on the total number of historical alignment angle difference values in the queue that are inliers.

2. The camera misalignment detection system of claim 1, wherein the one or more processors of the one or more controllers execute instructions to: in response to determining the misalignment condition of the one or more cameras, send a disable signal to one or more systems within the vehicle that rely on image data collected by the one or more cameras.

3. The camera misalignment detection system of claim 1, wherein the alignment angle difference represents a motion difference between an image frame captured at a current timestamp by the one or more cameras and a historical motion of the one or more cameras.

4. The camera misalignment detection system of claim 3, wherein the one or more processors of the one or more controllers execute instructions to: evaluate historical results of pose recovery between two camera coordinate systems to determine a convergence condition, each of the two camera coordinate systems corresponding to one of the two image frames over a period of time, wherein the historical motion of the one or more cameras is represented by the historical results of the pose recovery between the two camera coordinate systems, and wherein the potential misalignment condition of the one or more cameras is determined based further on the convergence condition.

5. The camera misalignment detection system of claim 4, wherein the convergence condition is determined by: Confirming the historical results of the pose recovery between the two camera coordinate systems includes a threshold sample size of image frames, each of the two camera coordinate systems corresponding to one of two image frames.

6. The camera misalignment detection system of claim 1, wherein the statistical filtering to determine the total number of historical alignment angle difference values in the queue that are inliers includes: evaluating all historical alignment angle difference values in the queue to determine an ideal bandwidth according to a bandwidth-based loss function.

7. The camera misalignment detection system of claim 6, wherein the bandwidth-based loss function is expressed as: where b * represents the ideal bandwidth, b represents a bandwidth corresponding to each of the historical alignment angle difference values in the queue, represents the historical alignment angle difference value, i represents an index of one of the historical alignment angle difference values in the queue, and j represents an index of another historical alignment angle difference value different from the current historical alignment angle difference value.

8. The camera misalignment detection system of claim 6, wherein the statistical filtering to determine the total number of historical alignment angle difference values in the queue that are inliers includes: evaluating all the historical alignment angle difference values within the queue according to an angle-based loss function to determine a primary alignment angle difference, wherein the primary alignment angle difference indicates a highest core density of all the historical alignment angle difference values in the queue.

9. The camera misalignment detection system of claim 8, wherein the angle-based loss function is expressed as: wherein denotes the main alignment angle difference, b * denotes the ideal bandwidth, denotes a historical alignment angle difference value, and i denotes an index of one of the historical alignment angle difference values in the queue.

10. The camera misalignment detection system of claim 1, wherein the feature matching ratio is determined by predicting a location of the particular pixel of each matching pixel pair that is part of the set of matching pixel pairs in a subsequent image frame based on a movement of the particular pixel indicated by the matching pixel pairs between the two image frames, a motion of the vehicle, camera-to-vehicle alignment parameters, and a location of a feature represented in three-dimensional space in the real world by the particular pixel.

Images