Method for detecting a change in camera position on moving vehicle parts

Abstract

A system for detecting sensor position changes (DCPC) on moving vehicle components comprises two or more sensors on at least one of the moving vehicle components. The sensors detect optical information within a specific field of view (FOV) of the environment surrounding the vehicle. The specific FOV of each sensor overlaps at least partially with the FOV of at least one other sensor. The system includes controllers that execute an application for the DCPC.The DCPC application acquires overlapping optical information from the sensors, calculates a conditional correspondence probability distribution for feature points in the overlapping optical information, calculates a normalized common entropy of the feature points, determines that the sensor is in a position that can be handled by the DCPC, continuously monitors the positions of each of the sensors, selectively dynamically aligns the sensors, and ensures that the sensors are calibrated for vehicle perception tasks, which include advanced driver assistance systems (ADAS) functions.

Description

introduction

[0001] The present disclosure relates to cameras mounted on vehicles, and more particularly to cameras mounted on movable vehicle components.

[0002] In modern vehicles, timely detection of displacement of moving vehicle components such as moving mirrors or tailgates is important, as sensors are often mounted on the moving vehicle components. Such sensors are increasingly used in advanced driver assistance systems (ADAS) and in automated driving processes in general. In an event such as an impact with a sensor housing or sensor housing mount, the view of the attached sensor can change significantly. If such displacement is not addressed in a timely and accurate manner, the sensor will provide inaccurate information to the vehicle's sensing and perception system. Similarly, some sensors are mounted on vehicle components that are automated in motion, such as automatically folding mirrors and the like.If the process for an automated movement fails partially or completely, the sensors may provide similarly inaccurate information to the vehicle's sensing and perception systems.

[0003] While current systems and methods for detecting camera positions achieve their intended purpose, there is a need for a new and improved system and method for detecting changes in the position of cameras or sensors mounted on moving vehicle parts that can be applied to detect changes in the positions of cameras mounted on a variety of moving vehicle parts in a variety of different vehicle applications and that can be retrofitted to existing vehicles or installed in new vehicles.Furthermore, there is a need for systems and methods for detecting position changes of cameras or sensors arranged on moving vehicle parts that can operate continuously without significantly increasing computational loads and without increasing system or component complexity, that improve system functionality and resilience, that improve robustness, increase redundancy, and improve customer satisfaction. Summary

[0004] According to several aspects of the present disclosure, a system for detecting a sensor position change on movable vehicle components includes a vehicle having one or more movable vehicle components. The system further includes two or more sensors disposed on at least one of the one or more movable vehicle components. Each of the two or more sensors detects optical information within a particular field of view (FOV) within an environment surrounding the vehicle. The particular FOV of each of the two or more sensors at least partially overlaps with an FOV of at least one other sensor. The system further includes one or more controllers. Each of the one or more controllers includes a processor, memory, and one or more input / output (I / O) ports. The I / O ports communicate with the two or more sensors.The memory stores programmatic control logic. The processor executes the programmatic control logic, which includes an application for detecting a sensor position change (DCPC). The DCPC comprises at least first, second, third, fourth, and fifth control logic. The first control logic acquires overlapping optical information from the two or more sensors. The second control logic calculates a conditional correspondence probability distribution for feature points in the overlapping optical information. The third control logic calculates a normalized joint entropy of the feature points in the overlapping optical information. The fourth control logic determines that the sensor is in a position that can be handled by the DCPC.The fifth control logic continuously monitors positions of each of the two or more sensors, selectively dynamically aligning the sensors and ensuring that the two or more sensors are properly calibrated for vehicle perception tasks that include advanced driver assistance system (ADAS) features.

[0005] In another aspect of the present disclosure, the first control logic also includes control logic that acquires optical information from at least one wide-angle satellite camera (YSAT) having a field of view of approximately 180° and acquires optical information, at least partially overlapping the YSAT field of view, from at least one perception satellite camera (PSAT) having a field of view of approximately 20° to about 100°. The movable vehicle components include one or more of: one or more exterior vehicle mirrors, one or more doors, a trunk, and a liftgate.

[0006] In yet another aspect of the present disclosure, the first control logic further comprises control logic that performs visual feature extraction using one or more hand-crafted and learning-based methods including one or more of Self-sUPERvised Interest Point Detection and Description (SUPERPOINT), Learned Invariant Feature Transform (LIFT), Scale Invariant Feature Transform (SIFT), and Oriented Fast and Rotated Brief (ORB).

[0007] In yet another aspect of the present disclosure, the second control logic further comprises control logic to calculate a conditional correspondence distribution for overlapping feature points by: p(xj'|xi)∝exp(−dij−dN(xi)λdN(xi)), subject to: ∑j=1n'p(xj'|xi)=1; where xj' and x i Feature points of images are each captured by separate sensors, dij=dist(des(xi),des(xj')) is a predefined distance metric relative to a Euclidean distance between descriptors of the points xj' and x i increases monotonically, dN(xi)=minj(dij) and a nearest neighbor of point x i in the feature space when the predefined metric is given, and λ is a tunable inverse scale parameter of an exponential probability distribution.

[0008] In yet another aspect of the present disclosure, the third control logic further comprises control logic for determining a normalized joint entropy from the conditional correspondence distribution for each of the overlapping feature points xj',xi etc. to be calculated according to: H(C,C')=1η∑i=1n∑j=1n'p(xi)p(xj'|xi)log(p(xi)p(xj'|xi)), where η = log(nn') is the maximum joint entropy and p(x i) is a uniform distribution when prior information is not available.

[0009] In yet another aspect of the present disclosure, the DCPC further includes control logic for applying temporal smoothing to outputs of the second and third control logic. The temporal smoothing takes unexpected vehicle movements into account.

[0010] In yet another aspect of the present disclosure, the fourth control logic further includes control logic for performing a threshold check. The threshold check compares a position of FOVs of the two or more sensors to a library of calibration values.

[0011] In yet another aspect of the present disclosure, the fourth control logic includes control logic that, when it is determined that a position of a sensor is shifted from an expected position by an amount greater than or equal to a threshold, causes the DCPC to generate a notification to a vehicle operator, and, when it is determined that the position of the sensor is shifted from the expected position by an amount less than the threshold, causes the DCPC to dynamically align the sensor.

[0012] In yet another aspect of the present disclosure, when it is determined that the position of a sensor is displaced from an expected position by an amount greater than or equal to the threshold, the DCPC generates outputs including setting a code in the vehicle's memory, sending a code to a service center via a wired or wireless connection, and disabling ADAS features involving the sensor that is out of position by an amount greater than or equal to the threshold.

[0013] In yet another aspect of the present disclosure, the fifth control logic further comprises selectively dynamically aligning the sensors by applying the conditional correspondence probability distribution for each feature point xj' and x i , determining a Euclidean distance d ij between the corresponding feature points xi' and x i and generating a calibration for the sensors that detects position discrepancies between corresponding feature points xi' and x i taken into account in the FOVs of each of the sensors.

[0014] According to several additional aspects of the present disclosure, a method for detecting a sensor position change on movable vehicle components comprises detecting optical information with two or more sensors disposed on one or more movable vehicle components of a vehicle. Each of the two or more sensors has a particular field of view (FOV) of an environment surrounding the vehicle. The particular FOV of each of the two or more sensors at least partially overlaps with an FOV of at least one other sensor. The method further comprises executing, via one or more controllers, programmatic control logic comprising a sensor position change detection application (DCPC). Each of the one or more controllers comprises a processor, memory, and one or more input / output (I / O) ports. The I / O ports are in communication with the two or more sensors.The memory stores the programmatic control logic, and the processor executes the programmatic control logic, including the DCPC. The DCPC includes control logic for: acquiring overlapping optical information from the two or more sensors; calculating a conditional correspondence probability distribution for feature points in the overlapping optical information; calculating a normalized joint entropy of the feature points in the overlapping optical information; determining that the sensor is in a position tractable by the DCPC; and continuously monitoring positions of each of the two or more sensors, selectively dynamically aligning the sensors and ensuring that the two or more sensors are properly calibrated for vehicle perception tasks including advanced driver assistance system (ADAS) features.

[0015] In yet another aspect of the present disclosure, the method further comprises acquiring optical information from at least one wide angle satellite camera (YSAT) having a field of view of approximately 180° and acquiring optical information at least partially overlapping the YSAT field of view from at least one perception satellite camera (PSAT) having a field of view of approximately 20° to about 100°.

[0016] In yet another aspect of the present disclosure, the method further comprises performing visual feature extraction using one or more hand-crafted and learning-based methods including one or more of Self-sUPERvised Interest Point Detection and Description (SUPERPOINT), Learned Invariant Feature Transform (LIFT), Scale Invariant Feature Transform (SIFT), and Oriented Fast and Rotated Brief (ORB).

[0017] In yet another aspect of the present disclosure, the method further comprises calculating a conditional correspondence distribution for overlapping feature points by: p(xj'|xi)∝exp(−dij−dN(xi)λdN(xi)), subject to: ∑j=1n'p(xj'|xi)=1; where xj' and x i Feature points of images are each captured by separate sensors, dij=dist(des(xi),des(xj')) is a predefined distance metric relative to a Euclidean distance between descriptors of the points xj' and x i increases monotonically, dN(xi)=minj(dij) and a nearest neighbor of point x i in the feature space when a predefined metric is given, and λ is a tunable inverse scale parameter of an exponential probability distribution.

[0018] In yet another aspect of the present disclosure, the method further comprises calculating a normalized joint entropy from the conditional correspondence distribution for each of the overlapping feature points xj′,xi etc. according to: H(C,C′)=−1η∑i=1n∑j=1n′p(xi)p(xj′|xi)log(p(xi)p(xj′|xi)), where η = log(nn') is the maximum joint entropy and p(x i ) is a uniform distribution when prior information is not available, and a selective dynamic alignment of the sensors by applying the conditional correspondence probability distribution for each feature point xj′ and x i , determining the distance d ij between the corresponding feature points xi′ and x i and initiating a calibration for the sensors, the position discrepancies between corresponding feature points xi′ and xi taken into account in the FOVs of each of the sensors.

[0019] In yet another aspect of the present disclosure, the method further comprises applying temporal smoothing to outputs of normalized entropy and conditional correspondence distribution calculations, wherein the temporal smoothing takes unexpected vehicle movements into account.

[0020] In yet another aspect of the present disclosure, the method further comprises performing a threshold check. The threshold check compares a position of FOVs of the two or more sensors to a library of calibration values.

[0021] In yet another aspect of the present disclosure, the method further comprises, when it is determined that a position of a sensor is shifted from an expected position by an amount greater than or equal to a threshold, causing the DCPC to generate a notification to a vehicle operator, and, when it is determined that the position of the sensor is shifted from the expected position by an amount less than the threshold, causing the DCPC to dynamically align the sensor.

[0022] In yet another aspect of the present disclosure, the method further comprises, when it is determined that the position of a sensor is displaced from an expected position by an amount greater than or equal to the threshold, causing the DCPC to generate outputs including setting a code in the vehicle's memory, sending a code to a service center via a wired or wireless connection, and disabling ADAS features involving the sensor that is out of position by an amount greater than or equal to the threshold.

[0023] In several additional aspects of the present disclosure, a method for detecting a sensor position change on movable vehicle components comprises detecting optical information with two or more sensors disposed on one or more movable vehicle components of a vehicle. Each of the two or more sensors has a particular field of view (FOV) of an environment surrounding the vehicle. The particular FOV of each of the two or more sensors at least partially overlaps with an FOV of at least one other sensor. The method further comprises executing, via one or more controllers, programmatic control logic including a sensor position change detection application (DCPC). Each of the one or more controllers comprises a processor, memory, and one or more input / output (I / O) ports. The I / O ports are in communication with the two or more sensors.The memory stores the programmatic control logic. The processor executes the programmatic control logic, including executing the DCPC. The DCPC includes control logic for acquiring overlapping optical information from the two or more sensors, including: acquiring optical information from at least one wide-angle satellite camera (YSAT) having a field of view of approximately 180°; and acquiring optical information, at least partially overlapping the YSAT field of view, from at least one perception satellite camera (PSAT) having a field of view of approximately 20° to about 100°.The DCPC further includes control logic for performing visual feature extraction using one or more handcrafted and learning-based methods including one or more of Self-supervised Interest Point Detection and Description (SUPERPOINT), Learned Invariant Feature Transform (LIFT), Scale Invariant Feature Transform (SIFT), and Oriented Fast and Rotated Brief (ORB). The DCPC further includes control logic for calculating a conditional correspondence distribution for overlapping feature points in the overlapping optical information by: p(x′j|xi)∝exp(−dij−dN(xi)λdN(xi)), subject to: ∑j=1n′p(x′j|xi)=1; where x′j and x i Feature points of images are each captured by separate sensors, dij=dist(des(xi),des(x′j)) is a predefined distance metric relative to a Euclidean distance between descriptors of the points x′j and x i increases monotonically, dN(xi)=minjdij and a nearest neighbor of point x i in the feature space, given a predefined metric, and λ is a tunable inverse scale parameter of an exponential probability distribution. The DCPC further comprises control logic for calculating a normalized joint entropy from the conditional correspondence distribution for each of the overlapping feature points x′j,xi etc. according to: H(C,C′)=−1η∑i=1n∑j=1n′p(xi)p(x′j|xi)log(p(xi)p(x′j|xi)), where η = log(nn') is the maximum joint entropy and p(x i) is a uniform distribution when prior information is not available. The DCPC further includes control logic for selectively dynamically aligning the sensors by applying the conditional correspondence probability distribution for each feature point x′j and x i , Determine the distance d ij between the corresponding feature points x′j and x i and initiating a calibration for the sensors, the position discrepancies between corresponding feature points xi' and x iin the FOVs of each of the sensors. The DCPC further includes control logic for applying temporal smoothing to outputs of normalized entropy and conditional correspondence distribution calculations. The temporal smoothing accounts for unexpected vehicle movements. The DCPC further includes control logic for determining that the sensor is in a position that can be handled by the DCPC by performing a threshold check. The threshold check compares a position of FOVs of the two or more sensors to a library of calibration values. If it is determined that the sensor's position is shifted from the expected position by an amount less than the threshold, the DCPC is caused to dynamically align the sensor by applying the conditional correspondence probability distribution for each feature point. xj' and x i, Determining a Euclidean distance d ij between the corresponding feature points xi' and x i and generating a calibration for the sensors that detects position discrepancies between corresponding feature points xi' and x iin the FOVs of each of the sensors. The DCPC further includes control logic to, when it is determined that a position of a sensor is displaced from an expected position by an amount greater than or equal to a threshold, cause the DCPC to generate a notification by performing one or more of the following operations: setting a code in the vehicle's memory, sending a code to a service center via a wired or wireless connection, notifying a vehicle operator via a human-machine interface (HMI), and at least temporarily disabling ADAS features involving the sensor that is out of position by an amount greater than or equal to the threshold.The DCPC further includes control logic for continuously monitoring positions of each of the two or more sensors, selectively dynamically aligning the sensors, and ensuring that the two or more sensors are properly calibrated for vehicle perception tasks including advanced driver assistance system (ADAS) features.

[0024] Further areas of applicability of the present disclosure will become apparent from the detailed description. It should be understood that the description and specific examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Brief description of the drawings

[0025] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. Fig. 1A is a schematic representation of a vehicle equipped with a system for detecting a change in camera position (DCPC) on moving vehicle parts and equipped with wide-angle satellite cameras (YSAT) according to one aspect of the present disclosure; Fig. 1B is a schematic representation of the vehicle from Fig. 1A, which is equipped with the system for DCPC on moving vehicle parts and is equipped with perception satellite cameras (PSAT) according to another aspect of the present disclosure; Fig. 2 is a flowchart showing a logical flow of functions of the system for the benefit of the DCPC of the Fig. 1A and Fig. 1B according to one aspect of the present disclosure; Fig. 3 is a schematic representation of a portion of the DCPC performing visual feature extraction using deep learning according to an aspect of the present disclosure; Fig. 4 is a flowchart illustrating a method for executing DCPC control logic functions on a set of overlapping fields of view of cameras of the system of Fig. 1A and Fig. 1B according to one aspect of the present disclosure. Detailed description

[0026] The following description is merely exemplary in nature and is not intended to limit the disclosure, application, or uses.

[0027] Referring to Fig. 1A and Fig. 1B, a system 10 for detecting camera position changes on moving vehicle parts is illustrated in schematic form. The system 10 generally includes a vehicle 12. While the illustrated vehicle 12 is a pickup truck, the vehicle 12 may be any of a variety of types of vehicle 12 without departing from the scope or intent of the present disclosure. The vehicle 12 may be any of a variety of vehicles 12, including, but not limited to, cars, trucks, sport utility vehicles (SUVs), buses, semi-trailers, tractors used in agriculture or construction, or the like, a watercraft, an aircraft such as airplanes, helicopters, gyrocopters, or the like. At least two sensors 14 are disposed on the vehicle 12.The sensors 14 may be any of a variety of sensors 14 that collect data about the environment around the vehicle 12, including electromagnetic and / or optical information in a variety of different wavelengths, including those visible to humans, as well as infrared, ultraviolet, and other such regions of the light spectrum not visible to humans. The sensors 14 may also include cameras 16, laser light detection and ranging (LiDAR) sensors, radio detection and ranging (RADAR) sensors, sound navigation and ranging (SONAR) sensors, and so on.: Sound Navigation and Ranging), ultrasonic sensors, and any of a variety of other sensors 14 capable of determining position information of the vehicle 12 relative to the environment around the vehicle 12 without departing from the scope or intent of the present disclosure. In several aspects, the sensors 14, including the cameras 16, may be integrated directly onto or into the vehicle 12 or may be installed during customer service performed by the manufacturer, the dealer, the customer of the vehicle 12, or other third parties without departing from the scope or intent of the present disclosure.

[0028] One or more of the sensors 14 of the vehicle 12 are formed with, arranged on, or otherwise mounted to movable components of the vehicle 12. That is, Fig. 1A and Fig. 1B is equipped with movable exterior mirrors 18A, 18B and a movable tailgate 20. A sensor 14, particularly a camera 16, is mounted on or otherwise disposed on both the movable exterior mirrors 18A, 18B and the movable tailgate 20. It should be appreciated that the vehicle 12 may have sensors 14 disposed on other movable and / or fixed exterior components without departing from the scope or intent of the present disclosure. In several examples, movable exterior components of the vehicle 12 to which the sensors 14 may be mounted may include exterior mirrors 18A, 18B, which may be manually folding or power folding, a manually or power folding tailgate 20, a trunk lid (not specifically shown), a liftgate, a roll-up door, a movable aerodynamic device (i.e.,a spoiler, an active front fascia, or the like), one or more movable panels such as doors, or the like. Immobile mounting locations for the sensors 14 may include front and / or rear fenders, bumpers, and the like without departing from the scope or intent of the present disclosure.

[0029] Now specifically on the Fig. 1A and Fig. 1B, each of the cameras 16 has its own field of view (FOV) 22. The Fig. 1A are shown as having wide-angle perception capabilities. That is, the cameras 16 with which the vehicle 12 can be Fig. 1A are wide-angle satellite cameras (YSAT) that have a field of view 22 of approximately 180° or, in some examples, a field of view 22 of more than 180°. In contrast, the cameras 16 with which the vehicle 12 is Fig. 1B, perception satellite cameras (PSAT) with FOVs 22 ranging between about 20° and about 100°, although the exact dimensions of the FOVs 22 of the PSAT cameras 16 may vary substantially without departing from the scope or intent of the present disclosure. Furthermore, it should be appreciated that the Fig. 1A and Fig. 1B displayed vehicles 12 are actually the same vehicle 12 and that Fig. 1A and Fig. 1B, for the sake of clarity, only show different sets of cameras 16 with which the vehicle 12 is equipped.

[0030] In particular, a left mirror PSAT camera 16A' mounted on the left movable outside mirror 18A has a left rear FOV 22A'', and a similarly or identically located left mirror YSAT camera 16A' has a left FOV 22' that substantially overlaps the left rear FOV 22A''. Likewise, a right mirror PSAT camera 16B' mounted on the right movable outside mirror 18B has a right rear FOV 22B'', and a similarly or identically located right mirror YSAT camera 16B' has a left FOV 22B' that substantially overlaps the right rear FOV 22B''. Similarly, a camera 16C'' mounted on the tailgate 20 has a rearward FOV 22C''. The tailgate 20 may also have a rear YSAT camera 16C' mounted thereon, providing a rear YSAT FOV 22C'.The left rear FOV 22A'' at least partially overlaps the rearward FOV 22C'', and the right rear FOV 22B'' also at least partially overlaps the rearward FOV 22C''. The left mirror camera 16A or a left fender-mounted camera 16D may also have a left front FOV 22D, and the right mirror camera 16B or a right fender-mounted camera 16E may also have a right front FOV 22E. Each of the left front FOV 22D and the right front FOV 22E may at least partially overlap with a forward FOV 22F'' of a front-mounted camera 16F''. Likewise, the vehicle 12 may be equipped with a front-mounted YSAT camera 16F' and has a forward YSAT FOV 22F' that at least partially or completely overlaps with the forward FOV 22F'', the left front FOV 22D, and the right front FOV 22E.It should be noted that although the terms used here are . Fig. 1A Left mirror, right mirror, rear and front mounted cameras 16A, 16B, 16C, 16D are specifically described and with respect to Fig. 1B, additional cameras 16 with additional FOVs 22 may be disposed on the vehicle 12 and may provide additional visual information to the vehicle 12. In some examples, the additional cameras 16 may include a windshield or interior rearview mirror-mounted camera 16G, a truck bed camera or other such high-mounted backup camera 16H, or the like. Since the Fig. 1A have larger FOVs 22 than those in Fig. 1B, it should be noted that in many respects the vehicle 12 of Fig. 1A and Fig. 1B data are provided by the YSAT cameras 16, which substantially overlap with the data provided by the PSAT cameras 16.

[0031] The vehicle 12 is further equipped with one or more controllers 24. The controllers 24 are non-generalized electronic control devices including a pre-programmed digital computer or processor 26, a non-transitory computer-readable medium or memory 28 used to store data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and a transceiver or input / output (I / O) port 30. The computer-readable medium includes any type of medium accessible by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.A "non-transitory" computer-readable memory 28 excludes wired, wireless, optical, or other communication connections that carry transitory electrical or other signals. A non-transitory computer-readable memory 28 includes media on which data can be permanently stored and media on which data can be stored and later overwritten, such as a rewritable optical disk or an erasable storage device. Computer code includes any type of program code, including source code, object code, and executable code. The processor 26 is configured to execute the code or instructions. In vehicles 12, the controller 24 may be a dedicated Wi-Fi controller or an engine control module, a transmission control module, a body control module, an infotainment control module, etc.The I / O ports 30 are configured to communicate wirelessly using IEEE 802.11x Wi-Fi protocols, cellular protocols such as Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Wireless in Local Loop (WLL), General Packet Radio Services (GPRS), 1G, 2G, 3G, 4G Long Term Evolution (LTE), 5G, or the like.

[0032] The memory 28 may store one or more applications 32. An application 32 is a software program configured to perform a particular function or set of functions. The application 32 may include one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, associated data, or a portion thereof suitable for implementation in suitable computer-readable program code. The applications 32 may be stored within the memory 28 of the on-board controllers 24 in the vehicles 12 or in additional or separate memory, such as within a memory 28 of a cloud computing device such as the cloud computing server 14.Examples of the applications 32 include audio or video streaming services, games, browsers, social media, and an application for detecting changes in the position of the camera 16 on moving components of the vehicle 12. For the sake of simplicity and to improve clarity, the application for detecting changes in the position of the camera 16 on moving components of the vehicle 12 is referred to hereinafter as DCPC 34.

[0033] Now on Fig. 2 with reference to and with continued reference to Fig. 1A and Fig. 1B, a schematic representation of a method 100 for utilizing the functions of the DCPC 34 is shown in flowchart form. When movable components of the vehicle 12 are moved, the cameras 16 mounted thereon are also moved, resulting in changes to the FOV 22 of the camera 16. When such movement occurs, cameras 16 used for autonomous driving or advanced driver assistance system (ADAS) functions may report information to the ADAS that is of reduced use or that may not provide sufficiently accurate information to enable ADAS functionality based thereon. In addition, it should be appreciated that electrical problems such as blown fuses, short circuits, damaged wiring, orWiring, non-functional or partially functional switches, motor failures or failures of moving parts, or mechanical problems such as bent mounting brackets or the like can cause changes in the FOV 22 of the camera 16. Accordingly, the DCPC 34 utilizes the overlapping FOV 22 of at least two cameras 16 to detect whether a change in the position of the camera 16 has occurred for one or more of the at least two cameras 16.

[0034] The DCPC 34 utilizes common features of pixel locations within image data acquired by each of the at least two cameras 16 and within a common or overlapping FOV 22 of the at least two cameras 16. The DCPC 34 measures a mutual information metric between the cameras 16 to determine a change in status based on preset thresholds and baseline values. In several aspects, the DCPC 34 can be applied to detect a change in the position of cameras 16 or sensors 14 disposed on moving components of the vehicle 12. The change in status acts as an enabling condition for sensor 14 alignment applications, such as camera-to-vehicle (C2V) and camera-to-lidar (C2L) algorithms.The DCPC 34 operates in a lightweight manner that eliminates the need for computationally intensive procedures to continuously run to fully align the sensor 14 for extrinsic monitoring of the sensor 14.

[0035] The method 100 begins at block 102, where a request is received from the operator of the vehicle 12. The operator request may be triggered as a dynamic alignment request on a per-sensor 14 basis. At block 104, a diagnostic application 36 may be executed during manufacture of the vehicle 12 and / or during service of the vehicle 12. The diagnostic application 36 is triggered in the manufacturing context at block 106 via a per-sensor 14 manufacturing alignment request. Because the sensors 14 are mounted on the vehicle 12 during manufacturing, the positions of such sensors 14 must be calibrated so that the sensors 14 provide expected and appropriate information to on-board systems of the vehicle 12. Accordingly, the diagnostic application 36 is triggered in the manufacturing context at block 106 to calibrate the positions of the sensors 14 during manufacture of the vehicle 12.

[0036] Likewise, in the maintenance context, the diagnostic application 36 is triggered at block 108 via a maintenance trigger alignment request per sensor 14. In several aspects, the maintenance context includes situations where a vehicle 12 is in the shop for scheduled maintenance, repair, or the like, and the position of a sensor 14 is changed during maintenance. Accordingly, the diagnostic application 36 is triggered at block 108 to address such position changes. At block 110, the system 10 may also automatically detect a misalignment of the sensor 14 on a per-sensor 14 basis.

[0037] The memory 28 contains known calibrations, such as a manufacturing calibration at block 112 with a library of calibration values ​​and parameters for initial calibrations of the sensor 14 and the camera 16 based on computer-aided design (CAD) values ​​and intrinsic data from a driver of the sensor 14 and live and stored data related to ride height information of the vehicle 12 and the like. In block 114, a dynamic calibration is generated that includes dynamic activity of the vehicle 12, such as roll, pitch, yaw, longitudinal and lateral accelerations and the like, as obtained from the vehicle's sensors 14 via the driver of the sensor 14. The manufacturing calibration of block 112 and the dynamic calibration of block 114 are stored in the memory 28 at block 116 as a new calibration CTM file in the library of calibration values.

[0038] At block 118, the DCPC 34 receives raw sensor 14 data from the various sensors 14 and cameras 16 with which the vehicle 12 is equipped. The raw sensor 14 data is sent to the DCPC 34 at block 120 along with the operator request from block 102, the results of the diagnostic application 36 from either manufacturing or maintenance contexts 106, 108, the misalignment detection from block 110, and the new calibration CTM file.

[0039] Within block 120, the DCPC 34 performs an arbitration process 122 and an alignment process 124 based on the operator request 102, the results of the diagnostic application 36 from block 104, the detection of a misalignment from block 110, and the new calibration CTM file from block 116. The arbitration process 122 determines, among multiple alignment requests, when an alignment process 124 should be started. The alignment process 124 uses raw acquisition data to calculate extrinsic and intrinsic parameters of the sensors 14. The results of the arbitration and alignment processes 122, 124 are then forwarded to the diagnostics at block 126 as an alignment status and adjustment value of the sensor 14. At block 126, the diagnostics process updates the CTM calibration based directly on the library of calibration values ​​stored in memory 28.The diagnostic process 126 generates a status of the sensor 14 and forwards it to a force measurement system (FMS) at block 128. If the FMS 128 indicates that the sensor 14 or sensors 14 are sufficiently miscalibrated such that the DCPC 34 cannot effectively make adjustments to account for the pose or attitude changes of the sensors 14, then the system 10 escalates a status of the positions of the sensor 14 and sends a notification request to a human-machine interface (HMI) of the vehicle 12 at block 130. In some examples, the diagnostic process at block 126 may directly notify the operator of the vehicle 12 via the HMI at block 130 that the sensors 14 are too far out of calibration for the DCPC 34 to make any necessary adjustments.In such examples, the HMI indicates to the operator of the vehicle 12 to take the vehicle 12 to the shop for service and that the sensors 14 and / or cameras 16 cannot be used for certain functions of the vehicle 12, such as ADAS functions or the like.

[0040] Conversely, if the diagnostic process at block 126 determines that the updated CTM values ​​are within the range of adjustment enabled by the DCPC 34, the method 100 proceeds to block 132, where the system 10 updates a configuration of the vehicle 12 via read / write data stored in the non-volatile memory (NVM) 28 and a learned status of the sensors 14 of the vehicle 12. The updated configuration of the vehicle 12 and read-only CAD values ​​from the NVM 28, along with the manufacturing calibration 112 and the dynamic calibration 114, are passed to block 116, where a new, updated CTM value is generated and subsequently passed to the DCPC 34 for sensor 14 alignment.

[0041] It is particularly noteworthy that the alignment process of the sensor 14 during manufacturing does not utilize the DCPC 34 itself. Rather, dynamic alignment of the sensor 14 is permitted while the vehicle 12 is in use after manufacturing and / or after maintenance. That is, the DCPC 34 is a lightweight monitoring system that operates while the vehicle 12 is under the command or control of the vehicle 12 operator.

[0042] Referring now to Fig. 3 and with continued reference to Fig. 1A, Fig. 1B and Fig. 2, the system 10 performs visual feature extraction using deep learning. In several aspects, the exact deep learning methodology may include, but is not limited to: deep neural networks (DNNs), convolutional neural networks (CNNs), heterogeneous convolutional neural networks (HCNNs), long short-term memory networks (LSTMs), recurrent neural networks (RNNs), generative adversarial networks (GANs), radial basis function networks (RBFNs), multilayer perceptrons (MLPs), self-organizing maps (SOMs).: Self-Organizing Maps), Deep Belief Networks (DBNs), and / or the use of learning-based feature points such as self-supervised Interest Point Detection and Description (SUPERPOINT), Learned Invariant Feature Transform (LIFT), or hand-crafted feature point techniques such as Scale Invariant Feature Transform (SIFT), Oriented Fast and Rotated Brief (ORB), or the like, without departing from the scope or intent of the present disclosure. In . Fig. 3, the deep learning methodology 200 of the present disclosure acquires image data of an input 202 from the one or more sensors 14 having a width (W) and a height (H). The input 202 is passed through a multi-layer encoder 204, which, via convolutional and pooling layers 206, reduces the input 202 into a significantly smaller and computationally less complex output 208. The output 208 is passed to an interest point decoder 210 and a descriptor decoder 212.

[0043] The interest point decoder 210 further reduces a size of the output 208 of the multi-layer encoder 204 by a predetermined factor. In several aspects, the reduction factor may vary in size from application to application and according to the computational and optical properties and capabilities of the components of the system 10. In the Fig. However, in the example illustrated in Figure 3, the interest point decoder 210 reduces the output 208 from the multi-layer encoder 204 by a factor of eight (8) to generate an interest point "x." The interest point decoder 210 applies a softmax activation function 214 to the interest point data x, which converts a vector of numbers into a vector of probabilities, where the probabilities of each value are proportional to the relative scale of each value in the vector defining x. The interest point decoder 210 then applies a reshaping function 216 to the output from the softmax activation function 214, generating an interest point output 218.

[0044] The descriptor decoder 212 further reduces a size of the output 208 of the multi-layer encoder 204 by a predetermined factor. In several aspects, the reduction factor may vary in size from application to application and according to the computational and optical properties and capabilities of the components of the system 10. In the Fig. However, in the example illustrated in Figure 3, the descriptor decoder 212 reduces the output 208 from the multi-layer encoder 204 by a factor of eight (8) to produce an interest or descriptor "D." The descriptor D is then passed to a bicubic interpolator 220, which uses a two-dimensional system employing cubic splines or other polynomial techniques to sharpen and augment an intermediate feature map from the output 208. The output from the bicubic interpolator 220 is processed through an L2 norm 222, which normalizes an interest or descriptor "D" with a Euclidean distance 224 of the vector coordinate from the origin of the vector space for the given feature or interest point x.

[0045] If you now turn Fig. 4 and further refers to Fig. 1A, Fig. 1B, Fig. 2 and Fig.3, the DCPC 34 is specifically illustrated as a series of method steps in the form of a flowchart together with image data 300A, 300B retrieved from two cameras 16 with overlapping FOVs 22. The DCPC 34 begins at block 302. At block 304, the DCPC 34 receives image data from the sensors 14 of the vehicle 12. More specifically, the obtained image data 300A, 300B include images with overlapping FOVs 22. At block 306, the DCPC 34 calculates a conditional correspondence probability distribution for each overlapping feature point. xj',xi etc. within the image data 300A, 300B. The conditional correspondence probability distribution can be described as: p(xj'|xi)∝exp(−dij−dN(xi)λdN(xi)), ∑j=1n'p(xj'|xi)=1; where xj' and x i are feature points of two comparison images, dij=dist(des(xi),des(xj')) the predefined distance metric between descriptors of the points xj' and x i is, dN(xi)=minj(dij) the nearest neighbor of point x i in the feature space when the predefined distance metric is given, and λ is a tunable inverse scale parameter of an exponential probability distribution.

[0046] At block 308, the DCPC 34 calculates a normalized joint entropy from the conditional correspondence probability distribution for each of the overlapping feature points xj',xi etc. according to: H(C,C')=−1η∑i=1n∑j=1n'p(xi)p(xj'|xi) log(p(xi)p(xj'|xi)), where η = log(nn') is the maximum joint entropy and p(x i ) is a uniform distribution when no prior information is available.

[0047] At block 310, the DCPC 34 performs temporal smoothing. As one example, the calculated normalized joint entropy may be averaged within a predefined time window or a predefined number of input frames. Specifically, the DCPC 34 smooths input data to account for movement of the vehicle 12 that may be unexpected, such as disturbances caused by road disturbances, including bumps, potholes, and the like, as well as changes in direction of the vehicle 12 that may or may not be planned. The temporal smoothing at block 310 may occur continuously while the DCPC 34 is operating. The output from the temporal smoothing at block 310 is sent to a compilation of threshold and baseline feature information stored in memory at block 312.

[0048] At block 314, the DCPC 34 evaluates features within the image data 300A, 300B with regard to displacement of moving parts and determines whether the sensor 14 or the camera 16 is in a known and declared position. To perform the evaluation at block 10, the DCPC 34 receives the threshold information and baseline feature information from block 312 stored in the memory 28, as well as an output from the temporal smoothing at block 310. If at block 314 the sensor 14 or the camera 16 is correctly and accurately placed, the DCPC 34 returns to block 304 where new image data 300A, 300B with one or more overlapping FOVs 22 and one or more overlapping feature points xj',xi etc. are obtained from the sensors 14.

[0049] However, if at block 316 the DCPC 34 determines that the sensor 14 or the camera 16 is not correctly and accurately placed, the DCPC 34 proceeds to block 316. At block 316, the DCPC 34 performs a threshold check for misalignment detection. In several aspects, the threshold check may include a broad range of data and a large number of variables, some or all of which may depend on the particular hardware, physical location, and sensors 14 of the system 10. The threshold may be defined as a normalized joint entropy value calibrated from road tests that tolerates the normal fluctuation in entropy under normal driving conditions, yet can still effectively detect unexpected displacement of the sensor 14.That is, the threshold check is hardware dependent, but should be understood to define a threshold at or above which the sensor 14 has been shifted so significantly from an expected position that the DCPC 34 cannot overcome the discrepancy between the FOVs 22 to correctly identify an attitude or position of the one or more sensors 14 relative to each other and relative to the vehicle 12. Accordingly, the DCPC 34 proceeds from the threshold check at block 316 to block 318, where, if the DCPC 34 has determined that one or more sensors 14 are at or above the threshold, the DCPC 34 escalates a response, generates a notification, and / or reports the vehicle 12 for service at block 320.The DCPC 34 may escalate the notification in various ways, including, but not limited to, displaying a notification on an HMI of the vehicle 12 that can be seen, heard, or felt by an operator of the vehicle 12, sending a wireless communication to a service center or other such back office, setting a code in the memory of the vehicle 12, or the like. After setting a notification indicating that the sensors 14 are at or above the threshold, the DCPC 34 proceeds to block 322, where the DCPC 34 terminates.

[0050] Conversely, it should be appreciated that the threshold check at block 316 is a threshold in a range where the sensor 14 has been significantly shifted from an expected position, but the DCPC 34 can overcome or correct the position change so that the locations of the sensor 14 and the FOVs 22 relative to each other and relative to the vehicle 12 can be correctly identified, thereby enabling the vehicle 12 to continue utilizing ADAS features and the like with little or no performance degradation. Accordingly, if the DCPC 34 determines that the one or more sensors 14 are out of alignment but below an upper threshold limit, the DCPC 34 proceeds to block 324, where a dynamic alignment process is initiated.The process of dynamic alignment may include a wide variety of software-based or physical, manual, mechanical, electromechanical, pneumatic, hydraulic processes, or combinations thereof that either physically or virtually change or calibrate a position of the affected sensors 14.

[0051] From block 324, the DCPC 34 proceeds to block 326, where the DCPC 34 stores a new calibration CTM file, taking into account new alignments of the various affected sensors 14, in the library of calibration values. The new calibration CTM file is then passed back to block 302 and used as input for the next iteration of the DCPC 34's application. It should be appreciated that the DCPC 34 may run only upon occurrence of a request from the operator of the vehicle 12, a request from the service center, a request from the manufacturer, or the like, or the DCPC 34 may run iteratively, continuously, and / or recursively while the vehicle 12 is in operation without departing from the scope or intent of the present disclosure.

[0052] A system 10 and method 100, 200, 300 for detecting a change in position of the camera 16 and the sensor 14 on moving parts or components of the vehicle 12 of the present disclosure offers several advantages. These include the ability to be used to detect changes in the positions of cameras mounted on a variety of moving vehicle parts in a variety of different vehicle applications, and which can be retrofitted to existing vehicles or equipped on new vehicles. Furthermore, without significantly increasing the computational load and without increasing the complexity of the system 10 or the components, the system 10 and methods 100, 200, 300 can be run continuously, providing improved robustness, increasing redundancy, improving the functionality and resilience of the system 10, and improving customer satisfaction.

[0053] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to be within the scope of the present disclosure. Such variations should not be considered a departure from the spirit and scope of the present disclosure.

Claims

[1] A system for detecting a sensor position change on moving vehicle components, the system comprising: a vehicle with one or more movable vehicle components; two or more sensors disposed on at least one of the one or more movable vehicle components, each of the two or more sensors detecting optical information within a particular field of view (FOV) about an environment around the vehicle, the particular FOV of each of the two or more sensors at least partially overlapping with an FOV of at least one other sensor; one or more controllers, each of the one or more controllers comprising a processor, a memory, and one or more input / output (I / O) ports, the I / O ports communicating with the two or more sensors, the memory storing programmatic control logic, the processor executing the programmatic control logic, the programmatic control logic including a sensor position change detection application (DCPC), the DCPC comprising: a first control logic for detecting overlapping optical information from the two or more sensors; a second control logic for calculating a conditional correspondence probability distribution for feature points in the overlapping optical information; a third control logic for calculating a normalized joint entropy of the feature points; a fourth control logic for determining that the sensor is in a position that can be handled by the DCPC; and a fifth control logic for continuously monitoring positions of each of the two or more sensors, selectively dynamically aligning the sensors and ensuring that the two or more sensors are properly calibrated for vehicle perception tasks including advanced driver assistance system (ADAS) functions. [2] The system of claim 1, wherein the first control logic further comprises control logic to: to acquire optical information from at least one wide-angle satellite camera (YSAT) with a field of view of approximately 180°; and to capture optical information at least partially overlapping the YSAT field of view from at least one perception satellite camera (PSAT) having a field of view of approximately 20° to about 100°, and wherein the movable vehicle components include one or more of one or more exterior vehicle mirrors, one or more doors, a trunk, and a tailgate. [3] The system of claim 1, wherein the first control logic further comprises: control logic to perform visual feature extraction using one or more hand-crafted and learning-based methods including one or more of Self-sUPERvised Interest Point Detection and Description (SUPERPOINT), Learned Invariant Feature Transform (LIFT), Scale Invariant Feature Transform (SIFT), and Oriented Fast and Rotated Brief (ORB). [4] The system of claim 1, wherein the second control logic further comprises: Calculate a conditional correspondence distribution for overlapping feature points by: p(xj'|xi)∝exp(−dij−dN(xi)λdN(xi)), Σj=1n'p(xj'|xi)=1; where xj' and x i Feature points of images are each captured by separate sensors, dij=dist(des(xi),des(xj')) is a predefined distance metric relative to a Euclidean distance between descriptors of points xj' and x i increases monotonically, dN(xi)=minj(dij) and a nearest neighbor of point x i in the feature space when the predefined metric is given, and λ is a tunable inverse scale parameter of an exponential probability distribution. [5] The system of claim 4, wherein the third control logic further comprises: Calculate a normalized joint entropy from the conditional correspondence distribution for each of the overlapping feature points xj', xi etc. according to: H(C,C')=−1η∑i=1n∑j=1n'p(xi)p(xj'|xi)log(p(xi)p(xj'|xi)), where η = log(nn') is the maximum joint entropy and p(x i ) is a uniform distribution when prior information is not available. [6] The system of claim 1, further comprising: a control logic for applying temporal smoothing to outputs of the second and third control logic, wherein the temporal smoothing takes unexpected vehicle movements into account. [7] The system of claim 1, wherein the fourth control logic further comprises: control logic to perform a threshold check, the threshold check comparing a position of FOVs of the two or more sensors to a library of calibration values. [8] The system of claim 7, wherein the fourth control logic comprises control logic that: if it is determined that a position of a sensor is shifted from an expected position by an amount greater than or equal to a threshold, causes the DCPC to generate a notification to a vehicle operator; and, If it is determined that the position of the sensor is shifted from the expected position by an amount less than the threshold, causes the DCPC to dynamically align the sensor. [9] The system of claim 8, wherein, if it is determined that the position of a sensor is displaced from the expected position by an amount greater than or equal to the threshold, the DCPC generates outputs comprising: setting a code in the vehicle's memory, sending a code to a service center via a wired or wireless connection, and disabling ADAS functions involving the sensor that is out of position by an amount greater than or equal to the threshold. [10] The system of claim 5, wherein the fifth control logic further comprises: selectively dynamically aligning the sensors by applying the conditional correspondence probability distribution for each feature point xj' and x i , Determining a Euclidean distance d ij between the corresponding feature points xi' and x iand generating a calibration for the sensors that detects position discrepancies between corresponding feature points xi' and x i taken into account in the FOVs of each of the sensors.

Images