Method for detecting position change of camera on moving carrier component

Abstract

A system for detecting a change in sensor position (DCPC) on a movable vehicle assembly includes two or more sensors on at least one movable vehicle assembly. The sensors detect optical information within different fields of view (FOV) regarding the vehicle surroundings. The different FOV of each sensor at least partially overlaps the FOV of at least one other sensor. The system includes a controller that executes a DCPC application. The DCPC application obtains overlapping optical information from the sensors, calculates condition-corresponding probability distributions of feature points in the overlapping optical information, calculates normalized joint entropies of the feature points, determines that the sensors are in DCPC manageable positions, continuously monitors the position of each sensor, selectively and dynamically aligns the sensors, and determines that the sensors are in the DCPC manageable positions. And ensuring that the sensor is calibrated for vehicle perception tasks including advanced driver assistance system (ADAS) functionality.

Description

Technical Field

[0001] The present disclosure relates to cameras disposed on a vehicle, and more particularly to cameras disposed on a movable vehicle assembly. Background Art

[0002] In modern vehicles, it is very important to know the displacement of moving vehicle components (such as movable side mirrors or lift gates) in a timely manner, because sensors are often attached to movable vehicle components. In general, such sensors are increasingly used for advanced driver assistance system (ADAS) operations and autonomous driving processes. In situations such as when the sensor housing or sensor housing bracket is hit, the field of view of the attached sensor may change significantly. If this displacement is not accurately resolved in a timely manner, the sensor will provide inaccurate information to the sensing and perception systems of the vehicle. Similarly, some sensors are installed on automatically moving vehicle components, such as automatically folding side mirrors. Therefore, if the automatic movement process fails partially or in whole, similarly, the sensor may provide inaccurate information to the sensing and perception systems of the vehicle.

[0003] Although current systems and methods for detecting camera position achieve their intended purposes, a new and improved system and method are needed to detect position changes of cameras or sensors disposed on moving vehicle components, which can be used to detect position changes of cameras mounted on various movable vehicle components and can be retrofitted to existing vehicles or equipped to new vehicles in a variety of different vehicle applications. In addition, a system and method for detecting position changes of cameras or sensors disposed on moving vehicle components is needed, which can be operated continuously without significantly increasing the computational burden and without increasing the complexity of the system or components, thereby improving system functionality and resilience, improving robustness, increasing redundancy, and improving customer satisfaction. Summary of the invention

[0004] According to several aspects of the present disclosure, a system for detecting a change in the position of a sensor on a movable carrier assembly includes a carrier having one or more movable carrier assemblies. The system also includes two or more sensors disposed on at least one of the one or more movable carrier assemblies. Each of the two or more sensors detects optical information within a different field of view (FOV) about the surrounding environment of the carrier. The different FOV of each of the two or more sensors overlaps at least partially with the FOV of at least one other sensor. The system also includes one or more controllers. Each of the one or more controllers has a processor, a memory, and one or more input / output (I / O) ports. The I / O port communicates with the two or more sensors. The memory stores programmed control logic. The processor executes the programmed control logic, which includes an application program (DCPC) for detecting a change in the position of the sensor. The DCPC includes at least first, second, third, fourth, and fifth control logics. The first control logic obtains overlapping optical information from two or more sensors. The second control logic calculates a conditional corresponding probability distribution of feature points in the overlapping optical information. The third control logic calculates a normalized joint entropy of feature points in the overlapping optical information. The fourth control logic determines that the sensor is in a position manageable by the DCPC. A fifth control logic continuously monitors the position of each of the two or more sensors, selectively dynamically aligns the sensors, and ensures that the two or more sensors are properly calibrated for vehicle perception tasks including advanced driver assistance system (ADAS) functionality.

[0005] In another aspect of the present disclosure, the first control logic further comprises a control logic that acquires optical information from at least one wide angle satellite (YSAT) camera having a field of view of about 180°, and acquires optical information from at least one perception satellite (PSAT) camera having a field of view of about 20° to about 100° that at least partially overlaps the YSAT field of view. The movable vehicle assembly includes one or more of the following: one or more vehicle exterior mirrors, one or more doors, a trunk, and a tailgate.

[0006] In another aspect of the present disclosure, the first control logic also includes control logic for performing visual feature extraction using one or more of manually designed methods and learning-based methods, wherein the methods include one or more of the following: self-supervised interest point detection and description (SUPERPOINT), learned invariant feature transform (LIFT), scale-invariant feature transform (SIFT), and oriented FAST and rotated BRIEF (ORB).

[0007] In another aspect of the present disclosure, the second control logic further includes control logic for calculating the conditional correspondence distribution of overlapping feature points by the following formula:

[0008] Ⅰ. Subject to:

[0009] Ⅱ.

[0010] where x j ′ and x i They are feature points in images acquired from separate sensors:

[0011] III.d ij = dist(des(x i ),des(x j ′));

[0012] is relative to point x j ′ and x i The Euclidean distance between the descriptors of is a predetermined distance metric that increases monotonically,

[0013] IV.d N (x i )=min j (d ij );

[0014] is the midpoint x in the feature space under a given predetermined metric i , and λ is the adjustable inverse scaling parameter of the exponential probability distribution.

[0015] In another aspect of the present disclosure, the third control logic further includes a method for calculating the overlapped feature points x according to the following formula: j ′,x i The control logic of calculating the normalized joint entropy for each condition in the distribution is:

[0016] Ⅴ.

[0017] where η = log(nn′) is the maximum joint entropy, and when prior information is not available, p(x i ) is uniformly distributed.

[0018] In yet another aspect of the present disclosure, the DCPC further comprises control logic for applying temporal smoothing to the outputs of the second control logic and the third control logic. The temporal smoothing accounts for undesired vehicle motion.

[0019] In yet another aspect of the present disclosure, the fourth control logic further includes control logic for performing a threshold check. The threshold check compares the positions of the FOVs of the two or more sensors to a library of calibration values.

[0020] In another aspect of the present disclosure, the fourth control logic includes control logic for the following operations: when it is determined that the displacement of the sensor's position relative to the expected position is greater than or equal to a threshold, the DCPC generates a notification to the vehicle operator; and when it is determined that the displacement of the sensor's position relative to the expected position is less than the threshold, the DCPC dynamically adjusts the sensor.

[0021] In another aspect of the present disclosure, when it is determined that the displacement of the sensor's position relative to the expected position is greater than or equal to a threshold, the DCPC generates an output, including: setting a code in the vehicle's memory, sending the code to a service center via wired or wireless communication, and disabling ADAS functions involving sensors whose position deviation is greater than or equal to the threshold.

[0022] In another aspect of the present disclosure, the fifth control logic further includes applying each feature point x j j and x i The condition corresponds to the probability distribution and determines the corresponding feature point x j ′ and x i The Euclidean distance d between ij , and generates for the sensors, taking into account the corresponding feature points x in the FOV of each sensor j ′ and x i The sensor can be selectively dynamically aligned by calibrating the position difference between the two.

[0023] According to several additional aspects of the present disclosure, a method for detecting a change in the position of a sensor on a movable carrier assembly includes: detecting optical information using two or more sensors disposed on one or more movable carrier assemblies of the carrier. Each of the two or more sensors has a different field of view (FOV) of the vehicle's surroundings. The different FOV of each of the two or more sensors overlaps at least partially with the FOV of at least one other sensor. The method also includes executing programmed control logic by one or more controllers, the programmed control logic including an application (DCPC) for detecting changes in sensor positions. Each of the one or more controllers has a processor, a memory, and one or more input / output (I / O) ports. The I / O port communicates with the two or more sensors. The memory stores the programmed control logic, and the processor executes the programmed control logic, including executing the DCPC. The DCPC includes control logic for the following operations: acquiring overlapping optical information from two or more sensors; calculating conditional correspondence probability distributions of feature points in the overlapping optical information; calculating normalized joint entropy of feature points in the overlapping optical information; determining that the sensors are at positions manageable by the DCPC; and continuously monitoring the position of each of the two or more sensors, selectively dynamically aligning the sensors, and ensuring that the two or more sensors are properly calibrated for vehicle perception tasks including advanced driver assistance system (ADAS) functions.

[0024] In another aspect of the present disclosure, the method also includes acquiring optical information from at least one wide angle satellite (YSAT) camera having a field of view of approximately 180°, and acquiring optical information from at least one perception satellite (PSAT) camera having a field of view of approximately 20° to approximately 100° that at least partially overlaps the YSAT field of view.

[0025] In another aspect of the present disclosure, the method also includes performing visual feature extraction using one or more of manually designed methods and learning-based methods, the methods including one or more of the following: self-supervised interest point detection and description (SUPERPOINT), learned invariant feature transform (LIFT), scale-invariant feature transform (SIFT), and oriented FAST and rotated BRIEF (ORB).

[0026] In another aspect of the present disclosure, the method further includes calculating the conditional correspondence distribution of overlapping feature points by the following formula:

[0027] Ⅰ. Subject to:

[0028] Ⅱ.

[0029] where x j ′ and x iThey are feature points in images acquired from separate sensors:

[0030] III.d ij = dist(des(x i ),des(x j ′));

[0031] is relative to point x j ′ and x i The Euclidean distance between the descriptors of is a predetermined distance metric that increases monotonically,

[0032] IV.d N (x i )=min j (d ij );

[0033] is the midpoint x in the feature space under a given predetermined metric i , and λ is the adjustable inverse scaling parameter of the exponential probability distribution.

[0034] In another aspect of the present disclosure, the method further includes: j ′,x i The normalized joint entropy is calculated for each of the conditional corresponding distributions in Etc.:

[0035] Ⅴ.

[0036] where η = log(nn′) is the maximum joint entropy, and when prior information is not available, p(x i ) is uniformly distributed; and by applying each feature point x j ′ and x i The condition corresponds to the probability distribution and determines the corresponding feature point x j ′ and x i The distance between ij , and enable the corresponding feature point x in the FOV of each sensor j ′ and x i The sensor can be selectively dynamically aligned by calibrating the position difference between the two.

[0037] In yet another aspect of the present disclosure, the method further includes applying temporal smoothing to the outputs of the normalized entropy and conditional correspondence distribution calculations, wherein the temporal smoothing accounts for undesired vehicle motion.

[0038] In yet another aspect of the present disclosure, the method further includes performing a threshold check. The threshold check compares the positions of the FOVs of the two or more sensors to a library of calibration values.

[0039] In another aspect of the present disclosure, the method also includes: when it is determined that the displacement of the sensor's position relative to the expected position is greater than or equal to a threshold, causing the DCPC to generate a notification to the vehicle operator; and when it is determined that the displacement of the sensor's position relative to the expected position is less than a threshold, causing the DCPC to dynamically adjust the sensor.

[0040] In another aspect of the present disclosure, the method also includes causing the DCPC to generate an output when it is determined that the displacement of the sensor's position relative to the expected position is greater than or equal to a threshold, including: setting a code in a memory of the vehicle, sending the code to a service center via wired or wireless communication, and disabling ADAS functions involving sensors whose position deviation is greater than or equal to the threshold.

[0041] In several additional aspects of the present disclosure, a method for detecting a change in the position of a sensor on a movable vehicle assembly includes: detecting optical information using two or more sensors disposed on one or more movable vehicle assemblies of the vehicle. Each of the two or more sensors has a different field of view (FOV) of the vehicle's surroundings. The different FOV of each of the two or more sensors overlaps at least partially with the FOV of at least one other sensor. The method also includes executing programmed control logic by one or more controllers, the programmed control logic including an application (DCPC) for detecting a change in the position of the sensor. Each of the one or more controllers has a processor, a memory, and one or more input / output (I / O) ports. The I / O port communicates with the two or more sensors. The memory stores the programmed control logic. The processor executes the programmed control logic, including executing the DCPC. The DCPC includes control logic for acquiring overlapping optical information from two or more sensors, including: acquiring optical information from at least one wide-angle satellite (YSAT) camera having a field of view of approximately 180°; and acquiring optical information that at least partially overlaps with the YSAT field of view from at least one perception satellite (PSAT) camera having a field of view of approximately 20° to approximately 100°. The DCPC also includes control logic for performing visual feature extraction using one or more of a manually designed method and a learning-based method, including one or more of the following: self-supervised interest point detection and description (SUPERPOINT), learning invariant feature transform (LIFT), scale-invariant feature transform (SIFT), and oriented FAST and rotated BRIEF (ORB). The DCPC also includes control logic for calculating the conditional correspondence distribution of overlapping feature points in overlapping optical information by the following formula:

[0042] Ⅰ. Subject to:

[0043] Ⅱ.

[0044] where x j ′ and x i They are feature points in images acquired from separate sensors:

[0045] III.d ij = dist(des(x i ),des(x j ′));

[0046] is relative to point x j ′ and x i The Euclidean distance between the descriptors of is a predetermined distance metric that increases monotonically,

[0047] IV.d N (x i )=min j (d ij );

[0048] is the midpoint x in the feature space under a given predetermined metric i The nearest neighbor of , and λ is an adjustable inverse scale parameter of the exponential probability distribution. DCPC also includes a method for calculating the nearest neighbor of the overlapping feature points x according to the following formula j ′,x i The control logic of calculating the normalized joint entropy for each condition in the distribution is:

[0049] VI.

[0050] where η = log(nn′) is the maximum joint entropy, and when prior information is not available, p(x i ) is uniformly distributed. DCPC also includes a control logic for applying each feature point x j ′ and x i The condition corresponds to the probability distribution and determines the corresponding feature point x j ′ and x i The distance between ij , and enable the corresponding feature point x in the FOV of each sensor j ′ and x i The DCPC also includes control logic for applying temporal smoothing to the outputs of the normalized entropy and conditional correspondence distribution calculations. Temporal smoothing takes into account undesired vehicle motion. The DCPC also includes control logic for determining that the sensors are in a position that the DCPC can manage by performing a threshold check. The threshold check compares the FOV positions of two or more sensors to a library of calibration values. When it is determined that the position of the sensor is less than the threshold relative to the expected position, the DCPC is caused to adjust the position of the sensor by applying each feature point x to the calibration value library. j ′ and x iThe condition corresponds to the probability distribution and determines the corresponding feature point x j ′ and x i The Euclidean distance d between ij , and generates for the sensors, taking into account the corresponding feature points x in the FOV of each sensor j ′ and x i The DCPC also includes a control logic for dynamically aligning the sensor by calibrating the position difference between the two or more sensors. The DCPC also includes a control logic for, when it is determined that the displacement of the position of the sensor relative to the expected position is greater than or equal to a threshold, causing the DCPC to generate a notification by performing one or more of the following: setting a code in a memory of the vehicle, sending the code to a service center via wired or wireless communication, notifying a vehicle operator via a human-machine interface (HMI), and at least temporarily disabling ADAS functions involving sensors whose position offset is greater than or equal to the threshold. The DCPC also includes a control logic for continuously monitoring the position of each of the two or more sensors, selectively dynamically aligning the sensors, and ensuring that the two or more sensors are properly calibrated for vehicle perception tasks including advanced driver assistance system (ADAS) functions.

[0051] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0053] Figure 1A is a schematic diagram of a vehicle equipped with a system for detecting camera position change (DCPC) on a moving vehicle component and equipped with a wide-angle satellite (YSAT) camera according to one aspect of the present disclosure;

[0054] Figure 1B According to another aspect of the present invention Figure 1A Schematic diagram of a vehicle equipped with a system for detecting camera position changes DCPC on a moving vehicle component and equipped with a perception satellite (PSAT) camera;

[0055] Figure 2 According to one aspect of the present disclosure, a method for utilizing Figure 1A and Figure 1B Flowchart of the functional logic flow of the DCPC system.

[0056] Figure 3 is a schematic diagram of a portion of a DCPC that performs visual feature extraction using deep learning according to one aspect of the present disclosure;

[0057] Figure 4 According to one aspect of the present disclosure, Figure 1A and Figure 1B A flow chart of a method for executing DCPC control logic functions for a set of overlapping fields of view of cameras of a system. DETAILED DESCRIPTION

[0058] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

[0059] refer to Figure 1A and Figure 1B , schematically illustrates a system 10 for detecting changes in the position of a camera on a movable vehicle component. The system 10 generally includes a vehicle 12. Although the vehicle 12 is shown as a pickup truck, the vehicle 12 can be any of a variety of vehicle 12 types without departing from the scope or intent of the present disclosure. The vehicle 12 can be any of a variety of vehicles 12, including but not limited to: a car, a truck, a sport utility vehicle (SUV), a bus, a semi-trailer tractor, a tractor used for agriculture or construction, etc., a boat, an aircraft such as an airplane, a helicopter, a rotorcraft, etc. At least two sensors 14 are disposed on the vehicle 12. The sensor 14 can be any of a variety of sensors 14 that capture data about the environment surrounding the vehicle 12, including electromagnetic and / or optical information of various different wavelengths, including those visible to humans and invisible to humans such as infrared, ultraviolet and other such portions of the spectrum. Without departing from the scope or intent of the present disclosure, the sensors 14 may also include cameras 16, light detection and ranging (LiDAR) sensors, radio detection and ranging (RADAR) sensors, sonic navigation and ranging (SONAR) sensors, ultrasonic sensors, and any of a variety of other sensors 14 capable of determining position information of the vehicle 12 relative to the environment surrounding the vehicle 12. In several aspects, the sensors 14, including the camera 16, may be integrated directly onto or into the vehicle 12, or may be installed in after-sales service performed by the vehicle 12 manufacturer, dealer, customer, or by other third parties, without departing from the scope or intent of the present disclosure.

[0060] One or more sensors 14 of the carrier 12 are formed with, disposed on, or otherwise mounted to a movable component of the carrier 12. That is, Figure 1A and Figure 1BThe illustrated vehicle 12 is equipped with movable exterior mirrors 18A, 18B and a movable tailgate 20. The sensor 14, in particular the camera 16, is mounted to or otherwise disposed on each of the movable exterior mirrors 18A, 18B and the movable tailgate 20. It should be understood that the vehicle 12 may have sensors 14 disposed on other movable and / or fixed exterior components without departing from the scope or intent of the present disclosure. In several examples, the movable exterior components of the vehicle 12 to which the sensor 14 may be attached may include: manually foldable or power-foldable exterior mirrors 18A, 18B, a manually or power-foldable tailgate 20, a trunk lid (not specifically shown), a hatchback, a rolling door, a movable aerodynamic device (i.e., a spoiler, an active front air dam, etc.), one or more movable panels (e.g., doors), etc. Fixed mounting locations for the sensor 14 may include front and / or rear fenders, bumpers, etc. without departing from the scope or intent of the present disclosure.

[0061] Now specific reference Figure 1A and Figure 1B The cameras 16 are shown, each having its own field of view (FOV) 22 . Figure 1A The camera 16 shown is depicted as having wide-angle perception capabilities. Figure 1A The camera 16 of the vehicle 12 is a wide-angle satellite (YSAT) camera having a FOV 22 of approximately 180° (or in some examples, a FOV 22 greater than 180°). Figure 1B The camera 16 of the vehicle 12 is a perception satellite (PSAT) camera 16 having a FOV 22 between about 20° and about 100°, and the precise measurement of the FOV 22 of the PSAT camera 16 may vary significantly without departing from the scope or intent of the present disclosure. In addition, it should be understood that Figure 1A and Figure 1B The vehicle 12 shown in FIG. 1 is actually the same vehicle 12 and is illustrated for clarity. Figure 1A and Figure 1B Only different arrangements of cameras 16 equipped to the vehicle 12 are depicted.

[0062] Specifically, the left rearview mirror PSAT camera 16A" mounted to the left movable outside rearview mirror 18A has a left rear FOV 22A", wherein the similarly or identically positioned left rearview mirror YSAT camera 16A' has a left FOV 22' that substantially overlaps with the left rear FOV 22A". Similarly, the right rearview mirror PSAT camera 16B" mounted to the right movable outside rearview mirror 18B has a right rear FOV 22B", wherein the similarly or identically positioned right rearview mirror YSAT camera 16B' has a right FOV 22B' that substantially overlaps with the right rear FOV 22B". Similarly, the camera 16C" mounted on the tailgate 20 has a rear FOV 22C. The tailgate 20 may also have a rear YSAT camera 16C' mounted thereon and providing a rear YSAT FOV 22C'. The left rear FOV 22A" at least partially overlaps with the rear FOV 22C", and the right rear FOV 22B" also at least partially overlaps with the rear FOV 22C". The left rearview mirror camera 16A or the camera 16D mounted on the left fender may also have a left front FOV 22D, and the right rearview mirror camera 16B or the camera 16E mounted on the right fender may also have a right front FOV 22E. Each of the left front FOV 22D and the right front FOV 22E may at least partially overlap with the front FOV 22F" of the front camera 16F". Similarly, the front YSAT camera 16F' may be mounted to the vehicle 12 and have a front YSAT FOV 22F' that at least partially or completely overlaps with the front FOV 22F", the left front FOV 22D, and the right front FOV 22E. It should be understood that although reference is made herein to the front FOV 22F", the left front FOV 22D, and the right front FOV 22E. Figure 1A The left rearview mirror, the right rearview mirror, the rear and front cameras 16A, 16B, 16C, 16D are described in detail, and reference is made to Figure 1B The left and right fender mounted cameras 16E and 16F are described, but additional cameras 16 with additional FOVs 22 may be provided on the vehicle 12 and provide additional optical information to the vehicle 12. In some examples, the additional cameras 16 may include a camera 16G mounted on a windshield or interior rearview mirror, a truck bed camera or other such high-mounted rear-view camera 16H, etc. In many aspects, because Figure 1A The YSAT camera depicted in Figure 1B The larger FOV 22 of the PSAT camera depicted in FIG. 2 is understood to be provided by the YSAT camera 16. Figure 1A and Figure 1B The data from the vehicle 12 overlaps significantly with the data provided by the PSAT camera 16 .

[0063] The vehicle 12 is also equipped with one or more controllers 24. The controller 24 is a non-universal electronic control device having a pre-programmed digital computer or processor 26, a non-transitory computer-readable medium or memory 28 for storing data (such as control logic, software applications, instructions, computer code, data, lookup tables, etc.), and a transceiver or input / output (I / O) port 30. Computer-readable media include any type of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard drive, compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer-readable memory 28 does not include wired, wireless, optical or other communication links that transmit temporary electrical or other signals. Non-transitory computer-readable memory 28 includes media in which data can be permanently stored and media in which data can be stored and subsequently rewritten, such as rewritable optical disks or erasable storage devices. Computer code includes any type of program code, including source code, object code, and executable code. The processor 26 is configured to execute code or instructions. In the vehicle 12 , the controller 24 may be a dedicated Wi-Fi controller or an engine control module, a transmission control module, a body control module, an infotainment control module, etc. The I / O port 30 is configured to communicate wirelessly using a Wi-Fi protocol under IEEE802.11x, a cellular protocol such as Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Wireless Local Loop (WLL), General Packet Radio Service (GPRS), 1G, 2G, 3G, 4G Long Term Evolution (LTE), 5G, etc.

[0064] The memory 28 may store one or more applications 32. An application 32 is a software program configured to perform a specific function or set of functions. The application 32 may include one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data, or a portion of the above suitable for implementation in a suitable computer-readable program code. The application 32 may be stored in the memory 28 of the onboard controller 24 in the vehicle 12, or in an additional or separate memory, such as the memory 28 of a cloud computing device such as a cloud computing server 14. Examples of the application 32 include audio or video streaming services, games, browsers, social media, and applications for detecting changes in the position of a camera 16 on a movable vehicle 12 component. In the following, for brevity and improved clarity, the application for detecting changes in the position of a camera 16 on a movable vehicle 12 component is referred to as a DCPC 34.

[0065] Reference now Figure 2 And continue to refer to Figure 1A and 1B, a schematic diagram of a method 100 utilizing the functionality of the DCPC 34 is shown in the form of a flow chart. When the movable vehicle 12 assembly moves, the camera 16 mounted thereon also moves, causing the FOV 22 of the camera 16 to change. When such movement occurs, the camera 16 used in the autonomous driving or advanced driver assistance system (ADAS) function may report information of reduced usefulness to the ADAS, or may not provide sufficiently accurate information to implement the ADAS function that relies on it. In addition, it should be understood that electrical problems (such as blown fuses, short circuits, damaged lines, inoperable or partially operable switches, damaged or faulty motors of movable parts) or mechanical problems (such as bent mounting brackets) may cause the FOV 22 of the camera 16 to change. Therefore, the DCPC 34 uses the overlapping FOV 22 of at least two cameras 16 to detect whether the position of one or more of the at least two cameras 16 has changed.

[0066] The DCPC 34 utilizes common features of pixel locations within image data acquired from each of at least two cameras 16 and within a shared or overlapping FOV 22 of at least two cameras 16. The DCPC 34 measures a mutual information metric between the cameras 16 to determine a change in state based on preset thresholds and baseline values. In several aspects, the DCPC 34 can be applied to detect a change in position of a camera 16 or sensor 14 disposed on a movable vehicle 12 assembly. The change in state is used as an enabling condition for sensor 14 alignment applications such as camera-to-vehicle (C2V) and camera-to-lidar (C2L) algorithms. The DCPC 34 operates in a lightweight manner, which eliminates the need for a computationally heavy full sensor 14 alignment process to continuously run the sensor 14 for external monitoring.

[0067] The method 100 begins at box 102 where a vehicle 12 operator request is received. The operator request may be triggered as a dynamic alignment request for each sensor 14. At box 104, the diagnostic application 36 may be executed during vehicle 12 manufacturing and / or during vehicle 12 service. In the manufacturing context of box 106, the diagnostic application 36 is triggered via a manufacturing alignment request for each sensor 14. When sensors 14 are installed to a vehicle 12 during manufacturing, the position of such sensors 14 must be calibrated so that the sensors 14 provide desired and appropriate information to the onboard vehicle 12 system. Therefore, in box 106, the diagnostic application 36 is triggered in the manufacturing context to calibrate the position of the sensors 14 during vehicle 12 manufacturing.

[0068] Likewise, in a maintenance context at block 108, the diagnostic application 36 is triggered via a maintenance-triggered alignment request for each sensor 14. In several aspects, the maintenance context includes situations where the vehicle 12 is undergoing scheduled maintenance, repair, etc., and the position of the sensor 14 changes during the maintenance. Thus, the diagnostic application 36 is triggered at block 108 to account for such position changes. At block 110, the system 10 may also automatically detect, for each sensor 14, a misalignment of the sensor 14.

[0069] The memory 28 contains known calibrations, such as the manufacturing calibration of block 112, which has a library of calibration values ​​and parameters for initial sensor 14 and camera 16 calibration based on computer aided design (CAD) values, as well as intrinsic data from the sensor 14 driver, and real-time data and stored data regarding the vehicle 12's ground clearance information, etc. A dynamic calibration is generated in block 114, including the dynamic activity of the vehicle 12, such as roll, pitch, yaw, longitudinal and lateral accelerations, etc., as obtained by the vehicle's sensors 14 via the sensor 14 driver. At block 116, the manufacturing calibration from block 112 and the dynamic calibration from block 114 are saved in the memory 28 as new calibration CTM files in the calibration value library.

[0070] At box 118, the DCPC 34 obtains raw sensor 14 data from the various sensors 14 and cameras 16 equipped to the vehicle 12. At box 120, the raw sensor 14 data is sent to the DCPC 34 along with the operator request from box 102, the diagnostic application 36 results from the manufacturing or maintenance context 106, 108, and the misalignment detection and new calibration CTM files from box 110.

[0071] In box 120, the DCPC 34 performs an arbitration process 122 and an alignment process 124 based on the operator request from box 102, the diagnostic application results 36 from box 104, the misalignment detection from box 110, and the new calibration CTM file from box 116. The arbitration process 122 determines when the alignment process 124 should be initiated among multiple alignment requests. The alignment process 124 uses the raw sensed data to calculate the extrinsic and intrinsic parameters of the sensor 14. The results of the arbitration process 122 and the alignment process 124 are forwarded to the diagnostics of box 126 as the sensor 14 alignment status and adjustment values. At box 126, the diagnostic process updates the CTM calibration directly based on the calibration value library stored in the memory 28. At box 128, the diagnostic process 126 generates the sensor 14 status and forwards it to the force measurement system (FMS). When the FMS 128 indicates that the sensor(s) 14 are sufficiently out of calibration that the DCPC 34 cannot effectively make adjustments to account for the change in the attitude of the sensor 14, then at box 130, the system 10 updates the status of the sensor 14 position and sends a notification request to the human machine interface (HMI) of the vehicle 12. In some examples, the diagnostic process of box 126 can directly notify the vehicle 12 operator via the HMI at box 130 that the sensor 14 is too out of calibration that the DCPC 34 cannot make the necessary adjustments. In such an example, the HMI will notify the vehicle 12 operator to send the vehicle 12 for repair, and the sensor 14 and / or camera 16 may not be used for certain vehicle 12 functions, such as ADAS functions, etc.

[0072] Conversely, when the diagnostic process of block 126 determines that the updated CTM value is within the range of possible adjustments that the DCPC 34 may make, the method 100 proceeds to block 132. At block 132, the system 10 updates the vehicle 12 configuration via read / write data stored in the non-volatile memory (NVM) 28 and the learned states of the sensors 14 of the vehicle 12. The updated vehicle 12 configuration and the read-only CAD values ​​from the NVM 28 are forwarded to block 116 along with the manufacturing calibration 112 and the dynamic calibration 114, where a new updated CTM value is generated and subsequently forwarded to the DCPC 34 for sensor 14 calibration.

[0073] It should be noted that the manufacturing sensor 14 calibration process does not utilize the DCPC 34 itself. Instead, it allows dynamic sensor 14 calibration to be performed when the vehicle 12 is used after manufacturing and / or after maintenance. That is, the DCPC 34 is a lightweight monitoring system that operates when the vehicle 12 is under the control of the vehicle 12 operator.

[0074] Now turn to Figure 3 And continue to refer to Figure 1A , Figure 1B and Figure 2, the system 10 performs visual feature extraction using deep learning. In several aspects, without departing from the scope or intent of the present disclosure, the exact deep learning method may include, but is not limited to: a deep neural network (DNN), a convolutional neural network (CNN), a heterogeneous convolutional neural network (HCNN), a long short-term memory (LSTM) network, a recurrent neural network (RNN), a generative adversarial network (GAN), a radial basis function network (RBFN), a multi-layer perceptron (MLP), a self-organizing map (SOM), a deep belief network (DBN), and / or using learning-based feature points, such as self-supervised interest point detection and description (Self-sUPERvised interest POINT detection and description, SUPERPOINT), learned invariant feature transform (Learned Invariant Feature Transform, LIFT), or using manually designed feature point techniques, such as scale-invariant feature transform (SIFT), oriented FAST and rotated BRIEF (Oriented Fast and Rotated Brief, ORB), etc. In Figure 3 In the example shown, the deep learning method 200 of the present disclosure obtains input 202 image data having width (W) and height (H) from one or more sensors 14. The input 202 passes through a multi-layer encoder 204, which reduces the input 202 to a significantly smaller and less computationally complex output 208 via convolutional layers and pooling layers 206. The output 208 is forwarded to an interest point decoder 210 and a descriptor decoder 212.

[0075] The point of interest decoder 210 further reduces the size of the output 208 of the multi-layer encoder 204 by a predetermined factor. In several aspects, the size of the reduction factor may vary from application to application and may vary depending on the computational and optical characteristics and capabilities of the components of the system 10. However, in Figure 3 In the example shown, the interest point decoder 210 reduces the output 208 from the multi-layer encoder 204 by a factor of eight (8) to generate an interest point "x". The interest point decoder 210 applies a Softmax activation function 214 to the interest point x data, which converts a vector of numbers into a vector of probabilities where the probability of each value is proportional to the relative scale of each value in the vector defining x. The interest point decoder 210 then applies a reshape function 216 to the output from the Softmax activation function 214 and generates an interest point 218 output.

[0076] The descriptor decoder 212 further reduces the size of the output 208 of the multi-layer encoder 204 by a predetermined factor. In several aspects, the size of the reduction factor may vary from application to application and may vary depending on the computational and optical characteristics and capabilities of the components of the system 10. Figure 3 In the example shown, however, the descriptor decoder 212 reduces the output 208 from the multi-layer encoder 204 by a factor of eight (8) to generate an interest or descriptor "D". The descriptor "D" is then passed to a bicubic interpolator 220, which utilizes a two-dimensional system using cubic, spline or other polynomial techniques to sharpen and amplify the intermediate feature maps from the output 208. The output from the bicubic interpolator 220 is processed via an L2 norm 222, which normalizes the interest or descriptor "D" using the Euclidean distance 224 of the vector coordinates from the origin of the vector space for a given feature or interest point "x".

[0077] Now turn to Figure 4 And continue to refer to Figure 1A , Figure 1B , Figure 2 and Figure 3 , the DCPC 34 is specifically illustrated as a series of method steps in the form of a flowchart, and illustrates image data 300A, 300B retrieved from two cameras 16 having overlapping FOVs 22. The DCPC 34 begins at block 302. At block 304, the DCPC 34 obtains image data from the sensor 14 of the vehicle 12. More specifically, the obtained image data 300A, 300B includes images having overlapping FOVs 22. At block 306, the DCPC 34 calculates each overlapping feature point x within the image data 300A, 300B. j ′,x i The conditional corresponding probability distribution can be described as:

[0078] Ⅰ Subject to:

[0079] Ⅱ.

[0080] where x j ′ and x i are the feature points of the two contrast images,

[0081] III.d ij = dist(des(x i ),des(x j ′));

[0082] It is point x j ′ and x i A predetermined distance metric between the descriptors of

[0083] IV.d N (x i )=min j (d ij );

[0084] is the point x in the feature space under a given predetermined distance metric i , and λ is the adjustable inverse scaling parameter of the exponential probability distribution.

[0085] At block 308, DCPC 34 calculates based on each overlapping feature point x j ′,x i The normalized joint entropy is calculated based on the probability distribution of the corresponding conditions:

[0086] Ⅴ.

[0087] where η = log(nn′) is the maximum joint entropy, and if no prior information is available, then p(x i ) is uniformly distributed.

[0088] At box 310, the DCPC 34 performs temporal smoothing. As an example, the calculated normalized joint entropy may be averaged within a predetermined time window or a predetermined number of input frames. Specifically, the DCPC 34 smoothes the input data to account for vehicle 12 motion that may be undesirable (e.g., disturbances caused by road disturbances (including bumps, potholes, etc.) and changes in vehicle 12 direction that may be planned or may not be planned. When the DCPC 34 is running, the temporal smoothing of box 310 may run continuously. At box 312, the output from the temporal smoothing of box 310 is sent to a compilation of threshold and baseline feature information stored in memory.

[0089] At block 314, the DCPC 34 evaluates features within the image data 300A, 300B for moving part displacement and determines whether the sensor 14 or camera 16 is in a known and calculated position. To perform the evaluation of block 10, the DCPC 34 obtains the threshold information and baseline feature information from block 312 and stored in the memory 28, as well as the temporally smoothed output from block 310. At block 314, when the sensor 14 or camera 16 is correctly and accurately positioned, the DCPC 34 returns to block 304, where a plurality of overlapping FOVs 22 and one or more overlapping feature points x are obtained from the sensor 14. j ′,x i New image data 300A, 300B etc.

[0090] However, when, at box 316, the DCPC 34 determines that the sensor 14 or camera 16 is not correctly and accurately positioned, the DCPC 34 proceeds to box 316. At box 316, the DCPC 34 performs a threshold check for misalignment detection. In several aspects, the threshold check can include a wide range of data and a large number of variables, some or all of which can depend on the specific hardware, physical location, and FOV of the sensor 14 of the system 10. The threshold can be defined as a value of normalized joint entropy calibrated according to road testing, which allows for normal fluctuations in entropy under normal driving conditions, but is still able to effectively capture undesired displacements of the sensor 14. That is, the threshold check is hardware dependent, but it should be understood that the threshold should be defined as a threshold at or above which the sensor 14 has been significantly displaced from the expected position, such that the DCPC 34 cannot overcome the differences between the FOV 22 to correctly identify the location or position of one or more sensors 14 relative to each other and relative to the vehicle 12. Thus, the DCPC 34 proceeds from the threshold check at block 316 to block 318 where, when the DCPC 34 has determined that one or more sensors 14 are at or above a threshold, the DCPC 34 escalates the response, generates a notification, and / or reports the vehicle 12 for maintenance at block 320. The DCPC 34 may escalate the notification by a number of different means including, but not limited to: displaying a notification on the HMI of the vehicle 12 that may be seen, heard, or felt by an operator of the vehicle 12, sending a wireless communication to a service center or other such backend, setting a code in a vehicle 12 memory, etc. After setting a notification indicating that the sensor 14 is at or above a threshold, the DCPC 34 proceeds to block 322 where the DCPC 34 ends.

[0091] In contrast, the threshold check of block 316 should be understood as a threshold range within which the sensor 14 has been significantly displaced from the expected position but within which the DCPC 34 is able to overcome or correct for the position change so that the position of the sensor 14 and the FOV 22 can be correctly identified relative to each other and relative to the vehicle 12, thereby allowing the vehicle 12 to continue to utilize ADAS features, etc., with little or no impairment to performance. Therefore, when the DCPC 34 determines that one or more sensors 14 are misaligned but below an upper threshold, the DCPC 34 proceeds to block 324 where a dynamic alignment process is initiated. The dynamic alignment process may include various software-based or physical, manual, mechanical, electromechanical, pneumatic, hydraulic processes, or combinations thereof, that physically or virtually change or calibrate the position of the affected sensor 14.

[0092] From block 324, the DCPC 34 proceeds to block 326. At block 326, the DCPC 34 stores a new calibration CTM file that takes into account the new alignment of the various affected sensors 14 into the calibration value library. The new calibration CTM file is then forwarded back to block 302 and used as input for the next iteration of the DCPC 34 application. It should be understood that the DCPC 34 may be run only upon a vehicle 12 operator request, a service center request, a manufacturer request, etc., or the DCPC 34 may be run iteratively, continuously, and / or recursively while the vehicle 12 is running without departing from the scope or intent of the present disclosure.

[0093] The disclosed system 10 and method 100, 200, 300 for detecting position changes of cameras 16 and sensors 14 on parts or components of a mobile vehicle 12 provide several advantages. These advantages include the ability to detect position changes of cameras mounted on various movable vehicle parts and retrofitted to existing vehicles or equipped on new vehicles in a variety of different vehicle applications. In addition, the system 10 and method 100, 200, 300 can run continuously without significantly increasing the computational burden and without increasing the complexity of the system 10 or components, which provides improved robustness, increased redundancy, improved functionality and resilience of the system 10, and increased customer satisfaction.

[0094] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A system for detecting a change in position of a sensor on a movable carrier assembly, the system comprising: a vehicle having one or more movable vehicle components; two or more sensors disposed on at least one of the one or more movable vehicle assemblies, each of the two or more sensors detecting optical information within a different field of view (FOV) about the vehicle's surroundings, wherein the different FOV of each of the two or more sensors at least partially overlaps with a FOV of at least one other sensor; one or more controllers, each of the one or more controllers having a processor, a memory, and one or more input / output (I / O) ports, the I / O ports communicating with the two or more sensors; the memory storing programmed control logic; The processor executes the programmed control logic; The programmed control logic includes an application program for detecting a sensor position change DCPC; The DCPC comprises: first control logic for acquiring overlapping optical information from the two or more sensors; A second control logic is used to calculate the conditional correspondence probability distribution of the feature points in the overlapping optical information; A third control logic is used to calculate the normalized joint entropy of the feature points; a fourth control logic for determining that the sensor is in a position manageable by the DCPC; and A fifth control logic is used to continuously monitor the position of each of the two or more sensors, selectively dynamically align the sensors, and ensure that the two or more sensors are properly calibrated for vehicle perception tasks including advanced driver assistance system (ADAS) functions.

2. The system according to claim 1, wherein: The first control logic also includes control logic for the following operations: Acquire optical information from at least one wide-angle satellite YSAT camera having a field of view of approximately 180°; and Optical information is obtained from at least one perception satellite PSAT camera having a field of view of about 20° to about 100° that at least partially overlaps the YSAT field of view, and wherein the movable vehicle components include one or more of the following: one or more vehicle exterior rearview mirrors, one or more doors, a trunk, and a tailgate.

3. The system according to claim 1, wherein: The first control logic further includes: Control logic for performing visual feature extraction using one or more of manually designed methods and learning-based methods, including one or more of the following: self-supervised interest point detection and description SUPERPOINT, learning invariant feature transform LIFT, scale-invariant feature transform SIFT, and orientation FAST and rotation BRIEF ORB.

4. The system according to claim 1, wherein: The second control logic further includes: The conditional correspondence distribution of overlapping feature points is calculated by the following formula: I. Subject to: II. where x′ j and x i They are feature points in images acquired from separate sensors: III.d ij =dist(des(x i ),dex(x′ j )); is relative to the point x′ j and x i The Euclidean distance between the descriptors of is a predetermined distance metric that increases monotonically, IV.d N (x) i )=min j (the ij ); is the midpoint x in the feature space given the predetermined metric i , and λ is the adjustable inverse scaling parameter of the exponential probability distribution.

5. The system according to claim 4, wherein: The third control logic further includes: According to the following formula based on the overlapping feature point x' j ,x i The normalized joint entropy of each of the corresponding distributions in the condition is calculated as: V. where η = log(nn′) is the maximum joint entropy, and when prior information is not available, p(x i ) is uniformly distributed.

6. The system according to claim 1, further comprising: Control logic for applying temporal smoothing to outputs of the second control logic and the third control logic, wherein the temporal smoothing accounts for undesired vehicle motion.

7. The system according to claim 1, wherein: The fourth control logic further includes: Control logic for performing a threshold check that compares the positions of the FOVs of the two or more sensors to a library of calibration values.

8. The system according to claim 7, wherein: The fourth control logic includes control logic for the following operations: causing the DCPC to generate a notification to a vehicle operator when it is determined that the position of the sensor has shifted by an amount greater than or equal to a threshold relative to the expected position; as well as When it is determined that the displacement of the position of the sensor relative to the expected position is less than the threshold, the DCPC is enabled to dynamically align the sensor.

9. The system according to claim 8, wherein: When it is determined that the displacement of the sensor's position relative to the expected position is greater than or equal to the threshold, the DCPC generates an output, including: setting a code in the vehicle's memory, sending the code to a service center via wired or wireless communication, and disabling ADAS functions involving sensors whose position offset is greater than or equal to the threshold.

10. The system according to claim 5, wherein: The fifth control logic further includes: By applying each feature point x′ j and x i The condition corresponds to the probability distribution, and the corresponding feature point x′ is determined i and x i The Euclidean distance d between ij , and generate corresponding feature points x′ for the sensors, taking into account the FOV of each sensor i and x i The sensor can be selectively dynamically aligned by calibrating the position difference between the two.

Images