Method and system for on-line external calibration of camera

Abstract

Systems and methods for external calibration of an in-cabin camera of a vehicle are disclosed. In one example, a method for performing external calibration of a vehicle camera includes: receiving a plurality of images from a camera of a vehicle; detecting and describing a plurality of two-dimensional (2D) feature points of the image using the trained neural network; mapping the detected 2D feature points to three-dimensional (3D) feature points stored in a feature point library generated by using the trained neural network; estimating a set of external parameters of the camera using the mapped 3D feature points; and performing an external calibration based on the set of external parameters.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to systems and methods for performing external calibration of a camera, such as an in-cabin camera of a vehicle. BACKGROUND

[0002] Advanced Driver Assistance Systems (ADAS) of a vehicle can rely on Driver Monitoring Systems (DMS) and / or Occupant Monitoring Systems (OMS) to adjust the operation of the vehicle. Cameras are widely used in various DMS and OMS systems. The cameras can extract information about the in-cabin environment, including the state, attributes, and posture of the driver and the occupants, as well as external features of the vehicle environment. The state and attributes of the driver and the occupants can generally be extracted according to the intrinsic parameters (e.g., local coordinates) of the camera, while the posture data of the driver and the occupants can be estimated according to the camera position and the extrinsic parameters of the camera. Since the information about the posture data of the driver and the occupants is obtained depending on the extrinsic parameters of the camera, it is necessary to perform flexible, accurate online extrinsic calibration of the camera.

[0003] However, existing online extrinsic calibration methods of cameras are hindered by complex setups and high costs. Moreover, existing methods can rely on supervised feature point detector and descriptor training algorithms, which rely on a large amount of manual annotation. In addition, the supervised feature point detector training algorithms can have poor detection rates, and the resulting feature point descriptors can have weak feature representations. SUMMARY

[0004] The present disclosure addresses at least one of the above problems by a method of performing extrinsic calibration of a camera, the method comprising receiving a plurality of images from a camera of a vehicle; detecting and describing a plurality of two-dimensional (2D) feature points of the images using a trained neural network; mapping the detected 2D feature points to three-dimensional (3D) feature points stored in a feature point library generated using the trained neural network, estimating a set of extrinsic parameters of the camera using the mapped 3D feature points, and performing extrinsic calibration based on the set of extrinsic parameters. The feature point library can be developed offline, e.g., in a laboratory of a vehicle manufacturer. The extrinsic calibration of the camera can be performed online on the vehicle during operation of the vehicle.

[0005] Other systems, methods, features, and advantages of the present disclosure will be or become apparent to one with skill in the art upon examination of the following drawings and detailed description. It is intended that all such additional systems, methods, features, and advantages be included within this description, be within the scope of the present disclosure, and be protected by the accompanying claims.

[0006] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art in the field of the embodiments described herein. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments described herein, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and not necessarily a limitation of the application.

[0007] Implementation of the method and / or system of some embodiments described herein can involve performing or completing selected tasks manually, automatically, or a combination thereof. Moreover, according to actual instrument and equipment and methodologies and materials described herein, several selected tasks could be implemented by hardware, by software, or by firmware or hardware and software in combination. BRIEF DESCRIPTION OF DRAWINGS

[0008] Some embodiments are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in

[0009] In the drawings:

[0010] Figure 1 A vehicle system including one or more cameras is shown in accordance with one or more embodiments of the disclosure;

[0011] Figure 2 A block diagram of a vehicle computing system configured to calibrate vehicle cameras using a trained neural network model is shown in accordance with one or more embodiments of the disclosure;

[0012] Figure 3 A schematic diagram of a camera calibration procedure is shown in accordance with one or more embodiments of the disclosure;

[0013] Figure 4 A block diagram of an exemplary embodiment of a calibration system configured to calibrate a camera is shown;

[0014] Figure 5 A block diagram of an exemplary embodiment of a model training system for training a neural network model is shown;

[0015] Figure 6 is an architecture diagram of a neural network model that can be trained in a training system in accordance with exemplary embodiments; Figure 5

[0016] Figure 7 ​A method for performing online external calibration of a camera, according to an exemplary embodiment, is shown;

[0017] Figure 8 A method for automatically generating labels for a pair of feature points and / or background points in an image, according to an exemplary embodiment, is shown;

[0018] Figure 9 The use according to an exemplary implementation is shown. Figure 5 The method of using an external calibration network training system to train neural network models.

[0019] Figure 10 A method for constructing a feature point library using a trained neural network model according to an exemplary implementation is shown;

[0020] Figure 11 A method for determining the external parameters of a camera according to an exemplary embodiment is shown;

[0021] Figure 12 An ADAS method for adjusting vehicle operation based on images received from an in-cabin camera of the vehicle, according to one or more embodiments of the present disclosure, is shown.

[0022] Figure 13 Example images of feature points detected in a reference object placed in the cab of a vehicle according to one or more embodiments of the present disclosure are shown; and

[0023] Figure 14 An example image of a chessboard used by a chessboard detection algorithm according to one or more embodiments of this disclosure is shown. Detailed Implementation

[0024] Advanced Driver Assistance Systems (ADAS) have been developed to adapt to and enhance vehicle operation, thereby improving driving quality and the driving experience. ADAS uses various types of sensors, such as cameras, lidar, radar, and ultrasonic sensors, to monitor the vehicle's internal and external environment. Sensor data can be processed by the onboard computing system to extract information about the vehicle operator / occupant and to implement actions based on that information. These actions may include warning the operator via audio or visual notifications, or controlling the vehicle's accelerator, brakes, or steering.

[0025] An ADAS can rely on a driver monitoring system (DMS) and / or an occupant monitoring system (OMS). The DMS includes a set of advanced safety features that apply outputs from in-cabin sensors (including cameras) to detect and track physical (e.g., the vehicle operator experiences drowsiness) or mental (e.g., the vehicle operator is distracted) states of the operator / driver. Moreover, the DMS can alert the vehicle operator when the vehicle operator is experiencing a physical or mental state that is not suitable for operating the vehicle. In contrast to the DMS, the OMS can monitor the entire in-cabin environment. The OMS includes features such as seatbelt detection, intelligent airbag deployment, left side object warning, and forgotten child warning. Thus, improving the efficiency of the DMS and / or OMS to detect visual information about the vehicle operator and / or occupant can improve the efficiency of the ADAS to implement remedial measures, thereby improving the quality of vehicle operation.

[0026] Cameras have been widely applied to various DMS and OMS due to their ability to provide visual information. By implementing machine learning (ML) and deep learning (DL) techniques, the DMS and OMS systems can extract information about the in-cabin environment, such as the state, attributes, and pose of the vehicle operator and / or occupant. The state and attributes of the vehicle operator and / or occupant can be extracted based on the local coordinates of the camera, while the pose of the vehicle operator and / or occupant can be estimated using extrinsic camera parameters, which refer to the position and orientation of the camera in the three-dimensional (3D) coordinates of the real world.

[0027] The extrinsic parameters have two components: rotation (R) and translation (t). Specifically, the extrinsic parameters of a camera include three rotation angles, i.e., rotations around the x, y, and z axes (e.g., roll, pitch, and yaw), and three translation values (e.g., a 3D vector with x, y, and z translation values). Thus, calibration of the extrinsic parameters of a camera (e.g., referred to herein as extrinsic calibration) can improve the efficiency of the DMS and OMS to detect the state, attributes, and pose of the vehicle operator and / or occupant.

[0028] Extrinsic calibration can be performed offline (e.g., not during routine driving), or extrinsic calibration can be performed online (e.g., while the vehicle operator is operating the vehicle). Offline calibration of extrinsic parameters typically relies on specially designed and placed reference objects, such as a chessboard, and can be performed at a factory or workshop. Online calibration of extrinsic parameters does not rely on specially designed and placed reference objects, but is based on images of the vehicle driver / occupant during vehicle operation. The goal of online extrinsic calibration is to estimate the extrinsic parameters (R, t) of a camera with respect to a reference on the vehicle coordinate system based on images of the vehicle driver or occupant, where the feature points include pixel locations / positions on the driver’s or occupant’s face or body.

[0029] When calibrating vehicle camera extrinsic parameters offline, a DL calibration model can be trained on multiple images to detect feature points and predict corresponding descriptions of the feature points. For the purposes of this disclosure, a feature point is an image pixel with a unique characteristic (e.g., visual property) of a subject that enables identification of the subject or subject pose. A feature point has 2D coordinates in an image.

[0030] A description of a feature point can contain information about the visual properties around the feature point. The description can be a vector with floating point / integer values. A library of feature points is constructed with the trained DL calibration model, containing detected 2D feature points and descriptions, and corresponding 3D feature points and descriptions. The library can be implemented as a reference table or lookup table, where during online extrinsic calibration, any 2D feature point can be mapped to a corresponding 3D feature point in the real world by matching the description of the 2D feature point to the corresponding description of a 3D feature point in the library of feature points. Online extrinsic calibration improves the flexibility and accuracy of the DMS and OMS life cycle compared to offline extrinsic calibration.

[0031] A feature matching based online extrinsic calibration method for in-cabin cameras is presented herein, which can robustly and accurately extrinsically calibrate a single camera of an in-cabin environment. The system includes three key aspects: a feature point detector and descriptor; feature matching; and a camera pose estimator. The feature point detector and descriptor are based on one or more DL calibration models (e.g., neural networks) that can robustly and accurately detect feature points and extract corresponding feature point descriptions. The DL calibration models can be trained without manual annotation or labeling, which can reduce the training time and / or cost of the neural networks. Thus, with the feature matching correspondence between the feature points contained in the library and the feature points acquired by the camera, the camera pose can be estimated based on the known Perspective-n-Point (PnP) algorithm.

[0032] Figure 1 An exemplary vehicle with an extrinsic calibration system integrated into a vehicle computer system is shown. Figure 2 An extrinsic camera calibration system for determining in-cabin camera extrinsic parameters is depicted. Figure 3 A general extrinsic calibration procedure using the extrinsic camera calibration system is shown. Figure 4 An image processing system for training a DL calibration model is shown. Figure 5 A DL calibration model training system for training a DL calibration model is shown. Figure 6 A neural network architecture implemented in the DL calibration model training system is depicted. Calibration of the online extrinsic camera calibration system proceeds according to the method described in Figure 7 . Figure 8A method for automatic generation of labels for training a DL calibration model is shown. The DL calibration model can be trained in accordance with the method of Figure 9 The trained DL calibration model can construct a feature point library in accordance with the method of Figure 10 The method of Figure 11 The trained DL calibration model can be used to implement ADAS intervention in accordance with the method described in Figure 12 Figure 13 An exemplary reference object (e.g., a checkerboard) used during calibration is shown placed inside the vehicle cabin, where the checkerboard is shown in detail in Figure 14

[0033] Turning now to the drawings, Figure 1 An exemplary vehicle 100 is shown schematically. The vehicle 100 includes an instrument panel 102, a driver seat 104, a first passenger seat 106, a second passenger seat 108, and a third passenger seat 110. In other examples, the vehicle 100 can include more or fewer passenger seats. The driver seat 104 and the first passenger seat 106 are located in the front of the vehicle, near the instrument panel 102, and can therefore be referred to as front seats. The second passenger seat 108 and the third passenger seat 110 are located in the rear of the vehicle and can be referred to as rear (or back) seats.

[0034] Additionally, the vehicle 100 includes a plurality of integrated speakers 114, which can be arranged around the periphery of the vehicle 100. In some embodiments, the integrated speakers 114 are electronically coupled to an electronic control system of the vehicle, such as a computing system 120, by a wired connection. In other embodiments, the integrated speakers 114 can be in wireless communication with the computing system 120. As an example, an occupant of the vehicle 100, such as a driver passenger, can select an audio file through a user interface 116, and the selected audio file can be projected through the integrated speakers 114. In some examples, an audio alert can be generated by the computing system 120 and can also be projected by the integrated speakers 114, such as will be explained in detail herein.

[0035] The vehicle 100 includes a steering wheel 112 and a steering column 122 through which a driver can input steering instructions for the vehicle 100. The vehicle 100 also includes one or more cameras 118. The camera 118 can be one of a plurality of cameras. In Figure 1 ​​In the illustrated embodiment, the camera 118 is positioned to the side of the driver seat 104, which can facilitate monitoring the driver from the side. However, in other examples, the camera 118 can be positioned elsewhere in the vehicle, such as on the steering column 122 and directly in front of the driver seat 104. Additionally, the camera 118 can be positioned to the side of the passenger seats 106, 108, and 110, or directly in front of the passenger seats 106, 108, and 110.

[0036] Additionally, as some examples, the camera 118 can be configured to monitor the driver seat 104 and / or the passenger seats 106, 108, and 110 for the presence of a driver and / or passenger, respectively. For example, the camera 118 can be configured to monitor the driver seat 104 and / or the passenger seats 106, 108, and 110 for the presence of a driver and / or passenger, respectively, by detecting a body, a head, a hand, a foot, a seatbelt, and / or a seat position. Figure 1 In the illustrated embodiment, the camera 119 can be positioned outside of the rear end of the vehicle 100, which can help monitor the position of the vehicle 100 in the lane and / or monitor the position of the vehicle 100 relative to other vehicles and / or the surrounding environment. In other examples, the camera 119 can be positioned elsewhere in the vehicle, such as outside of the front end of the vehicle 100 or to the side of the vehicle 100.

[0037] The camera 118 can include one or more optical (e.g., visible light) cameras, one or more infrared (IR) cameras, or a combination of optical and IR cameras with one or more perspectives. In some examples, the camera 118 can have an interior perspective as well as an exterior perspective. In some examples, the camera 118 can include more than one lens and more than one image sensor. For example, the camera 118 can include a first lens that directs light to a first visible light image sensor (e.g., a charge-coupled device or a metal-oxide-semiconductor) and a second lens that directs light to a second thermal imaging sensor (e.g., a focal plane array), thereby enabling the camera 118 to collect light of different wavelength ranges to produce both visible and thermal images. In some examples, the camera 118 can also include a depth camera and / or sensor, such as a time-of-flight camera or a LiDAR sensor.

[0038] In some examples, the camera 118 can be a digital camera configured to acquire a series of images (e.g., frames) at a programmable frequency (e.g., frame rate) and can be electronically and / or communicatively coupled to the computing system 120. Additionally, the camera 118 can output the acquired images to the computing system 120 in real-time, such that the computing system 120 and / or a computer network can process the images in real-time. As used herein, the term “real-time” denotes a process with no intentional delay, e.g., substantially contemporaneous with the occurrence of an event. In other examples, the camera 118 can acquire images to determine the state, attributes, and posture of a driver or passenger (e.g., occupant), such as will be described in detail herein.

[0039] The vehicle 100 can also include a driver seat sensor 124 coupled to or within the driver seat 104 and a passenger seat sensor 126 coupled to or within the first passenger seat 106. The rear seats can also include seat sensors, such as a passenger seat sensor 128 coupled to the second passenger seat 108 and a passenger seat sensor 130 coupled to the third passenger seat 110. The driver seat sensor 124 and the passenger seat sensor 126 can each include one or more sensors, such as a weight sensor, a pressure sensor, and one or more seat position sensors, that output measurement signals to the computing system 120. For example, the computing system 120 can use the output of the weight sensor or the pressure sensor to determine whether the respective seat is occupied, and if so, the weight of the person occupying the seat. As another example, the computing system 120 can use the output of the one or more seat position sensors to determine one or more of: a seat height, a longitudinal position relative to the dashboard 102 and the rear seats, and an angle (e.g., recline) of a seat back of the corresponding seat.

[0040] In some examples, the vehicle 100 also includes a driver seat motor 134 coupled to or positioned within the driver seat 104 and a passenger seat motor 138 coupled to or positioned within the first passenger seat 106. Although not shown, in some embodiments, the rear seats can also include seat motors. The driver seat motor 134 can be used to adjust seat positions, including seat height, longitudinal seat position, and angle of a seat back of the driver seat 104, and can include adjustment inputs 136. For example, the adjustment inputs 136 can include one or more toggle switches, buttons, and switches. The passenger seat motor 138 can be used to adjust seat positions, including seat height, longitudinal seat position, and angle of a seat back of the first passenger seat 106, and can include adjustment inputs 140. The adjustment inputs 140 can include one or more toggle switches, buttons, and switches. Although not shown, in some embodiments, the rear seats can be adjusted in a similar manner.

[0041] The computing system 120 can receive inputs and output information to the user interface 116. The user interface 116 can be included in, for example, a digital cockpit, and can include a display and one or more input devices. The one or more input devices can include one or more touchscreens, knobs, dials, hard buttons, and soft buttons for receiving user inputs from vehicle occupants.

[0042] The computing system 120 includes a processor 142 configured to execute machine readable instructions stored in a memory 144. The processor 142 can be single or multi-core, and programs executed by the processor 142 can be configured for parallel or distributed processing. In some embodiments, the processor 142 is a microcontroller. The processor 142 can optionally include components distributed across two or more devices, which can be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 142 can be virtualized and executed by a remotely accessible network computing device configured as a cloud computing configuration. For example, the computing system 120 can be communicatively coupled with the wireless network 132 through a transceiver 146, and the computing system 120 can communicate with network computing devices through the wireless network 132.

[0043] The computing system 120 can include a DMS 147 that can monitor a driver of the vehicle 100. For example, a camera 118 can be located on a front of the vehicle 100 (e.g., on the dashboard 102) or on a side of the vehicle 100 proximate to the driver seat 104, and can be positioned to view a face of the driver. The DMS can detect facial features of the driver. In some embodiments, the DMS 147 can be used to retrieve a driver profile of the driver based on the facial features, which can be used to customize a position of the driver seat 104, the steering wheel 112, and / or other components or software of the vehicle 100.

[0044] The computing system 120 can include an OMS 148 that can monitor one or more passengers of the vehicle 100. For example, a camera 118 can be located on a side of the vehicle 100 proximate to one or more of the passenger seats 106, 108, and 110, and can be positioned to view a face of a passenger of the vehicle.

[0045] The computing system 120 can include an ADAS 149 that can provide assistance to the driver based at least in part on the DMS 147. For example, the ADAS 149 can receive facial expression data from the DMS 147, and the ADAS 149 can process the facial expression data to provide assistance to the driver. For example, the ADAS 149 can process the facial expression data to determine whether the driver appears fatigued or stressed. In response to detecting a fatigue or stress condition of the driver, the ADAS 149 can alert the driver, or play music, or perform a different action to address the fatigue or stress condition of the driver. In some embodiments, the DMS 147 and / or the OMS 148 can be integrated into the ADAS 149.

[0046] In various embodiments, the trained ML or DL calibration model can be integrated into the DMS 147, OMS 148, or ADAS 149, which can facilitate detection of facial features or expressions. For example, the trained ML or DL calibration model can utilize sensor and / or camera data from the camera 118 to determine whether ADAS intervention is required. More particularly, the trained ML or DL calibration model can be trained to recognize unique features (e.g., visual characteristics) of a subject, thereby enabling recognition of the subject and / or interpretation of facial expressions or gestures of the subject.

[0047] As described in greater detail herein, the trained ML or DL calibration model can rely on prior external calibration of the camera 118, which can make it easier for the ADAS to recognize gestures of the vehicle operator and / or occupants.

[0048] Additionally or alternatively, the computing system 120 can communicate directly with networked computing devices through a short-range communication protocol, such as Bluetooth®. In some embodiments, the computing system 120 can include other electronic components capable of performing processing functions, such as a digital signal processor, a field-programmable gate array (FPGA), or a graphics board. In some embodiments, the processor 142 can include multiple electronic components capable of implementing processing functions. For example, the processor 142 can include two or more electronic components selected from a plurality of possible electronic components, including a central processor, a digital signal processor, a field-programmable gate array, and a graphics board. In still further embodiments, the processor 142 can be configured as a graphics processing unit (GPU), including parallel computing architecture and parallel processing capabilities.

[0049] Further, the memory 144 can include any non-transitory, tangible computer- readable medium having stored therein programming instructions. As used herein, the term “tangible computer-readable medium” is expressly defined to include any type of computer- readable storage. The example methods described herein can be implemented using encoded instructions (e.g., computer-readable instructions) stored on non-transitory computer- readable media, such as flash memory, read-only memory (ROM), random-access memory (RAM), a cache, or any other storage medium in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and / or for caching of the information).

[0050] The computer memory referred to herein as a computer-readable storage medium may include volatile and non-volatile, or removable and non-removable media for storing electronically formatted information (such as computer-readable program instructions or computer-readable program instruction modules, data, etc.), which may be standalone or as part of a computing device. Examples of computer memory may include any other media that can be used to store information in a desired electronic format and is accessible by at least a portion of one or more processors or a computing device. In various embodiments, memory 144 may include an SD memory card, an internal and / or external hard disk, a USB storage device, or similar modular memory.

[0051] Furthermore, in some examples, computing system 120 may include multiple subsystems or module tasks that perform specific functions related to image acquisition and analysis. As used herein, the terms "system," "unit," or "module" may include hardware and / or software systems that operate to perform one or more functions. For example, a module, unit, or system may include a computer processor, controller, or other logic-based means that performs operations based on instructions stored on a tangible and non-transitory computer-readable storage medium such as computer memory. Alternatively, a module, unit, or system may include a hardwired means that performs operations based on hardwired logic of the means. The various modules or units shown in the figures may represent hardware that operates based on software or hardwired instructions, software that directs the hardware to perform operations, or a combination thereof. For example, as referenced herein... Figures 2-13 As described in detail, the computing system 120 can perform online external calibration of the camera 118. Images received from the camera 118 can be input into a trained neural network model, which can then... Figure 9 Training is performed to detect and describe feature points of the cabin environment, thereby estimating the external parameters of camera 118 based on feature matching using an offline-developed feature point library.

[0052] Figure 2 An in-cabin camera calibration system 200 according to an embodiment is shown, which includes a vehicle computing system 202 communicatively coupled to a camera 218. The camera 218 provides image data to the computing system 202. The camera 218 can interact with... Figure 1 The camera 118 in the system is similar to or the same as the camera 118 in the system, and the computing system 202 may be similar to or the same as the camera 118 in the system. Figure 1FIG. 2 illustrates a non-limiting version of a computing system 202 that can be used in place of the computing system 120 in FIG. 1. The computing system 202 includes a processor 204 configured to execute machine readable instructions stored in a non-transitory memory 206. The processor 204 can be single- or multi-core and the programs executing thereon can be configured for parallel or distributed processing. In some embodiments, the processor 204 can optionally include components distributed across two or more devices, which can be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 204 can be virtualized and executed by a remotely accessible networking computing device configured as a cloud computing configuration.

[0053] The non-transitory memory 206 includes an image acquisition module 208, an image analysis module 210, and an advanced driver assistance system (ADAS) module 212. The ADAS module 212 can include a DMS module 222 and an OMS module 232 that monitor a vehicle operator and an occupant, respectively. The image acquisition module 208 can send and receive data with the camera 218. For example, the image acquisition module 208 can control acquisition settings of the camera 218, such as aperture, light sensitivity, depth of focus, field of view, shutter speed, frame rate, etc. In some examples, the camera 218 can operate at a frame rate in the range of 4-24 frames per second to capture images of the interior of the vehicle substantially continuously. In other examples, the frame rate can be lower, such as one frame per second or one frame per multiple seconds (e.g., 30-60 seconds), or higher (e.g., 30 frames per second). As one example, the frame rate can be selected based on the processing speed of the image analysis module 210 such that a comprehensive analysis of each image can be performed before the image analysis module 210 receives the next image in the sequence.

[0054] Further, the image acquisition module 208 can update the acquisition settings of the camera 218 based on feedback received from the image analysis module 210. For example, the image analysis module 210 can determine that the images acquired by the camera 218 are too dark and update one or more of the aperture, light sensitivity, and shutter speed at the image acquisition module 208 accordingly.

[0055] The image analysis module 210 can access images / videos stored in memory (e.g., an image library) and analyze images received from the camera 218 in real-time to identify one or more features within each received image. As one example, the image analysis module 210 can receive a plurality of images from the camera 218. The plurality of images can be utilized to train a neural network to detect and describe feature points, where, for purposes of the present disclosure, a feature point is a pixel location of a unique feature (e.g., visual characteristic) of a subject that enables identification of the subject or subject position / posture. The feature points can be utilized to construct a feature point library for feature point matching, as described in greater detail below in Figure 10 FIG. 3.

[0056] Additionally, the image analysis module 210 can receive online images from the camera 218. The images can be input into a trained neural network model to detect and describe feature points of the images, which can be matched with a library of feature points to determine extrinsic parameters of the camera 218 in an online extrinsic calibration process. Online extrinsic calibration of the camera 218 can improve the quality accuracy of images used for algorithms in the ADAS module 212 that determine the state, attributes, and posture of the vehicle operator and / or occupants.

[0057] Further, it can be appreciated that, in some examples, the image analysis module 210 can be included in the in-vehicle computing system 202 or accessed via a user-provided mobile computing system such as a smartphone, computer, or tablet. As another example, the image analysis module 210 can be included and accessed remotely (e.g., via a wireless network such as the wireless network 132 shown in FIG. 1) via a networked computing system such as a cloud computer or similar computing environment. Figure 1 Figure 2 Thus, while the image analysis module 210 is shown in the example of FIG. 1 as being within the computing system 202, in other implementations at least portions of the image analysis module 210 can be stored in a computing device and / or network that is outside of the computing system 202 and communicatively coupled to the computing system 202.

[0058] The vehicle can include a user interface 216, which can include one or more speakers 214 and a display 220. The ADAS 212 and / or DMS 222 can detect certain states, attributes, and postures of the driver and can use the user interface 216 to notify the occupants. As one example, the ADAS module 212 can communicate with the user interface 216 to output an alert via the display 220 and / or the speakers 214. The speakers 214 can be in-vehicle speakers such as the integrated speakers 114 of FIG. 1 or speakers of a smartphone running a companion application. Similarly, the display 220 can be integrated in the vehicle (e.g., integrated in the dashboard of the vehicle) or can be a display of a smartphone running a companion application. For example, the display 220 can output an alert message / symbol and the speakers 214 can play an alert sound / cue. Figure 1

[0059] Referring now to FIG. 3, Figure 3 schematic calibration diagram 300 shows calibration of a vehicle camera (e.g., the camera 118 of the vehicle 100 in FIG. 1 and / or the camera 218 of the vehicle 200 in FIG. 2). Figure 1 Figure 2 ​​​The process involves a two-stage external calibration of the in-cabin camera (218) within the vehicle. This two-stage process includes: a first stage 302, for training a deep neural network model and constructing a feature point library using images acquired by the camera; and a second stage 304, for performing online external calibration of the in-cabin camera using the trained neural network model and the feature point library. The first stage 302 can be performed offline, i.e., in a laboratory or factory, rather than during vehicle operation. The second stage 302 can be performed online, i.e., during vehicle operation. The first stage 302 includes a data acquisition step 312, a deep model training step 322, and a feature point library construction step 332. The second stage 304 includes a real-time data acquisition step 314, a deep model inference step 324, a feature matching step 334, and a pose estimation step 344.

[0060] During the first phase 302, data acquisition step 312 includes data acquisition from an in-cabin camera (e.g., Figure 1 Images are acquired via a camera (118) to train the DL calibration model. The acquired images may include images of specially designed and placed reference objects, such as a chessboard placed on the driver's seat. In some implementations, images can be acquired via an external camera (e.g., Figure 1 The camera 119 acquires external images to train the DL calibration model. Data acquisition step 312 also includes classifying vehicle images into various vehicle image subsets. Deep model training step 322 includes automatically generating labels for vehicle images and feature point / background point pairs in the image subsets, and training the DL calibration model to predict the probability that a location in the image is a 2D feature point as a first output, and predicting a description of that location as a second output, which may include information about the visual characteristics surrounding that location. Feature point library construction step 332 includes building a library of corresponding 3D feature points and descriptions based on each 2D feature point and description using the trained DL calibration model.

[0061] Regarding the second phase 304, the real-time data acquisition step 314 includes acquiring images of the vehicle operator and / or occupants from the in-cabin camera during vehicle operation. The deep model inference step 324 includes detecting feature points and generating corresponding descriptions for each detected feature point using a trained DL calibration model. The feature matching step 334 includes matching 2D feature points from the images acquired during vehicle operation with 3D feature points in a library based on the extracted feature point descriptions. The pose estimation step 344 includes estimating the camera's extrinsic parameters using the 2D and 3D feature points and descriptions of the images. (See below for reference.) Figures 4-14 The first stage 302 and the second stage 304 are described in more detail.

[0062] refer to Figure 4According to one embodiment, an image processing system 400 is shown, where the image processing system 400 includes a camera calibration system 402. The image processing system 400 is incorporated into a computing system that is independent of the vehicle 100 (e.g., is not the computing system 120 of the vehicle 100). For example, the image processing system 400 can be installed on a server. Figure 1

[0063] The camera calibration system 402 can be operatively / communicatively coupled to a user input device 432 and a display device 434. At least in some examples, the user input device 432 can comprise the user interface 115 of the computing system, and the display device 434 can comprise the display device 118 of the computing system.

[0064] The camera calibration system 402 includes a processor 404 configured to execute machine readable instructions stored in a non-transitory memory 406. The processor 404 can be single- or multi-core, and programs executing thereon can be configured for parallel or distributed processing. In some embodiments, the processor 404 can optionally include individual components distributed among two or more devices, which can be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 404 can be virtualized and executed by remotely accessible networking computing devices configured as a cloud computing arrangement.

[0065] The non-transitory memory 406 can store a neural network module 408, a network training module 410, an inference module 412, and vehicle image data 414. The neural network module 408 can include at least a DL calibration model, as well as instructions for implementing the DL calibration model to detect feature points and generate feature point descriptions, as described in greater detail below. The neural network module 408 can include trained and / or untrained neural networks, and can also include various data or metadata related to the neural network(s) stored therein.

[0066] The non-transitory memory 406 can also store a training module 410 that includes instructions for training one or more neural networks stored in the neural network module 408. The training module 410 can include instructions that, when executed by the processor 404, cause the camera calibration system 402 to perform one or more steps of the following methods: a method 700 for preparing and calibrating an in-cabin calibration system, a method 800 for labeling training data in a training data set, a method 900 for training a neural network model using a training data set, and a method 1000 for building a feature point library, described below with reference to Figures 7-10 ​A more detailed discussion is provided. In some embodiments, the training module 410 includes instructions to implement one or more gradient descent algorithms, apply one or more loss functions, and / or training routines for adjusting parameters of one or more neural networks of the neural network model 408.

[0067] The non-transitory memory 406 also stores an inference module 412 that contains instructions to test new data using the trained DL calibration model. After the DL calibration model is trained, the trained DL calibration model can be used by a vehicle computing system (e.g., the computing system 120) to perform external online calibration of a camera, as shown in Figure 11

[0068] The non-transitory memory 406 also stores vehicle image data 414. The vehicle image data 414 can include vehicle images acquired, for example, by in-cabin cameras and external cameras. For example, the vehicle image data 414 can store images acquired by in-cabin cameras and external cameras for different vehicles. In some embodiments, the vehicle image data 414 can include a plurality of training sets. The vehicle image data 414 can include images acquired from cameras with constrained camera poses (i.e., the images are acquired using the same camera pose) and images acquired from cameras with unconstrained camera poses (i.e., the images are acquired using different camera poses).

[0069] In some embodiments, the non-transitory memory 406 can include components disposed at two or more devices that can be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the non-transitory memory 406 can include a remotely accessible networked storage device configured as a cloud computing configuration.

[0070] The user input device 432 can include one or more of a touchscreen, a keyboard, a mouse, a trackpad, a motion-sensing camera, or other devices configured to enable a user to interact with and manipulate data within the camera calibration system 402. In one example, the user input device 432 can enable a user to select vehicle images for training the DL calibration model.

[0071] The display device 434 can include one or more display devices that utilize virtually any type of technology. In some embodiments, the display device 434 can include a computer monitor. The display device 434 can be combined with the processor 404, the non-transitory memory 406, and / or the user input device 432 in a shared housing, or can be a peripheral display device, and can include a monitor, a touchscreen, a projector, or other display devices known in the art that can enable a user to interact with various data stored in the non-transitory memory 406.​

[0072] It can be appreciated that, Figure 4 The illustrated camera calibration system 402 is for illustration and not limitation. Another suitable in-cabin camera calibration system can include more, less, or different components.

[0073] Referring to Figure 5 , an example of a DL calibration model training system 500 is shown, which can be used to train a DL calibration model 524. The DL calibration model 524 can be trained to detect feature points of an input image and predict a description of the detected feature points. According to one or more operations described in more detail below with reference to Figure 9 , after training is complete, the trained DL calibration model 530 can be used to estimate extrinsic parameters of a vehicle in-cabin camera (e.g., camera 118 of vehicle 100 in Figure 1 ). Estimating extrinsic parameters can also include matching 2D feature points detected in an input image with 3D feature points in a feature point library 538, where the feature point library 538 is generated using the trained DL calibration model 530. The DL calibration model training system 500 can be implemented by a camera calibration system, such as camera calibration system 402 of Figure 4 .

[0074] The DL calibration model 524 can be stored in a neural network module 518 of the camera calibration system, which can be the same as or similar to neural network module 408 of camera calibration system 402 of Figure 4 . The neural network module 518 can include instructions for training, validating, and testing the DL calibration model 524.

[0075] The DL calibration model 524 can be trained on a plurality of training pairs 520 and testing pairs 522, where each of the training pairs 520 and testing pairs 520 can include input data and target ground truth data. The training pairs 520 and testing pairs 522 can be stored in a training module 516 of the camera calibration system, which can be the same as or similar to training module 410 of camera calibration system 402 of Figure 4 .

[0076] In various embodiments, the input data can include one or more pairs of images 504, which can be captured by a camera 502 of a vehicle (e.g., camera 118 of vehicle 100 in Figure 1The input data can include pairs of images 504 captured by the in-cabin camera 118 or the external camera 119. For example, the input data can include four pairs, or eight pairs, or 16 pairs of images 504. The images 504 can include images of a reference object (e.g., a checkerboard) placed in the front seat (e.g., the seat 104) of the vehicle. The images 504 can be classified into at least two sub-datasets, such as constrained camera pose images 508 and unconstrained camera pose images 510. A constrained camera pose refers to a camera mounted in a fixed pose and having a specific set of extrinsic parameters (e.g., rotation R and translation t) with respect to the vehicle coordinate system. In other words, a constrained camera pose refers to images captured using the same in-cabin camera and shooting pose. An unconstrained camera pose refers to a camera mounted in any pose and having variable sets of extrinsic parameters with respect to the vehicle coordinate system.

[0077] Additional pairs of images using unconstrained camera poses can be obtained using homographies to enable matching of 2D feature points and descriptors between the pairs of images. In particular, a homography can be applied to a first image such that a second image generated using the homography is different from the first image. The first and second images can have the same height and width. Additionally, the second image can be different from the first image in terms of rotation R, translation t, and scaling of the subject in the image.

[0078] The feature point detection and description process 512 can be performed on the constrained camera pose images 508 and the unconstrained camera pose images 510. The feature point detection and description process 512 includes detecting the locations of feature points and background points in the constrained camera pose images 508 and the unconstrained camera pose images 510. In various embodiments, the feature point detection and description process 512 can be performed using one or more tools and / or algorithms known in the art. In one embodiment, the feature point detection and description process 512 can be performed using a scale-invariant feature transform (SIFT) algorithm. The output of the feature point detection and description process 512 can be used to assign labels to the feature points and non-feature (e.g., background) points. The results can also include encoded descriptions of the feature points and background points, which can be used to label pairs of images, as described below with reference to Figure 7

[0079] The DL calibration model 524 can be jointly trained on two different tasks. The DL calibration model 524 can generate a first output based on a first task and a second output based on a second task. The first task can be a feature point detection task, in which the DL calibration model 524 learns to detect feature points in either or both images of an input pair of images. In particular, the DL calibration model 524 can output, for each pixel of each image of an input pair of images, a prediction of whether the point at the pixel location in the respective image is a feature point or a background point. A feature point is a pixel location that has unique characteristics (e.g., visual properties) such as edges. ​

[0080] For the first feature point detection task, ground truth data for each training pair 520 can include labels assigned by the feature point detection and description process 512 for each pixel of each image of the input image pair to indicate whether the point at the pixel location is a feature point or a background point. For example, the labels can include a value of 0 to indicate that the point is a feature point or a value of 1 to indicate that the point is a background point. Thus, the DL calibration model 524 can output an indication of whether the point at the pixel location is a feature point or a background point. In some embodiments, the labels can be a single value (e.g., 0 or 1) as described above. In other embodiments, different values, encodings, and / or arrays of values can be used to indicate whether the point is a feature point or a background point.

[0081] In at least one embodiment, the output of the DL calibration model 524 can include an array with two values, where the first value of the two values is the probability that the point at the pixel location is a feature point and the second value of the two values is the probability that the point at the pixel location is a background point. For example, a first array [1, 0] can indicate that the point is a feature point and a second array [0, 1] can indicate that the point is a background point. For example, for a certain point of an input image, the DL calibration model 524 can output a first output array [0.85, 0.15], which can indicate that there is an 85% probability that the point is predicted to be a feature point. Alternatively, the DL calibration model 524 can output a second output array [0.2, 0.8], which can indicate that there is an 80% probability that the point is predicted to be a background point.

[0082] Using a two-value array can have advantages depending on the loss function to be applied. For example, when using a single value output, a sigmoid cross-entropy loss function can be used for the detection task. When using a two-value array output, a Softmax negative log-likelihood loss function can be used for the detection task.

[0083] The second task can be a feature point description task, in which the DL calibration model 524 learns to predict a description of a feature point in either or both images of the input image pair. Specifically, the DL calibration model 524 can output, for each pixel of each image of the input image pair, an array of values encoding a description of the feature point. The array of values can be a numerical representation of visual characteristics around the feature point that distinguishes one feature point from different feature points. The feature point description can have various characteristics, for example, each value of the feature point description can be between -1 and 1; the feature point description can not be human-readable; and the feature point description can be normalized, for example, by Euclidean normalization or L2 normalization. For example, each feature point description can be a vector of length 256, of floating point data type, as follows:

[0084] [-1.43169031e-01 4.58271384e-01 -1.62901267e-01 -9.65002239e-01 -7.17037261e-01 -8.68379533e-01 6.71189949e-02 -5.21657281e-02 2.39461720e-01 -6.16888225e-01 8.75887394e-01 6.35065436e-01 -2.57862490e-02 3.20669636e-03 5.76226473e-01 -3.28777730e-01 -4.57405478e-01 -4.16866779e-01 -2.27968097e-01 -1.42556045e-03 1.73203960e-01 -6.28954291e-01 -2.34341636e-01 4.26671445e-01 1.21125684e-03 5.80443144e-01 -4.99145150e-01 3.49743128e-01 -2.11896852e-01 8.78908753e-01 7.64355540e-01 -5.22589147e-01 2.72085488e-01 5.54731190e-01 3.82850558e-01 2.53446668e-01 3.39921325e-01 -1.68883428e-01 -9.34922993e-01 5.87750636e-01 6.89451158e-01 -1.71338841e-01 5.46122551e-01 7.27262378e-01 6.45273268e-01 3.72664660e-01 2.46289998e-01 -3.18505615e-01 -3.03976774e-01 -1.08477525e-01 4.75602865e-01 -5.36214888e-01 3.86447906e-01 3.87157202e-01 -3.76178890e-01 -7.75162876e-02 -2.44647771e-01 3.88342381e-01 1.23110577e-01 -2.89371341e-01 3.52955461e-01 -1.89600699e-02 -2.12719738e-01 -5.27302861e-01 -8.97994041e-02 1.3 061 2120e-01 -4.66043055e-01 2.15788528e-01 -4.05010879e-01 -1.19115345e-01 3.42263430e-01 4.71167266e-01 -4.28491294e-01 -1.74742043e-01 6.27758741e-01 2.61402637e-01 1.86255246e-01 -2.21359700e-01 -5.95517635e-01 -7.48736933e-02 4.49042469e-01 5.70093215e-01 -2.85882741e-01 -2.86982089e-01 -3.23122501e-01 -1.81990162e-01 9.86513495e-02 -3.35428774e-01 -6.73396215e-02 -7.36446559e-01 9.56638396e-01 1.80800140e-01 -1.02010280e-01 6.29265487e-01 -1.51900202e-01 -1.31430030e-01 2.68870711e-01 -3.88126642e-01 6.13419175e-01 -2.88700238e-02 2.10499957e-01 -3.22122365e-01 1.15255229e-01 -3.04747492e-01 -3.52769971e-01 1.44565836e-01 2.02660516e-01 -4.77666736e-01 -1.01362363e-01 -2.42617667e-01 -6.98205471e-01 -1.29087687e-01 -4.10853803e-01 -2.66897500e-01 -1.26621351e-01 -4.21785444e-01 -4.97935086e-01 1.16082102e-01 -1.44718587e+00 -6.26395047e-01 3.21106195e-01 9.15868729e-02 4.33236808e-01 -2.10027426e-01 1.40686095e-01 -6.03126109e-01 6.59793437e-01 4.38640624e-01 4.29766238e-01 2.1 370 10517e-01 -7.52806887e-02 -4.95135516e-01 -2.87820548e-01 3.89957935e-01 -1.37962446e-01 -7.70931184e-01 -4.24577138e-06 -3.29461172e-02 8.46419483e-03 -4.49042261e-01 -1.46077070e-02 8.03662956e-01 8.05445239e-02 -2.96235561e-01 -8.08543637e-02 -3.30588102e-01 -6.74770027e-02 1.17786482e-01 -2.31662884e-01 9.97441292e-01 -3.27596277e-01 -1.08897902e-01 6.57808125e-01 -7.97800243e-01 -1.03186488e-01 -4.29404408e-01 -4.59218949e-01 2.39089713e-01 1.72060460e-01 7.54326284e-01 6.24122322e-01 1.13830939e-01 1.71436444e-01 1.11234319e+00 3.78195792e-01 2.70797670e-01 2.33033281e-02 -3.58675152e-01 6.89783320e-02 5.64170539e-01 6.52764440e-01 2.42922097e-01 3.54792804e-01 -5.60296834e-01 -9.65531096e-02 7.44043827e-01 1.74399897e-01 -1.97993472e-01 9.34646651e-02 -9.12238598e-01 -1.67903960e-01 -1.70885161e-01 -5.69103718e-01 2.25428969e-01 4.15478855e-01 4.70013827e-01 8.25168788e-01 3.41797695e-02 -1.05919838e+00 2.74597853e-01 6.25017464e-01 4.14869398e-01 6.05404496e-01 4.02130932e-01 -1.20958388e-01 -1.90321550e-01 -8.04612994e-01 -3.17567624e-02 4.70515490e-01 3.50759387e-01-5.17713241e-02 -3.94614153e-02 -5.01945913e-01 1.71834037e-01 3.44270110e-012.13412657e-01 -1.27609670e+00 -4.41648245e-01 -1.57617524e-01 4.22273725e-014.70683932e-01 -4.12169518e-03 -7.08442703e-02 2.22695559e-01 4.22131903e-021.11095881e+00 -3.19704264e-01 3.50734860e-01 -3.90814453e-01 3.63916457e-023.78545493e-01 4.71075058e-01 -8.70998621e-01 -2.62030244e-01 -4.84437734e-015.62739134e-01 -8.11361730e-01 -4.22443628e-01 -2.15263188e-01 -2.83228219e-01 7.48567164e-01 1.25580892e-01 -2.23700665e-02 7.06127107e-01 -6.42825484e-01 -3.03269535e-01 -3.00211273e-02 -4.00734067e-01 2.97484547e-01 -1.18004173e-01 6.47764802e-01 -8.27421904e-01 -2.08601549e-01 1.85368642e-01-1.65401697e-01 1.83179334e-01 7.30600297e-01 -5.28879762e-01 1.06119502e+001.02211368e+00 6.10878527e-01 -4.40691859e-01 -4.36311185e-01 -1.21444136e-01-7.27298930e-02 1.65899292e-01].

[0085] Thus, when comparing the first feature point and the second feature point, the Euclidean distance between the feature points located at the same pixel position can be smaller, while the Euclidean distance between the feature points not located at the same pixel position can be larger.

[0086] For the second feature point description task, no additional ground truth data needs to be input into the DL calibration model 524. Instead, the DL calibration model 524 can train the feature point description task in an unsupervised manner based on the distances between the pairs of feature points and background points in the input image pairs, as described below with reference to Figure 9 In more detail.

[0087] The DL calibration model training system 500 can include a dataset generator that can generate the training pairs 520 and the testing pairs 522 from the labeled images produced by the feature point detection and description process 512. In various embodiments, the input images and the corresponding labels can be combined into a 3D data array with a height H, a width W, and a depth D, where H and W are also the dimensions of the constrained camera pose images 508 and the unconstrained camera pose images 510. The 3D data array can include one or more pairs of images, and for each image, there is a corresponding 2D ground truth data array for each pixel of the image. For example, the depth of one 3D data array can be 4, including a first 2D image located at the first layer of the 3D data array; a 2D ground truth data array for each pixel of the first 2D image located at the second layer of the 3D data array; a second 2D image located at the third layer of the 3D data array; and a second 2D ground truth data array for each pixel of the second 2D image located at the fourth layer of the 3D data array. The dataset generator can designate the 3D data array as a training pair 520, or as a testing pair 522. The testing pairs 522 can be used to evaluate the performance of the DL calibration model 524. In addition, multiple training pairs 520 can be assigned to a validation set to ensure that there is enough training data to prevent overfitting, such that the DL calibration model 524 learns to map features specific to the training set samples that do not exist in the testing set.

[0088] In one embodiment, the 3D data arrays can be randomly assigned to the training pair 520 or the testing pair 522 in a pre-determined ratio. For example, the 3D data arrays can be randomly assigned to the training pair 520 or the testing pair 522 such that 90% of the generated 3D data arrays are assigned to the training pair 520 and 10% of the generated 3D data arrays are assigned to the testing pair 522. Alternatively, the 3D data arrays can be randomly assigned to the training pair 520 or the testing pair 522 such that 85% of the generated 3D data arrays are assigned to the training pair 520 and 15% of the generated 3D data arrays are assigned to the testing pair 522. It should be understood that the examples provided herein are for illustrative purposes only and that the 3D data arrays can be assigned to the training pair 520 dataset or the testing pair 522 dataset by different procedures and / or in different ratios without departing from the scope of the present disclosure.

[0089] The DL calibration model training system 500 can include a validator 520 that validates the performance of the DL calibration model 524 against the testing pair 522. The validator 526 can take as input the trained or partially trained DL calibration model 524 and the dataset of the testing pair 522 and can output an evaluation of the performance of the trained or partially trained DL calibration model 524 on the dataset of the testing pair 522.

[0090] Once the DL calibration model 524 is validated, the trained DL calibration model 530 can be used to generate a library of feature points 538. The trained DL calibration model 530 can be stored in an inference module 518 of the camera calibration system, which can be the same or similar to the inference module 412 of the camera calibration system 402. The inference module 528 can contain instructions for deploying the trained DL calibration model 530 to detect one or more feature points and predict their corresponding descriptions from one or more new images 504 generated by the camera 502. Figure 4

[0091] The library of feature points 538 can be constructed using the triangulator 536 based on the results of the feature matcher 536 and the camera pose estimator 532. The trained DL calibration model 530 can detect one or more feature points and corresponding descriptions from images 504 that are corrected for camera distortion using the distortion corrector 506. The feature matcher 536 can use the one or more feature points and corresponding descriptions to identify pairs of feature points. The corrected images 504 can be used by the camera pose estimator 532 to determine the camera pose of each corrected image.

[0092] Turning to Figure 6 , an architectural diagram of a convolutional neural network (CNN) 600 is shown, according to one embodiment. The CNN 600 can be used to estimate the extrinsic parameters of an in-cabin camera as part of an extrinsic calibration procedure, as referenced to FIG. 1. The CNN 600 can be trained using the DL calibration model training system 500 of FIG. 5.​Figure 7 In more detail, the CNN 600 can be trained in an external DL calibration network training system, such as the training system 500 described with reference to Figure 5 In particular, the CNN 600 can be a non-limiting example of the DL calibration model 524.

[0093] The CNN 600 can have an encoder / decoder architecture, including an encoding portion 604 and a decoding portion 606. The encoding portion 604 includes an encoder 604a. The decoding portion 606 includes a first decoder 606a and a second decoder 606b. The encoder 604a, the first decoder 606a, and the second decoder 606b can each include a plurality of convolutional layers that constitute a feature map, represented in the figures as a solid rectangle. Figure 6 In one embodiment, the CNN 600 can be implemented using a U-NET. The first decoder 606a can be trained to perform a first task, and the second decoder 606b can be trained to perform a second task. As described above with reference to Figure 5 The first task can include detecting feature points in an input image, and the second task can include predicting a description of the detected feature points.

[0094] The CNN 600 includes solid rectangles corresponding to feature maps, where a feature map includes a height (as represented by the length from top to bottom), a width (as represented by the length from left to right), and a depth (as represented by the length from front to back). Figure 6 The height corresponds to a y spatial dimension in the xy plane, the width, assuming its size is equal to the height, corresponds to an x spatial dimension in the xy plane, and the depth corresponds to a number of features within each feature channel. Figure 6 The first decoder 606a can be trained to perform a first task, and the second decoder 606b can be trained to perform a second task. As described above with reference to Figure 6 The first task can include detecting feature points in an input image, and the second task can include predicting a description of the detected feature points.

[0095] From the input layer 612, the input 602 can be input to the CNN 600 and mapped to a first set of features. The input 602 can be a three-dimensional (3D) data structure having a width 620, a height 622, and a depth 624. The input 602 can include one or more pairs of 2D input images, where the width of the input images can be equal to the width 620 and the height of the input images can be equal to the height 622. The depth 624 can depend on the number of pairs of images included in the input 602. For example, if one pair of images is included in the input 602, the depth of the input 602 can be 2 (e.g., storing 2 images). At the first layer depth 624, one grayscale pixel intensity value can be included for each pixel of the first image. At the second layer depth 624, one grayscale pixel intensity value can be included for each pixel of the second image. In other examples, additional pairs of images can also be included in the input 602, and / or red / green / blue (RGB) pixel intensity values can be used, which can increase the depth 624. For example, if two pairs of images are included in the input 602, and each image includes RGB pixel intensity values, the depth 624 can be 12 (e.g., four images times three color values per image). Each pixel intensity value can be input to a single neuron of the input layer 612.

[0096] The CNN 600 includes a series of mappings from the input 602 received at the input layer 612, through a plurality of feature maps, to a first output layer 614 of the first decoder 606a and a second output layer 616 of the second decoder 606b. The first output 608 can be generated at the first output layer 614, and the second output 610 can be generated at the second output layer 616.

[0097] Each feature map can receive input from a previous feature map and can transform / map the received input to an output to produce a next feature map. Each feature map can include a plurality of neurons, where in some embodiments, each neuron can receive input from a subset of neurons of a previous layer / feature map and can compute a single output based on the received input, where the output can be propagated to a subset of neurons in a next layer / feature map. A feature map can be described using spatial dimensions, such as length and width (which can correspond to features for each pixel of an input image), where the dimensions refer to the number of neurons that make up the feature map (e.g., the number of neurons along the length of a specified feature map and the number of neurons along the width of a specified feature map).

[0098] In some embodiments, a neuron of a feature map can compute an output by performing a dot product on received inputs using a set of learned weights (each set of learned weights can be referred to herein as a filter), where each received input has a unique corresponding learned weight, where the learned weights are learned during CNN training.

[0099] The first output 608 can be a 3D data structure having a width 630, a height 632, and a depth 634. The width 630 and the height 632 can be equal to the width 620 and the height 622, respectively (e.g., the width and height of the input images). The depth 634 can include a plurality of layers depending on the number of image pairs contained in the input 602. For example, if one pair of images is contained in the input 602, a first layer of the output 608 can include a first value for each pixel of the first input image indicating whether the location of the pixel is a feature point, a second layer of the output 608 can include a second value for each pixel of the first input image indicating whether the location of the pixel of the first input image is a background point, a third layer of the output 608 can include a third value for each pixel of the second input image indicating whether the location of the pixel is a feature point, and a fourth layer of the output 608 can include a fourth value for each pixel of the second input image indicating whether the location of the pixel is a background point. Thus, through the first output 608, the CNN 600 can classify the points (pixel locations) of various input images (e.g., pairs of input images) as either feature points or background points.

[0100] The second output 610 can be a 3D data structure having a width 640, a height 642, and a depth 644. The width 640 can be equal to the width 620 and the width 630, and the height 642 can be equal to the height 622 and the height 632 (e.g., the width and height of the input images). The depth 644 can correspond to an array of values, where the array of values can be an encoded description of the points (e.g., feature points or background points) at the corresponding pixel’s location. Thus, through the second output 610, the CNN 600 can output a predicted description of each feature point or background point detected by the first decoder 606a. The CNN 600 illustrates the feature map transformations that occur as the input images propagate through the neuron layers of the convolutional neural network to output the point classifications and predicted descriptions.

[0101] The weights (and biases) of the convolutional layers in the CNN 600 are learned during training, as will be discussed in more detail below with reference to Figure 9 The difference between the output of the CNN 600 and the target output (e.g., ground truth data contained in the respective training pair) can be backpropagated through the layers of the CNN 600 to update the weights (and biases) of the convolutional layers. In other words, the weights in the convolutional layers of the first decoder 606a can be adjusted based on the difference between the output 608 and the true classifications provided in the input 602 according to a first loss function. The weights in the convolutional layers of the second decoder 606b can be adjusted based on the difference between the descriptions of the point pairs selected from the image pairs of the input 602 according to a second loss function. The CNN 600 can be trained on a plurality of training data pairs.

[0102] It should be appreciated that the present disclosure encompasses neural network architectures that include one or more regularization layers, including batch normalization layers, dropout layers, Gaussian noise layers, and other regularization layers known in the art of machine learning that can be used during training to mitigate overfitting and improve training efficiency while reducing training time. The regularization layers are used during training of the CNN and are deactivated or removed during post-training implementation of the CNN. These layers can be interspersed between the layers / feature maps shown, or can replace one or more of the layers / feature maps shown. Figure 6 It should be appreciated that the CNN 600 architecture and configuration shown in

[0103] It should be appreciated that the CNN 600 architecture and configuration shown in Figure 6 is for illustration only and not a limitation, and other suitable neural networks can be used to estimate extrinsic parameters of an in-cabin camera without departing from the scope of the present disclosure.

[0104] Referring now to Figure 7 , a method 700 for extrinsic calibration of a vehicle in-cabin camera (e.g., camera 118 in Figure 1 The extrinsic calibration of the in-cabin camera can include a first phase in which a DL calibration model is trained to estimate extrinsic parameters of the in-cabin camera. The first phase can be the first phase 302 of Figure 3 The extrinsic calibration of the in-cabin camera can also include a second phase in which the trained DL calibration model is used to perform extrinsic calibration on the vehicle during vehicle operation. The second phase can be the second phase 304 of Figure 3 The first phase of the method 700 can be implemented using the DL calibration model training system 500 of Figure 5 and / or the camera calibration system 402 of Figure 4 The second phase of the method 700 can be implemented using the computing system 202 of Figure 2 .

[0105] One or more operations in the method 700 related to the first phase can be stored in a non-transitory memory and executed by a processor, e.g., the non-transitory memory 406 and processor 404 of the camera calibration system 402, respectively. One or more operations in the method 700 related to the second phase can be stored in a non-transitory memory 206 and executed by a processor 204 of the computing system 202. Figure 2 At the end of the first phase, the trained DL calibration model can be deployed within a DMS or OMS of an ADAS (e.g., the DMS 222 or OMS 232 of the ADAS 212 of Figure 2 ) so as to perform the second phase.

[0106] At 702, method 700 includes receiving multiple images acquired via an in-cabin camera. The multiple images may include images acquired from the same, constrained, or fixed camera pose (referred to herein as constrained images) and / or images acquired from various unconstrained camera poses (referred to herein as unconstrained images). A constrained camera pose refers to a camera mounted in a fixed pose with a reference vehicle coordinate system having a specific set of extrinsic parameters (e.g., rotation R and translation t). In other words, a constrained camera pose refers to images captured using the same in-cabin camera and shooting pose. An unconstrained camera pose refers to a camera mounted in any pose with a reference vehicle coordinate system having several variable sets of extrinsic parameters. The in-cabin camera can acquire multiple images of the in-cabin environment. Furthermore, in some embodiments, the multiple images may include images from external vehicle cameras, which may contain richer information than those from in-cabin cameras, potentially increasing the representativeness of the DL calibration model. External cameras (which may be...) Figure 1 The camera (119) can acquire multiple images of the vehicle's external environment.

[0107] At 704, method 700 includes creating dataset D using images of constrained and unconstrained camera poses from both in-cabin and external cameras. 模型 D 模型 It can be used to generate training data to train a DL calibration model. Dataset D 模型 This can include multiple subsets of data, including data acquired using constrained camera poses and data acquired from any environment using unconstrained camera poses. For example, a first subset may include images acquired using various constrained camera poses. A second subset may include additional datasets acquired using the same constrained camera poses. A second subset may include images acquired using unconstrained camera poses. A second subset may include additional datasets acquired using specific unconstrained camera poses, as well as images generated by applying a homography matrix to images with specific unconstrained camera poses, as referenced below. Figure 7 As stated above.

[0108] In one implementation, a specially designed reference object can be placed inside the vehicle. The specially designed reference object may be an object with a highly structured geometry. In various implementations, a chessboard can be used, which can be placed on the driver's seat or passenger seat (e.g., within the field of view of a camera). The in-cabin camera can acquire images of the chessboard in either a constrained or unconstrained camera pose. One advantage of using a chessboard is that existing tools and / or algorithms for detecting chessboard feature points can be utilized during the labeling of training samples, as described in more detail below.

[0109] Images acquired using the same constrained camera pose can contain the same captured content and objects. However, the color, brightness, and contrast of images acquired using the same constrained camera pose can vary. For example, images acquired at different times of day with an in-cabin camera mounted in a constrained camera pose can exhibit different brightness characteristics due to changes in the position of the sun.

[0110] At 706, the method 700 includes creating a second dataset D 库 containing images from unconstrained camera poses. 库 The images contained in D Figure 10 are described in more detail below with reference to Figure 10 . Each image can be captured with a different camera pose (e.g., rotation R and translation t) so that triangulation can be applied during the construction of the feature library. By acquiring images with different camera poses, pixel locations with different visual characteristics in a first image can be matched to the same pixel locations in a second image, even if the images are acquired using different cameras or different poses.

[0111] The dataset D 库 may include images of a reference object placed inside the cab. In various embodiments, the reference object is a checkerboard. The checkerboard can serve as a reference to accurately estimate the camera pose and the scaling factor between the image pixels and the real world. By acquiring images with the checkerboard, pixel locations with unique characteristics in one image can be matched to the same pixel locations in another image, despite scaling differences between the two images.

[0112] At 708, the method 700 includes extracting feature points for each image in D 模型 and D 库 using a feature point detector. In some embodiments, the feature point detector can be part of the camera calibration system 402 in Figure 4 . In other embodiments, the feature point detector can be a separate system independent of the camera calibration system 402. In one embodiment, feature point detection is performed by a scale-invariant feature transform (SIFT) algorithm. The feature point detector can take an image as input and output, for each pixel location of the image, a probability that the pixel location is a feature point. The feature point detector can also output a description for each pixel location. The description can contain information about the visual characteristics around the feature point. The feature point detector can detect the same feature points in different images of the reference object, regardless of whether the images are acquired using constrained or unconstrained camera poses, and regardless of whether the coordinates of the same feature points change between the different images.

[0113] Briefly referring to Figure 13Image 1300 is shown, indicating multiple feature points detected by a feature point detector in a vehicle cab image, where the feature points are indicated as circles (e.g., such as first feature point 1302 and second feature point 1304). Figure 13 In the illustrated implementation, the SIFT algorithm is used to detect feature points. As shown in the figure, the SIFT algorithm extracts feature points from both the cab and the chessboard. However, due to the high contrast at the chessboard edges, the SIFT algorithm may be more efficient and / or accurate on the chessboard than on the cab. All other points in image 1300 not marked with circles can be considered background points.

[0114] Returning to method 700, at point 710, method 700 includes extracting D using a chessboard detection algorithm. 库 Feature points in each image. The chessboard detection algorithm may be an alternative feature point detector, capable of extracting D... 库 The 2D feature points of each image. In particular, a chessboard algorithm can be used to detect the pixel positions of the corner points.

[0115] At 712, method 700 includes D. 模型 For each pixel location in each image, a first set of positive (e.g., feature point) and negative (e.g., background) labels is generated to train a deep learning calibration model on a feature point detection task. The feature point detection task involves performing a binary classification on each pixel location in the image, classifying it as a feature point or a background point, and using the labeled image as ground truth data. For the purposes of this disclosure, a positive label is a first assignment value (e.g., 0) indicating that a feature point is contained at the pixel location, while a negative label is a second assignment value (e.g., 1) indicating that a background point is contained at the pixel location. The positive and negative labels can be generated using any one or both of a feature point detector and a checkerboard detection algorithm.

[0116] In one implementation, the feature point detector can output the probability that a pixel location is a feature point for each pixel location. Positive labels can be assigned to pixel locations in the image whose probability of being a feature point is higher than a threshold probability. Negative labels can be assigned to pixel locations whose probability of being a feature point is lower than a threshold probability. In other words, positive labels can be generated from a set of all pixel locations, and negative labels can be assigned to the remaining pixel locations. In some implementations, points on the boundary between feature points and background points can be further filtered.

[0117] At 714, method 700 includes D. 模型The image pairs in the dataset generate a second set of positive and negative labels for training the DL calibration model on a pixel location description task. The pixel location description task involves predicting a robust low-dimensional description for each feature point and background point. This description consists of an n-length vector with floating-point or integer values. Instructions stored in the DL calibration model system and executed by the processor can generate image pairs and can be used to... Figure 8 The method generates positive and negative labels for points in an image pair.

[0118] At 716, method 700 includes using data from D. 模型 The training pairs are used to jointly train the neural network on both detection and description tasks. The training pairs can be trained according to the above reference. Figure 5 The description is used to generate the second unconstrained image, where each training pair includes at least one pair of images and the corresponding ground truth information for each image in the pair. For constrained images, any two constrained images can be paired into an image pair. For unconstrained images, a homography matrix can be applied to the first image acquired using an unconstrained camera pose to randomly generate the second unconstrained image, where the mapping of the homography matrix allows points in the second unconstrained image to be associated with corresponding points in the first unconstrained image. Thus, the second unconstrained image can have the same content and objects as the first unconstrained image, but can differ from the first unconstrained image in terms of rotation and scaling. In some examples, random color transformations can also be performed to generate the second unconstrained image. Unconstrained image pairs can be randomly generated in real time during DL calibration model training. See below for reference. Figure 9 The joint training of the DL calibration model is described in more detail.

[0119] At 718, method 700 includes using a trained neural network to leverage data from D. 库 The image is used to construct a feature point library. See below for reference. Figure 10 The construction of the feature point library is described in more detail.

[0120] At 720, method 700 includes using a feature point library to perform online external calibration of the camera. See below for reference. Figure 11 The execution of online external calibration is described in more detail. Method 700 ends.

[0121] Figure 8 A method 800 for generating image pairs with feature point pairs and / or background point pairs for task description is shown, wherein the feature point pairs have positive and negative labels. Method 800 can be used as... Figure 5 This is implemented as part of the DL calibration model training system 500. In one implementation, one or more operations of method 800 can be stored in non-transitory memory and controlled by... Figure 4 The processor, non-transitory memory 406, and processor 404 of the camera calibration system 402 are executed.

[0122] At 802, the method 800 includes generating a set of constrained images D 模型 and D 库 with positive feature point pairs. For two different constrained images, a real-world object will be referenced at the same pixel location. Thus, any two constrained images can be paired as an image pair. In various embodiments, constrained images with different environmental settings can be selected as an image pair. For example, a first constrained image in an image pair can show a reference object (e.g., a checkerboard) in a real-world in a first lighting condition, while a second constrained image in the image pair can show the reference object in a second lighting condition. A feature point detector (e.g., SIFT) can not be able to identify the same feature point pair of the checkerboard in the same image, under different lighting conditions. By generating pairs with different lighting conditions, a trained DL calibration model can be able to identify the same feature point pair, regardless of the lighting condition difference in the image pair.

[0123] A positive feature point pair is a feature point of different images (e.g., an image pair) that corresponds to the same reference location in the real-world. For example, a first image of an image pair can include a checkerboard placed on a vehicle driver seat, and a second image of the image pair can also include the checkerboard. One corner of the checkerboard can be a feature point in both the first image and the second image. Thus, the feature point corresponding to the corner of the checkerboard in the first image and the feature point corresponding to the corner of the checkerboard in the second image can be a positive feature point pair. For images in D 模型 acquired using constrained camera poses, feature points at the same pixel location of two images can be applied as positive pairs. For images in D 库 , all feature points on the checkerboard can also be applied as positive pairs.

[0124] At 804, the method 800 includes generating positive feature point pairs in constrained and unconstrained images using data augmentation. Data augmentation can be particularly relevant to unconstrained images. Since unconstrained images do not share camera poses, real-world reference points can appear at different pixel locations in different unconstrained images. Thus, positive feature point pairs can not be established between two unconstrained images. To generate positive feature point pairs using data augmentation, it is necessary to mathematically create an image pair from a single unconstrained image, where a first set of feature points in the original unconstrained image corresponding to a reference location in the real-world can be mapped to a second set of feature points in the mathematically generated unconstrained image corresponding to the same reference location. To generate the mapping, a random homography matrix can be used to transform the original image. The feature points of the original image and the transformed image can then be applied as positive pairs.

[0125] At 806, the method 800 includes generating a set of unconstrained images D 库 and D 模型negative pairs are pairs of points in the images that do not point to the same reference location in the real world. For example, a feature point of a first image of a pair of images and a background point of a second image of the pair of images can be labeled as a negative pair (e.g., where the feature point and the background point are detected by a feature point detector, e.g., using SIFT). Or, a distance metric between SIFT descriptions of different points of the images in D 库 and D 模型 the distance between SIFT descriptions of different points of the images in D

[0126] As an example of how positive and negative labels are generated, for each iteration, the DL calibration model can be trained on n pairs of images. For each pair of images with the same constrained camera pose, feature points of a first image and feature points of a second image are extracted and sampled. The feature points from the first image and the second image can be concatenated to form a total number of feature points.

[0127] To generate pairs of images with unconstrained camera poses, after the feature points are extracted and sampled, a second image can be generated from the first image by applying a homography matrix to the first image. The second image can adjust the rotation, translation, and / or scaling of the first image. The feature point locations of the first image are projected onto the second image generated by the homography matrix. The feature points of the second image that are located at the image border can be filtered. The same feature points in the first image that are located at the image border are also filtered.

[0128] After the feature points at the image border are filtered, background points of the first image and background points of the second image can be extracted and sampled. The background points of the first image and the second image can be randomly sampled. In one embodiment, the ratio of background points to feature points can be a floating point number less than or equal to two. The feature points of the second image are projected onto the first image by applying the inverse homography matrix.

[0129] In this way, training labels can be generated for pairs of images acquired using constrained camera poses and unconstrained camera poses. Using operations such as “blur”, “random contrast”, and “random brightness”, the input images can be transformed in color space and the images are input to the DL calibration model during training. Applying these transformations to the images does not change the locations of the feature points.

[0130] During training, the locations of feature points in the first image are predicted as the first output, and a first set of descriptions is predicted as the second output by the DL calibration model. The first set of descriptions can be considered as anchor data. The locations of feature points in the second image are also predicted in the first output, and a second set of descriptions is predicted in the second output by the DL calibration model. The descriptions in the second set of descriptions can be associated with the corresponding descriptions in the first set of descriptions as positive pairs. The locations of background points in the first image and the locations of feature points with descriptions outside the anchor data threshold distance can be randomly sampled and assigned as negative pairs.

[0131] refer to Figure 9 An exemplary implementation is shown for training a DL calibration model (e.g. Figure 5 The flowchart of method 900 (DL calibration model 524) of external DL calibration training system 500. The DL calibration model can be a deep neural network with multiple convolutional layers, for example... Figure 6 The CNN 600 in the example. Method 900 can be implemented as part of an external DL calibration training system 500. One or more operations of method 900 can be stored in non-transitory memory and executed by the processor of the camera calibration system, for example... Figure 4 The camera calibration system 402 in the middle has a non-transitory memory 406 and a processor 404.

[0132] Method 900 begins with 902, where method 900 includes receiving training pairs from the training dataset, such as those referenced above. Figure 5 The training pair 520 is described. Each training pair may include one or more image pairs and corresponding ground truth data. In one implementation, the training set may be stored in the training module of the camera calibration system, such as training module 410.

[0133] At 904, method 900 includes inputting the input images of the training pairs (e.g., pixel intensity values ​​of each image in one or more image pairs) into the input layer of the DL calibration model. Each pixel intensity value of the input image may be input into a different neuron in the input layer of the DL calibration model.

[0134] At 906, the method 900 includes receiving an output of the DL calibration model. The image data of the input image propagates through the DL calibration model from the input layer, through one or more convolutional layers, until reaching a first output layer corresponding to the detection task of the DL calibration model, and a second output layer corresponding to the description task of the DL calibration model. As described above, the first output of the first output layer corresponding to the detection task can be a first 3D data structure having a width, a height, and a depth. The width and the height of the first 3D structure of the detection task can be equal to the width and the height of the input image, respectively. The depth of the first 3D structure of the detection task can include a number of layers depending on the number of image pairs included in the input of the DL calibration model.

[0135] The second output of the second output layer corresponding to the description task can be a second 3D data structure having a width, a height, and a depth. The width and the height of the second 3D structure can be equal to the width and the height of the input image, respectively. The depth of the second 3D structure can correspond to an array of values, where the array of values can be a predicted description of a point (e.g., a feature point or a background point) at a corresponding pixel position.

[0136] At 908, the method 900 includes determining and backpropagating a first loss through the plurality of convolutional layers (e.g., the first decoder 606a) of the detection task portion of the DL calibration model based on a difference between the first output of the DL calibration model and ground truth data of the training image pair. The first loss can be backpropagated according to a first loss function. For example, the first loss function can be a Softmax negative log likelihood loss function. Figure 6

[0137]

[0138] wherein, represents a current class of the point (e.g., a feature point or a background point), which can take a value of 1 or 2; represents a number of all classes of the point, which is 2 in this example; represents a probability that the point belongs to the current class; represents a true probability that the point belongs to the current class (e.g., supervised information generated using a SIFT algorithm). In other embodiments, different loss functions can be used. For example, the first loss function can be a Sigmoid cross-entropy loss function.

[0139] The backpropagation of the loss can be performed according to a gradient descent algorithm, in which a gradient (a first derivative or an approximation of a first derivative) of the loss function is determined for each weight and bias of the DL calibration model. Then, each weight (and bias) of the DL calibration model is updated by adding a negative value of a product of the determined (or approximated) gradient for the weight (or bias) and a predetermined step size.

[0140] ​At 910, the method 900 includes determining and backpropagating a second loss through the plurality of convolutional layers of the description task portion of the DL calibration model (e.g., the second decoder 606b). The second loss can be backpropagated according to a second loss function. The second loss can not be based on ground truth data input to the DL calibration model. For example, the second loss function can be a triplet loss function, as described below: Figure 6

[0141]

[0142]

[0143] is anchor data, i.e., a description of the first feature point (a vector of floating point data type);

[0144] is positive data, i.e., a description of the second feature point;

[0145] is negative data, i.e., a description of the background point;

[0146] is a predefined constant greater than 0; and

[0147] is a distance, i.e., a Euclidean distance.

[0148] The triplet loss function can compute distances between positive pairs and negative pairs, which can then be pushed / pulled according to the loss value. In other words, distances between positive pairs (e.g., two feature points from two images that reference the same point in the real world) will be minimized, while distances between negative pairs (e.g., a feature point and a background point from one image or pair of images) will be maximized. As a result, the weights and biases of the DL calibration model can converge to values that output accurate predicted descriptions generated for each pixel location of each input image. In other words, during training, the description loss forces the descriptions of all points to change in an attempt to reach an equilibrium in which distances between positive pairs are small, while distances between negative pairs are large. It should be appreciated that, in some cases, the descriptions can change in a non-converging manner, resulting in a failed training.

[0149] At 912, the method 900 includes computing and backpropagating a final loss through the shared portion of the DL calibration model (e.g., the encoder 604a) and adjusting the weights and biases of the shared portion of the DL calibration model. The final loss may be a sum of a loss (e.g., a cross-entropy loss) for the detection task and a triplet loss for the description task based on both the first and second outputs, as described below:

[0150] ​​

[0151] wherein are balancing factors for the two tasks (e.g., the detection task and the description task).

[0152] The method 900 can then end. It is noted that the method 900 can be repeated until the weights and biases of the DL calibration model converge, or the rate of change of the weights and / or biases of the deep neural network used for each iteration of the method 900 is below a threshold.

[0153] Although not described in the method 900, it is understood that to avoid overfitting, the training of the DL calibration model can be interrupted periodically to verify the performance of the DL calibration model on a set of test pairs (e.g., the test pairs 522). The test pairs can be generated in the manner described in the method 500 of Figure 5 , and can be randomly sampled from a larger training dataset. In one embodiment, the training of the DL calibration model can end when the performance of the DL calibration model on the test pairs converges (e.g., when the error rate on the test set converges to a minimum).

[0154] Figure 10 A flowchart illustrating a method 1000 for deploying a trained DL calibration model (e.g., the trained DL calibration model 530 of Figure 5 and / or the CNN 600 of Figure 6 to construct a feature point library, in accordance with example embodiments, is shown. In various embodiments, the feature point library can be a reference table or lookup table stored in a memory of a camera calibration system, such as the non-transitory memory 406 of the camera calibration system 402 of Figure 4 . The feature point library can include a plurality of 2D feature points and descriptions generated using the trained DL calibration model. In one embodiment, one or more operations of the method 1000 can be stored in a non-transitory memory and executed by a processor of the camera calibration system, such as the processor 404. The trained DL calibration model can be stored in an inference module (e.g., the inference module 412) of the camera calibration system. The feature point library produced by the method 1000 can be deployed in a vehicle, where it can be used for online external calibration of cameras (e.g., the cameras 118 and / or the cameras 119) of the vehicle, as described below with reference to Figure 11 and 12 .

[0155] At 1002, the method 1000 includes correcting a dataset D 库camera distortion of the image in the original image. The camera distortion can include radial distortion and tangential distortion. The radial distortion can be caused by camera lens curvature defects. The tangential distortion can be caused by the non-parallelism of the camera lens and the image plane when the image is acquired. The camera distortion can be corrected using the intrinsic matrix of the camera, which includes an intrinsic matrix K raw and distortion coefficients d raw The calibrated camera intrinsic parameters can be determined by the manufacturer or the pre-intrinsic calibration of the camera. In this way, a new undistorted image can be generated from the original distorted image.

[0156] In particular, to correct the distortion, a new intrinsic matrix with calibrated camera intrinsic parameters and without distortion coefficients is used to generate a new image. Each pixel of the new image can be described by the following equation (4):

[0157]

[0158] which can be matched with the corresponding pixel in the original image, can be described by the following equation (5):

[0159]

[0160] Each point in the new image coordinate system of the camera is converted into the sensor coordinate system of the camera according to the following equation:

[0161]

[0162] wherein

[0163] is the 2D camera pixel, is the undistorted 2D camera pixel, is the new intrinsic matrix, is the new 2D camera pixel. The 2D distorted pixel, 原始 is calculated using the distortion model and distortion coefficients d .

[0164] Each point in the sensor coordinate system can then be converted into the original image coordinate system of the camera using the following equation:

[0165]

[0166] wherein, is the 2D original image pixel, is the original intrinsic matrix, is the 2D distorted pixel. Using the one-to-one pixel relationship, an undistorted camera image can be generated.

[0167] At 1004, the method 1000 includes detecting 2D feature points in the images D 库 and predicting corresponding descriptions of the image pixels using the trained DL calibration model. The images from D 库 may be input into the trained DL calibration model, and the DL calibration model can output, for each pixel, a first probability that the pixel contains a feature point and a second probability that the pixel contains a background point as a first output (e.g., by the decoder 606a of the CNN 600). Thus, the feature points of each image can be detected by the trained DL calibration model. The DL calibration model can also output a corresponding description of each detected 2D feature point as a second output (e.g., by the decoder 606b of the CNN 600).

[0168] At 1006, the method 1000 includes determining a relative camera pose between each of a pair of images. In one embodiment, a checkerboard detection algorithm can be used to detect the corner points of a checkerboard contained in the images D 库 . Based on the detected checkerboard corner points, the camera pose between the camera and the checkerboard in each image can be estimated. Since the physical size of the checkerboard is known, the real-world scale of the images can be determined. The detected checkerboard corner points in two images can be used to determine the relative camera pose between the two images.

[0169] Referring briefly Figure 14 to FIG. 14, an exemplary checkerboard 1400 is shown in a 3D (e.g., x / y / z) coordinate system, where the side length of any square on the checkerboard 1400 is . The 3D coordinates of the points on the checkerboard 1400 can be calculated directly and represented as .

[0170] Based on the checkerboard detection algorithm, the 2D coordinates of the checkerboard points on an image can be detected . The 2D coordinates can be converted to the coordinate system of the camera (e.g., the camera 118 of the device 100) using the following equations: Figure 1

[0171]

[0172] Pose estimation (PnP) on this image against the checkerboard coordinate system can be implemented as follows:

[0173]

[0174] Similarly to the above steps, given a new image, the following equations 11, 12, and 13 can also be used to obtain : ​

[0175]

[0176] Thus, the relative camera pose is given by Equation 14:

[0177]

[0178] Returning to Figure 10 At 1008, the method 1000 includes matching 2D feature points of each pair of images with corresponding descriptors. For each pair of images in the set of pairs of images, pairs of feature points are matched based on similarity of the feature point descriptors (e.g., the distance between the descriptors of the respective feature points is within a distance threshold). The distance can be the Euclidean distance between two numerical vectors containing the two different descriptors of the two matching feature points. 库

[0179] At 1010, the method 1000 includes performing triangulation of the matching feature points for the pair of images to determine respective 3D feature points and corresponding descriptors. Triangulation is performed using a pair of matching feature points and the known camera position to determine a 3D feature point that is closest to a 3D ray (originating from the camera center in a particular direction) that corresponds to the matching 2D feature point observed by the camera.

[0180] In particular, the triangulation of the matching 2D feature points in conjunction with the known camera position can be achieved by projecting a 3D point in a world coordinate system onto the first image using the matrix of the first image and using the first camera pose of the first image as the world coordinate system. The projection of a 3D point in a world coordinate system can be performed by the following equation:

[0181]

[0182] where K is the camera intrinsic matrix (e.g., a 3x3 matrix) of the first image that includes a set of camera intrinsic parameters, R is a rotation matrix (e.g., a 3x3 matrix), and t is a translation vector (e.g., a 3x1). When the first camera pose is used as the world coordinate system, Equation 15 can be adjusted as:

[0183]

[0184] The matrix of the second image can project a 3D point in a world coordinate system onto the first image using Equation 15. When the first camera pose of the first image is used as the world coordinate system, the matrix of the second image (e.g., a 3x4 matrix) can be related to the relative camera pose according to the following equation:

[0185]

[0186] where K is the intrinsic matrix with camera intrinsic parameters, R​相对 is a relative rotation matrix (e.g., a 3x3 matrix), t 相对 is a relative translation vector (e.g., a 3x1).

[0187] Thus, equation 17 can be used to match a feature point in the second image with a feature point in the first image and determine a corresponding 3D point and description based on the first camera coordinate system.

[0188] Referring now to Figure 11 , a flowchart of a method 1100 of using a trained DL calibration model (e.g., the DL calibration model 530 of Figure 2 in a computing system (e.g., the computing system 200 of Figure 5 ) of a vehicle in accordance with example embodiments is shown. The method 1100 can be performed by a processor of a computing system, such as the processor 204 in Figure 2 . The operations of the method 1100 can be stored in a non-transitory memory of the computing system (e.g., in the image analysis module 210).

[0189] At 1102, the method 1100 includes receiving a plurality of images from a camera of a vehicle, which can be the camera 218 in Figure 2 . In some embodiments, the plurality of images can be acquired in real-time or near real-time by a DMS and / or OMS system of the vehicle, which can be the DMS 147 and / or OMS 148 in Figure 1 .

[0190] At 1104, the method 1100 includes detecting and describing a plurality of feature points in the images using the trained DL calibration model. The received plurality of images can be input into the trained DL calibration model, and the DL calibration model can output 2D feature points and generate feature point descriptions for the 2D feature points.

[0191] At 1106, the method 1100 includes matching the 2D feature points detected from the images with 3D feature points of a feature point library. By matching the 2D feature point descriptions of the images with the 3D feature point descriptions in the feature point library, the 2D feature points of the images can be mapped to corresponding 3D feature points in the feature point library. In this way, the corresponding 3D feature points in the feature point library can be determined based on the 2D feature points.

[0192] At 1108, method 1100 includes determining a set of extrinsic parameters of the camera based on mapped 3D feature points. A one-to-one correspondence point set is established for the 2D and 3D feature points. Combining the one-to-one correspondence point sets in 2D and 3D, the extrinsic parameters of the cabin camera can be solved using the Perspective-n-Point (PnP) algorithm. Examples of available PnP algorithms include Efficient Perspective-n-Point (EPnP) and Levenberg-Marquardt optimization.

[0193] Figure 12 An exemplary method 1200 is shown for determining whether to adjust vehicle operation based on the output of a trained neural network model for vehicle-based ADAS. In at least one embodiment, the trained neural network model can use the output of an in-cabin camera (e.g., ...) to... Figure 1 The ADAS takes one or more images of the vehicle driver as input from an in-cabin camera (118), which may be part of the vehicle's DMS system (e.g., DMS 147). In the described implementation, the trained neural network model can output the probability that the driver's alertness level is low enough to warrant ADAS intervention. In other implementations, the trained neural network model, or different trained neural network models of the ADAS, can predict different emotional states of the driver. For example, the driver may be anxious, angry, etc. Based on the output, the ADAS may adjust the vehicle's operation, notify the driver via sound or message, or perform different actions of the vehicle.

[0194] Before inputting one or more images into a trained neural network model, the in-cabin camera can be calibrated. Calibrating the in-cabin camera can improve the performance of the trained neural network model, thereby enabling more timely and accurate ADAS interventions. The in-cabin camera can be calibrated using a DL calibration model trained according to the methods described herein. Instructions for implementing method 1200 can be stored in non-transitory memory and controlled by the vehicle computing system (such as…). Figure 1 The computing system 120) and / or the vehicle's ADAS (such as Figure 1 The processor of ADAS 149 is used to execute it.

[0195] Method 1200 begins at 1202, where the method includes estimating and / or measuring vehicle operating conditions. Vehicle operating conditions can be estimated based on one or more outputs of various sensors of the vehicle, such as oil temperature sensors, engine or wheel speed sensors, torque sensors, etc. Vehicle operating conditions can include engine speed and load, vehicle speed, transmission oil temperature, exhaust gas flow, air mass flow, coolant temperature, coolant flow, engine oil pressure (e.g., oil gallery pressure), mode of operation of one or more intake and / or exhaust valves, electric motor speed, battery charge level, engine torque output, wheel torque, etc. Estimating and / or measuring vehicle operating conditions can include determining whether the vehicle is being driven by the engine or the electric motor.

[0196] Estimating and measuring vehicle operating conditions can also include determining proximity of other vehicles in the vehicle’s lane or other lanes to the vehicle by one or more cameras and / or sensors positioned on or in the vehicle, such as camera 128 of the vehicle 100. Figure 1 Vehicle operating conditions can include road conditions (e.g., whether the road is paved or unpaved, wet, icy, etc.), weather conditions (e.g., whether it is raining or snowing), and / or other external conditions of the vehicle (e.g., whether it is daytime or nighttime, etc.).

[0197] At 1204, the method includes receiving image data of a driver of the vehicle from an in-cabin camera and / or DMS of the vehicle. For example, the in-cabin camera can capture images of the driver as the driver looks at a forward road segment, including traffic and / or other vehicles ahead of the vehicle. The DMS can capture images of the driver looking out of one or more side windows and / or looking at a rearview mirror of the vehicle. The DMS can capture images of the driver interacting with other occupants of the vehicle.

[0198] At 1205, the method includes performing extrinsic calibration of the in-cabin camera using the trained DL calibration model. During extrinsic calibration, a set of extrinsic parameters of the camera can be estimated using images acquired by the in-cabin camera, as described above with reference to Figure 11 Extrinsic calibration can be performed using the following flow: detecting 2D feature points and generating corresponding descriptions using the trained DL calibration model, feature matching the 2D feature points and descriptions of the image with 2D feature points and descriptions in a library to identify corresponding 3D feature points and descriptions of the image, and determining extrinsic parameters using a PnP algorithm based on camera intrinsic parameters, 2D feature points and descriptions, and 3D feature points and descriptions. The extrinsic parameters of the camera can be used to perform extrinsic calibration.

[0199] At 1206, the method includes processing the image data to determine whether to adjust operation of the vehicle based on the image data. In the depicted embodiment, the ADAS can adjust operation of the vehicle in response to a predicted alertness level of the driver falling below a threshold alertness level.

[0200] At 1208, processing the image data to determine whether to adjust operation of the vehicle includes using the trained neural network to predict a level of alertness of the driver. For example, the level of alertness of the driver can be predicted based on facial expressions of the driver, signs of fatigue in the driver’s posture such as slouching or hunching over, a period of time in which the driver is not moving, detected movement or slippage of the driver’s eyelids, etc. The predicted alertness can be a value between 0.0 and 1.0, where 1.0 indicates a highest level of alertness and 0.0 indicates a lowest level of alertness.

[0201] In other embodiments, the ADAS can adjust operation of the vehicle in response to different predicted states of the driver, identification of the driver, or different characteristics of the driver’s behavior. It will be appreciated that the examples and embodiments described herein are for illustrative purposes only, and that the trained neural network model can be used by the ADAS or DMS to adjust operation of the vehicle in other ways without departing from the scope of the present disclosure.

[0202] At 1210, the method includes determining whether the predicted level of alertness output by the trained neural network model is below a threshold level of alertness. For example, the threshold level of alertness can be 0.5, and if the predicted level of alertness is 0.6, then it is determined that the level of alertness of the driver is above the threshold level of alertness (e.g., the alertness is sufficient to enable driving without ADAS intervention). Or, if the predicted level of alertness is 0.4, then it is determined that the level of alertness of the driver is below the threshold level of alertness (e.g., the alertness is not sufficient to enable driving without ADAS intervention).

[0203] If it is determined at 1210 that the predicted level of alertness output by the trained neural network model is not below the threshold level of alertness, then the method proceeds to 1212. At 1212, the method includes continuing to operate the vehicle without ADAS system intervention, and the method ends. Or, if it is determined at 1210 that the predicted level of alertness is below the threshold level of alertness, then the method proceeds to 1214.

[0204] At 1214, the method includes adjusting operation of the vehicle based on the predicted level of alertness. In other words, the ADAS can perform one or more actions to increase the level of alertness of the driver. For example, the ADAS can notify the driver through an audible sound, alerting the driver that they appear to be fatigued, or the ADAS can play an audio file with music or other content that can increase the level of alertness of the driver. In some embodiments, the ADAS can adjust acceleration or speed of the vehicle, or perform different actions or operations of the vehicle in response to the level of alertness being below the threshold level of alertness.

[0205] By calibrating the in-cabin camera before using the trained neural network model to predict the alertness level of the driver, the ability of the ADAS to determine whether intervention is needed can be improved. Specifically, in-cabin camera calibration can improve the accuracy of the mathematical relationship between the feature points detected in the images acquired by the in-cabin camera, which can more accurately predict the driver’s posture and facial features.

[0206] Accordingly, a vehicle camera calibration system based on a trained DL calibration model (which can be a CNN) is proposed. The camera calibration system can be used in conjunction with the ADAS of a vehicle to improve the performance of the ADAS.

[0207] In one example of implementing the advantages provided by the proposed camera calibration system, a first vehicle can include a first ADAS but can not include the proposed camera calibration system. The controller of the first vehicle can receive images with varying degrees of illumination, brightness, and / or contrast acquired by the DMS of the first vehicle from the in-cabin camera of the first vehicle. Since the neural network model of the ADAS is not trained using images with varying degrees of illumination, brightness, and / or contrast, the first ADAS can not be as accurate in identifying feature points of the driver images. The decrease in feature point detection accuracy can result in a decrease in the performance of the neural network model of the ADAS, which can not accurately predict the alertness level of the driver of the first vehicle. Since the alertness level of the driver of the first vehicle cannot be accurately predicted, the ADAS can not be able to intervene in time to improve the alertness level of the driver.

[0208] In contrast, a second vehicle can include a second ADAS and include the proposed camera calibration system. The controller of the second vehicle can receive images with varying degrees of illumination, brightness, and / or contrast acquired by the DMS of the first vehicle from the in-cabin camera of the second vehicle. The controller can use the trained DL calibration model of the proposed camera calibration system to externally calibrate the in-cabin camera of the second vehicle. After the external calibration, the second ADAS can more accurately identify feature points of the driver images due to the performance of such external calibration. The increase in feature point detection accuracy can result in an increase in the performance of the neural network model of the ADAS, which can more accurately predict the alertness level of the driver of the second vehicle (as compared to the first ADAS). Since the alertness level of the driver of the second vehicle is accurately predicted, the ADAS can intervene in time to improve the alertness level of the driver. For example, the ADAS can play music through the speakers of the vehicle, or lower the temperature of the driver’s cabin of the second vehicle, or perform different actions. In this way, the quality of operation of the vehicle by the second driver and the driving performance of the second driver can be improved.

[0209] The present disclosure also provides support for a method for performing extrinsic calibration of a vehicle camera, the method comprising: receiving a plurality of images from the camera of the vehicle, detecting and describing a plurality of two-dimensional (2D) feature points of the images using a trained neural network model, mapping the detected 2D feature points to three-dimensional (3D) feature points stored in a feature point library generated using the trained neural network model, estimating a set of extrinsic parameters of the camera using the mapped 3D feature points, performing extrinsic calibration based on the set of extrinsic parameters. In a first example of the method, a high-level driver assistance system (ADAS) of the vehicle uses images acquired with the camera after the camera is calibrated. In a second example of the method, the plurality of images comprises a first image dataset of a reference object acquired using a constrained camera pose or an unconstrained camera pose of the camera, and a second image dataset of the reference object acquired using an unconstrained camera pose of the camera. In a third example of the method, optionally including one or both of the first and second examples, the reference object is a chessboard. In a fourth example of the method, optionally including one or more or each of the first through third examples, the neural network model has an encoder / decoder architecture comprising: a first decoder portion trained to detect 2D feature points in a pair of images of the plurality of images; a second decoder portion trained to predict descriptions of the detected 2D feature points; and a third shared encoder portion trained to detect the 2D feature points and predict the descriptions of the detected 2D feature points; wherein the neural network model is jointly trained to detect the 2D feature points and predict the descriptions of the detected 2D feature points. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, jointly training the neural network model to detect the 2D feature points in a pair of images and predict descriptions of the detected 2D feature points further comprises: using a feature point detector to label pixel locations of each image of the pair of images as either a feature point or a background point, and to generate a description of each pixel location; backpropagating a loss from a first output of the first decoder portion to adjust weights and biases of convolutional layers of the first decoder portion of the neural network model according to a first loss function using the labeled pixel locations as ground truth data; backpropagating a loss from a second output of the second decoder portion to adjust weights and biases of convolutional layers of the second decoder portion of the neural network model according to a second loss function; and backpropagating a loss from the first and second outputs and adjusting weights and biases of convolutional layers of the third shared encoder portion of the neural network model according to both the first and second loss functions.In a sixth example of the method optionally including one or more or each of the first through fifth examples, the first loss function is a sigmoid cross-entropy loss function and the second loss function is a triple loss function based on a negative log-likelihood loss, and adjusting the weights and biases of the convolutional layers of the second decoder portion according to the second loss function further includes: calculating a first distance between a first feature point of a first image of the pair of images and a second matching feature point of a second image of the pair of images; calculating a second distance between a first background point or feature point of the first image and a second background point or feature point of the second image, the first and second background points or feature points being a distance greater than a threshold distance apart; minimizing the first distance and maximizing the second distance using the triple loss function. In a seventh example of the method optionally including one or more or each of the first through sixth examples, the feature point detector is one of a scale-invariant feature transform (SIFT) algorithm and a checkerboard detection algorithm. In an eighth example of the method optionally including one or more or each of the first through seventh examples, each description is a vector of length n whose floating point or integer values represent visual characteristics around a respective feature point to distinguish the respective feature point from different feature points. In a ninth example of the method optionally including one or more or each of the first through eighth examples, marking pixel locations of the image as feature points or background points using the feature point detector further includes: in response to a probability that the pixel location output by the feature point detector is a feature point exceeding a threshold probability, marking a pixel location as a feature point, and in response to the probability not exceeding the threshold probability, marking the pixel location as a background point. In a tenth example of the method optionally including one or more or each of the first through ninth examples, the pair of images includes two images taken using the same constrained camera pose. In an eleventh example of the method optionally including one or more or each of the first through tenth examples, the pair of images includes a first image taken using an unconstrained camera pose and a second image generated using a homography. In a twelfth example of the method optionally including one or more or each of the first through eleventh examples, generating the feature point library further includes: using the trained DL calibration model to predict descriptions of 2D feature points detected in a pair of images of the second dataset, the 2D feature points being detected by the trained DL calibration model; determining a relative camera pose between the two images of the pair of images; matching 2D feature points and corresponding descriptions of the 2D feature points for the pair of images; triangulating the matching feature points of the pair of images to establish respective 3D feature points including the corresponding descriptions; and storing the 3D feature points and the corresponding descriptions in the feature point library.In a thirteenth example of the method optionally including one or more or each of the first through twelfth examples, mapping the detected 2D feature points to 3D feature points stored in the feature point library further includes mapping a first detected 2D feature point in an image to a 3D feature point stored in the feature point library using a first description of the first detected 2D feature point, wherein the 3D feature point includes a second description of the 3D feature point that is within a threshold distance of the first description. In a fourteenth example of the method optionally including one or more or each of the first through thirteenth examples, estimating the set of extrinsic parameters of the camera using the mapped 3D feature points further includes detecting 2D feature points in pairs of images and predicting corresponding descriptions of the detected 2D feature points using the trained DL calibration model; based on the descriptions, identifying 3D feature points in the feature point library that correspond to the 2D feature points; and based on camera intrinsic parameters, the 2D feature points and descriptions, and the 3D feature points and descriptions, determining extrinsic parameters of the camera using a camera pose estimation algorithm.

[0210] The present disclosure also provides support for an external camera calibration system for a vehicle camera, the system comprising: a processor; and a non-transitory memory comprising instructions that, when executed, cause the processor to: receive a first plurality of images acquired using the camera; train a neural network model to detect two-dimensional (2D) feature points and generate a description of the detected 2D feature points; construct a feature point library using the trained neural network model, the feature point library comprising three-dimensional (3D) feature points having similar descriptions to the detected 2D feature points; and deploy the trained neural network model and the feature point library on a driver monitoring system (DMS) of the vehicle, the DMS configured to: receive a plurality of images of a driver of the vehicle acquired using the camera; detect 2D feature points in the images of the driver and predict a description of the detected 2D feature points using the trained neural network model; map the detected 2D feature points to corresponding 3D feature points of the feature point library; estimate a set of extrinsic parameters of the camera based on the 3D feature points; and externally calibrate the camera using the estimated extrinsic parameters. In a first example of the system, the neural network model has an encoder / decoder architecture comprising: a first decoder portion trained to detect 2D feature points in a pair of images of the plurality of images; a second decoder portion trained to predict a description of the detected 2D feature points; and a third shared encoder portion trained to detect the 2D feature points and predict the description of the detected 2D feature points; wherein the neural network model is jointly trained to detect the 2D feature points and predict the description of the detected 2D feature points. In a second example of the system optionally including the first example, training the neural network model to detect the 2D feature points and generate a description of the detected 2D feature points further comprises: using a feature point detector to label pixel locations of each image of the pair of images as either a feature point or a background point and generate a description of each pixel location; backpropagating a first loss from a first output of the first decoder portion to adjust weights and biases of convolutional layers of the first decoder portion of the neural network model according to a first loss function using the labeled pixel locations as ground truth data; backpropagating a second loss from a second output of the second decoder portion to adjust weights and biases of convolutional layers of the second decoder portion of the neural network model according to a second loss function; and backpropagating a third loss from the first and second outputs and adjusting weights and biases of convolutional layers of the third shared encoder portion of the neural network model according to both the first and second loss functions.In a third example of the system, optionally including one or both of the first and second examples, the first loss function is a sigmoid cross-entropy loss function, and the second loss function is a triplet loss function based on a negative log-likelihood loss, and adjusting the weights and biases of the convolutional layers of the second decoder portion according to the second loss function further comprises: computing a first distance between a first feature point of a first image of the pair of images and a second matching feature point of a second image of the pair of images; computing a second distance between a first background point or feature point in the first image and a second background point or feature point in the second image, the first and second background points or feature points being at a distance greater than a threshold distance; minimizing the first distance and maximizing the second distance using the triplet loss function.

[0211] The present disclosure also provides support for a method comprising: estimating a set of extrinsic parameters of a camera of a vehicle based on determining three-dimensional (3D) feature points of a driver of the vehicle, the 3D feature points corresponding to two-dimensional (2D) feature points of an image of the driver acquired by the camera, the 2D feature points detected by a trained neural network model, wherein the neural network model is jointly trained to detect 2D feature points in images acquired by the camera and to predict a description of the 2D feature points, and map the 2D feature points to the 3D feature points based on the predicted description, and performing extrinsic calibration of the camera using the set of extrinsic parameters.

[0212] The description of the various embodiments described herein has been presented for purposes of illustration but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technology found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0213] It is expected that during the life of this patent many relevant systems, methods and computer programs will be developed and the scope of the terms ML model, DL calibration model, neural network and vehicle operation data is intended to include all such new technologies a priori.

[0214] The terms "comprises", "comprising", "includes", "including", "has", "having", and their conjugates, mean "including but not limited to". This term encompasses the terms "consisting of" and "consisting essentially of".

[0215] As used herein, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" can include a plurality of compounds, including mixtures thereof.

[0216] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0217] The word "optionally" is used herein to mean "may or can not be present." Any particular implementation of the embodiments described herein can include a plurality of the "optional" features, unless such features conflict.

[0218] It should be appreciated that certain features of the embodiments described herein, which are, for clarity, described in the context of separate embodiments, can also be provided in combination in a single embodiment. Conversely, various features of the embodiments described herein, which are, for brevity, described in the context of a single embodiment, can also be provided separately or in any suitable sub-combination or as suitable in any other embodiment of the application described herein. Certain features described in the context of various embodiments are not to be interpreted as essential features of those embodiments, unless explicitly stated otherwise.

[0219] Although the application has been described in connection with specific embodiments thereof, it will be readily appreciated by those skilled in the art that numerous alternatives, modifications and adaptations thereof will be apparent. Therefore, it is intended that the application not be limited to the

[0220] It is the applicant's intent that all publications, patents and / or patent applications mentioned in this specification are to be incorporated by reference in their entirety into the specification for information presented in this publication, patent and / or patent application to the extent permitted by applicable law. In case of any conflict in the context of the specification, the present specification shall control. Additionally, any reference to a prior art publication, patent or patent application can not be construed as an admission that such publication, patent or patent application is prior art with respect to the present application. In the event of any conflict in the context of the present specification, the present specification shall prevail. Furthermore, the section headings provided can not be construed as necessarily limiting the scope of the application. Additionally, any priority document(s) are hereby incorporated by reference in their entirety in the present application.

Claims

1. A method for extrinsic calibration of a vehicle camera, the method comprising: receiving a plurality of images from the camera of the vehicle; detecting and describing a plurality of two-dimensional (2D) feature points of the images using a trained neural network model; mapping the detected 2D feature points to three-dimensional (3D) feature points stored in a feature point library generated using the trained neural network model; estimating a set of extrinsic parameters of the camera using the mapped 3D feature points; performing extrinsic calibration based on the set of extrinsic parameters.

2. The method of claim 1, wherein a high-level driver assistance system (ADAS) of the vehicle uses images acquired with the camera after the camera is calibrated.

3. The method of claim 1, wherein the plurality of images comprises a first image dataset of a reference object acquired using a constrained camera pose or an unconstrained camera pose of the camera; and a second image dataset of the reference object acquired using an unconstrained camera pose of the camera.

4. The method of claim 3, wherein the reference object is a chessboard.

5. The method of claim 1, wherein the neural network model has an encoder / decoder architecture comprising: a first decoder portion trained to detect 2D feature points in a pair of images of the plurality of images; a second decoder portion trained to predict a description of the detected 2D feature points; and a third shared encoder portion trained to detect the 2D feature points and predict the description of the detected 2D feature points; wherein the neural network model is jointly trained to detect the 2D feature points and predict the description of the detected 2D feature points.

6. The method of claim 5, wherein jointly training the neural network model to detect the 2D feature points in a pair of images and predict a description of the detected 2D feature points further comprises: labeling pixel locations of each image of the pair of images as feature points or background points using a feature point detector and generating a description of each pixel location; backpropagating a loss from a first output of the first decoder portion to adjust weights and biases of convolutional layers of the first decoder portion of the neural network model according to a first loss function using the labeled pixel locations as ground truth data; backpropagating a loss from a second output of the second decoder portion to adjust weights and biases of convolutional layers of the second decoder portion of the neural network model according to a second loss function; and backpropagating a loss from the first and second outputs and adjusting weights and biases of convolutional layers of the third shared encoder portion of the neural network model according to both the first and second loss functions.

7. The method of claim 6, wherein the first loss function is one of a sigmoid cross-entropy loss function and a Softmax negative log-likelihood loss function, and the second loss function is a triad loss function based on a negative log-likelihood loss, and adjusting the weights and biases of the convolutional layers of the second decoder portion according to the second loss function further comprises: computing a first distance between a first feature point of a first image of the pair of images and a second matching feature point of a second image of the pair of images; computing a second distance between a first background point or feature point of the first image and a second background point or feature point of the second image, the first and second background points or feature points being more than a threshold distance apart; minimizing the first distance and maximizing the second distance using the triad loss function.

8. The method of claim 6, wherein the feature point detector is one of a scale-invariant feature transform (SIFT) algorithm and a chessboard detection algorithm.

9. The method of claim 6, wherein each description is a length-n vector whose floating point or integer values represent visual characteristics around a respective feature point to distinguish the respective feature point from different feature points.

10. The method of claim 6, wherein using the feature point detector to label pixel locations of the images as feature points or background points further comprises: labeling a pixel location as a feature point in response to a probability that the pixel location is a feature point output by the feature point detector exceeding a threshold probability, and labeling the pixel location as a background point in response to the probability not exceeding the threshold probability.

11. The method of claim 6, wherein the pair of images comprises two images taken using the same constrained camera pose.

12. The method of claim 6, wherein the pair of images comprises a first image taken using an unconstrained camera pose and a second image generated using a homography.

13. The method of claim 3, wherein generating the feature point library further comprises: using the trained DL calibration model to predict descriptions of 2D feature points detected in a pair of images of the second dataset, the 2D feature points being detected by the trained DL calibration model; determining a relative camera pose between the two images of the pair of images; matching 2D feature points and corresponding descriptions of the 2D feature points for the pair of images; triangulating the matching feature points of the pair of images to establish respective 3D feature points including the corresponding descriptions; and storing the 3D feature points and the corresponding descriptions in the feature point library.

14. The method of claim 1, wherein mapping the detected 2D feature points to 3D feature points stored in the feature point library further comprises: mapping the detected first 2D feature points to 3D feature points stored in the feature point library using first descriptions of the first 2D feature points detected in the images, wherein the 3D feature points comprise second descriptions of the 3D feature points that are within a threshold distance of the first descriptions.

15. The method of claim 1, wherein estimating the set of extrinsic parameters of the camera using the mapped 3D feature points further comprises: detecting 2D feature points in pairs of images using the trained DL calibration model and predicting corresponding descriptions of the detected 2D feature points; based on the descriptions, identifying 3D feature points in the feature point library that correspond to the 2D feature points; and determining extrinsic parameters of the camera using a camera pose estimation algorithm based on camera intrinsic parameters, the 2D feature points and descriptions, and the 3D feature points and descriptions.

16. An extrinsic camera calibration system for a vehicle camera, comprising: a processor; and a non-transitory memory comprising instructions that, when executed, cause the processor to: receive a first plurality of images acquired using the camera; train a neural network model to detect two-dimensional (2D) feature points and generate descriptions of the detected 2D feature points; construct a feature point library using the trained neural network model, the feature point library comprising three-dimensional (3D) feature points having similar descriptions to the detected 2D feature points; and deploy the trained neural network model and the feature point library on a driver monitoring system (DMS) of the vehicle, the DMS configured to: receive a plurality of images of a driver of the vehicle acquired using the camera; detect 2D feature points in the images of the driver and predict descriptions of the detected 2D feature points using the trained neural network model; map the detected 2D feature points to corresponding 3D feature points of the feature point library; estimate a set of extrinsic parameters of the camera based on the 3D feature points; and extrinsically calibrate the camera using the estimated extrinsic parameters.

17. The extrinsic camera calibration system of claim 16, wherein the neural network model has an encoder / decoder architecture comprising: a first decoder portion trained to detect 2D feature points in a pair of images of the plurality of images; a second decoder portion trained to predict descriptions of the detected 2D feature points; and a third shared encoder portion trained to detect the 2D feature points and predict the descriptions of the detected 2D feature points; wherein the neural network model is jointly trained to detect the 2D feature points and predict the descriptions of the detected 2D feature points.

18. The extrinsic camera calibration system of claim 17, wherein training the neural network model to detect the 2D feature points and generate descriptions of the detected 2D feature points further comprises: using a feature point detector to label pixel locations of each image of the pair of images as either a feature point or a background point and generate a description of each pixel location; backpropagating a first loss from a first output of the first decoder portion, adjusting weights and biases of convolutional layers of the first decoder portion of the neural network model according to a first loss function, using the labeled pixel locations as ground truth data; backpropagating a second loss from a second output of the second decoder portion, adjusting weights and biases of convolutional layers of the second decoder portion of the neural network model according to a second loss function; and backpropagating a third loss from the first and second outputs, and adjusting weights and biases of convolutional layers of the third shared encoder portion of the neural network model according to both the first and second loss functions.

19. The external camera calibration system of claim 18, wherein the first loss function is a sigmoid cross-entropy loss function, and the second loss function is a triple loss function based on a negative log-likelihood loss, and adjusting the weights and biases of the convolutional layers of the second decoder portion according to the second loss function further comprises: computing a first distance between a first feature point of a first image of the pair of images and a second matching feature point of a second image of the pair of images; computing a second distance between a first background point or feature point in the first image and a second background point or feature point in the second image, the first and second background points or feature points being farther apart than a threshold distance; minimizing the first distance and maximizing the second distance using the triple loss function.

20. A method comprising: estimating a set of extrinsic parameters of a camera of a vehicle based on determining three-dimensional (3D) feature points of a driver of the vehicle, the 3D feature points corresponding to two-dimensional (2D) feature points of an image of the driver taken by the camera, the 2D feature points detected by a trained neural network model, wherein the neural network model is jointly trained to detect 2D feature points in images taken by the camera and to predict a description of the 2D feature points, and map the 2D feature points to the 3D feature points based on the predicted description; and externally calibrating the camera using the set of extrinsic parameters.

Images