Automatic calibration of multiple cameras using near-infrared illuminators
The image is captured by the near-infrared illuminator and processed by the automatic calibration module, the problem of the calibration failure of the stereo near-infrared camera system when the scene texture is insufficient is solved, and the reliable automatic calibration of the stereo near-infrared camera system is realized to ensure the success of facial authentication.
Patent Information
- Application Number
- CN201980008571.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-10-04
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art is difficult to automatically calibrate the stereoscopic near-infrared camera system without using external lighting, especially when the scene texture is insufficient, resulting in the camera being unable to recognize key points, resulting in the calibration failure.
The near-infrared illuminator is used to capture point images and flood images, and the contrast normalization of the automatic calibration module is used to detect point features and flood features, disambiguate point features, determine new calibration parameters, and update the camera calibration parameters.
It realizes reliable automatic calibration of the stereoscopic near-infrared camera system without using external lighting, ensuring the success of facial authentication, and improving the stability and user experience of the system.
Smart Images

Figure CN112912931B_ABST
Abstract
Description
Background Art
[0001] Computer security using facial authentication solutions provides users with a convenient and simple way to securely access accounts or sign transactions, which enhances the user experience. The user device can capture images for facial authentication using a stereo camera, which typically requires accurate calibration to enable depth measurement. However, events may occur to the camera that invalidate its factory calibration. For example, the user device may be dropped on a hard surface, which can cause one or both cameras to be out of alignment. Another example includes thermal expansion, which can change the camera calibration at different temperatures. Typical automatic calibration solutions without external lighting may fail due to insufficient texture in the scene for the camera to recognize key points for calibration. Summary of the invention
[0002] This document describes techniques and systems for implementing automatic calibration of multiple cameras using near infrared (NIR) illuminators. The techniques and systems include a user device that uses NIR images captured by multiple cameras, the NIR images including point images and flood images. The user device implements an automatic calibration module that normalizes the contrast of each image. Subsequently, the automatic calibration module detects point features in the point images and detects flood features in the flood images. The automatic calibration module uses the flood features to disambiguate point features in the point images. Subsequently, the automatic calibration module uses the disambiguated point features to determine new calibration parameters for recalibration of multiple cameras. In some aspects, the automatic calibration module uses point images instead of flood images in a flood pipeline to detect flood features in the point images to disambiguate the point features.
[0003] This summary is provided to introduce simplified concepts about automatic calibration of multiple cameras using near infrared illuminators, which are further described below in the detailed description and figures. This summary is neither intended to identify key features of the claimed subject matter nor to be used to determine the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS
[0004] Details of one or more aspects of automatic calibration of multiple cameras using near infrared illuminators are described in this document with reference to the following figures. The same reference numerals are used throughout the drawings to refer to the same features and components.
[0005] Figure 1 An example environment in which techniques for automatic calibration of multiple cameras using near infrared illuminators may be implemented is illustrated.
[0006] Figure 2 Picture shows Figure 1An example implementation of a user device of the present invention includes an automatic calibration module for calibrating multiple cameras.
[0007] Figure 3 An example stereo near-infrared camera system is illustrated.
[0008] Figure 4 An example schematic diagram of an auto-calibration module operable to calibrate multiple cameras using near infrared illuminators is illustrated.
[0009] Figure 5 The following diagrams are further detailed based on the use of dot projectors and flood illuminators. Figure 4 An example schematic diagram of the automatic calibration module.
[0010] Figure 6 The dot projector is used to further illustrate the Figure 4 An example schematic diagram of the automatic calibration module.
[0011] Figure 7 A method for automatic calibration of multiple cameras using a near infrared illuminator including a dot projector and a flood illuminator is described.
[0012] Figure 8 A method for automatic calibration of multiple cameras using a near infrared illuminator including a dot projector is described.
[0013] Fig. 9 An example computing system is illustrated, which may be implemented as described in reference Figures 1 to 8 Any type of client, server, and / or electronic device described herein may be used to implement automatic calibration of multiple cameras using near-infrared illuminators, or any type of client, server, and / or electronic device in which the technology for automatic calibration of multiple cameras using near-infrared illuminators can be implemented. DETAILED DESCRIPTION
[0014] Overview
[0015] This document describes techniques and systems for implementing automatic calibration of multiple cameras using near infrared (NIR) illuminators. The calibration of a multi-camera system such as a stereo camera can be invalidated based on various events including impact forces or thermal expansion. General automatic calibration solutions may fail for multi-camera systems that do not use external illumination because the texture on the scene is insufficient for the camera to identify key points for calibration. In order to provide reliable automatic calibration techniques for multi-camera systems that do not use external illumination in the visible spectrum, such as stereo near infrared (NIR) cameras using near infrared illuminators, techniques and systems are described for finding correspondences between pairs of images captured by stereo near infrared cameras and then feeding these correspondences to an optimization solver that provides a new calibration for the camera. More specifically, the camera captures a point image pair based on NIR light output by a point projector, and a flood image pair based on NIR light output by a flood projector. The point image pair is fed into a point pipeline to determine point features. The flood image pair is fed into a flood pipeline to detect flood feature matches between the flood image pairs. Subsequently, ambiguity between point features in the point image pair is resolved using flood feature matching, which enables the point features to be matched between the point images. Using the matched point features, the solver generates a camera calibration pattern that minimizes the error between the matched point features from each image in the point image pair (e.g., a left point image and a right point image) and estimates new calibration parameters, which are used to update the camera.
[0016] In various aspects, a method for automatic calibration of multiple cameras using near-infrared illuminators performed by a user device is described. The method includes: receiving multiple images from a stereo near-infrared camera system, the multiple images including point images and flood images. The method also includes: normalizing the contrast of the images to provide normalized point images and normalized flood images. In addition, the method includes: detecting point features in each normalized point image. In addition, the method includes: detecting flood features in each normalized flood image by identifying the location of feature points in the normalized flood image and determining the surrounding context of the feature points; determining flood feature matching based on comparison of flood features between normalized flood images. In addition, the method includes: using flood feature matching to disambiguate the detected point features to determine point feature correspondence between normalized point images. In addition, the method includes: determining updated calibration parameters for the stereo near-infrared camera system based on the point feature correspondence. The updated calibration parameters are used for image capture using the stereo near-infrared camera system.
[0017] In various aspects, a method for automatic calibration of multiple cameras in a user device using a near-infrared illuminator is described. The method includes: receiving a point image pair from a stereo near-infrared camera system; and normalizing the contrast of each point image to provide a normalized point image pair. The method also includes: in a point pipeline, detecting a point feature in each of the normalized point images. In addition, the method includes, in a flood pipeline: detecting a flood feature in a local area of each of the normalized point images, the flood feature corresponding to a descriptor of a feature point location and a surrounding context of the feature point location; and determining a flood feature match based on a comparison of the flood features between the normalized point image pairs. In addition, the method includes: using flood feature matching to disambiguate the detected point features and determine the point feature correspondence between the normalized point image pairs. In addition, the method includes: determining updated calibration parameters for the stereo near-infrared camera system based on the point feature correspondence.
[0018] These are just a few examples of how the described techniques and devices can be used to implement automatic calibration of multiple cameras using near infrared illuminators. Other examples and implementations are described throughout the document. This document now turns to an example operating environment, after which example devices, methods, and systems are described.
[0019] Operating Environment
[0020] Figure 1An example environment 100 in which techniques for automatic calibration of multiple cameras using near infrared illuminators may be implemented is illustrated. The example environment 100 includes a user device 102 (e.g., an electronic device) that includes or is associated with a near infrared (NIR) camera system 104, an automatic calibration module 106. As further described below, the NIR camera system 104 is a stereo camera system that includes a NIR illuminator 108, a first NIR camera 110, and a second NIR camera 112. The NIR camera system 104 is implemented to capture an image of a user 114 for facial authentication to unlock the user device 102. Images are captured using near infrared (NIR) light, which generally refers to light in a wavelength range between approximately 750 nm and 2,500 nm. NIR light is invisible to the human eye and, therefore, can be used to capture images of the user 114 without external illumination (e.g., in the dark). As further described below, the NIR illuminator 108 projects a dot pattern of NIR light and also projects a flood of NIR light. A first NIR camera 110 positioned on the right side of the device (to the user's left when the device is facing the user) captures a right flood image 116 and a right dot image 118. Similarly, a second NIR camera 112 positioned on the left side of the device (to the user's right when the device is facing the user) captures a left flood image 120 and a left dot image 122.
[0021] If the user device 102 experiences an impact force, such as by being dropped on a hard surface 124, the first NIR camera 110 and the second NIR camera 112 may physically move within the user device 102, which may invalidate the calibration of the cameras relative to each other. The user device 102 may experience thermal expansion (not shown), which may also negatively affect the calibration of the stereo camera by physically changing the position of the stereo camera. The next time the user 114 attempts to unlock the user device 102 using facial authentication, the unlock attempt may fail due to incorrect calibration of the first NIR camera 110 and the second NIR camera 112. For example, the facial features of the user 114 in each of the left image and the right image are unlikely to match, thereby generating data that cannot be verified compared to the known profile data used to unlock the user device 102.
[0022] To address these calibration invalidators, the techniques use images captured by the NIR camera system 104 to automatically calibrate the user device 102. By doing so, the techniques automatically update the calibration of the first NIR camera 110 and the second NIR camera 112. The auto-calibration module 106 can perform auto-calibration during a facial authentication attempt using the NIR camera system 104 to unlock the user device 102. If the authentication is denied, the user device 102 remains in the locked state 126-1, as represented by the lock icon 128. If the user 114 has been authenticated, the user device 102 transitions to the unlocked state 126-2, as shown in FIG. Figure 1 The unlock icon 130 is used to represent the unlocking function.
[0023] Throughout this disclosure, examples are described in which a computing system (e.g., user device 102, client device, server device, computer, or other type of computing system) can analyze information associated with a user (e.g., radar, inertial, and facial recognition sensor data) such as the facial features just mentioned. However, the computing system can be configured to use the information only after the computing system receives explicit permission to use the data from the user of the computing system. For example, in the case where the user device 102 analyzes sensor data of facial features to authenticate the user 114, the sensor data is contained in the security pipeline and cannot be removed from the security pipeline. Individual users can continuously control which programs can or cannot process sensor data. In addition, before the collected information is delivered, stored, or otherwise used, the collected information can be preprocessed in one or more ways so that personally identifiable information can be removed. For example, before the user device 102 shares sensor data with another device (e.g., to train a model executed at another device), the user device 102 can preprocess the sensor data to ensure that any user-identifying information or device-identifying information embedded in the data is removed. Thus, the user may control whether information about the user and the user's device is collected and, if collected, how such information may be used by the computing device and / or remote computing systems.
[0024] In more detail, considering the example implementation 200 of the user device 102 is illustrated Figure 2 , the user device can implement automatic calibration of multiple cameras using near infrared illuminators. Figure 21, a user device 102 having various example devices, including a smart phone 102-1, a tablet 102-2, a laptop computer 102-3, a desktop computer 102-4, a computing watch 102-5, computing glasses 102-6, a gaming system 102-7, a home automation and control system 102-8, and a microwave oven 102-9. The user device 102 may also include other devices such as televisions, entertainment systems, audio systems, cars, drones, trackpads, drawing tablets, netbooks, e-readers, home security systems, and other home appliances. Note that the user device 102 may be wearable, non-wearable but mobile, or relatively fixed (e.g., desktops and appliances).
[0025] The user device 102 also includes one or more computer processors 202 and one or more computer-readable media 204, which include memory media and storage media. Applications and / or operating systems (not shown) implemented as computer-readable instructions on the computer-readable medium 204 can be executed by the computer processor 202 to provide some or all of the functionality described herein. For example, the computer-readable medium 204 may include an automatic calibration module 106 and a facial recognition module 206. The computer-readable medium 204 includes a secure storage (not shown) that is inaccessible to processes or applications in the user space. The secure storage is configured to store security data (e.g., user credentials) for privacy control, such as controls for unlocking the user device 102 (including facial authentication data, password / password information, fingerprint data, etc.). Although the security data can be used to authenticate the user 114 to unlock the user device 102 using facial authentication, password / password authentication, fingerprint authentication, etc., personally identifiable information about the user 114 cannot be obtained through the security data. Specifically, the user 114 cannot be identified by the security data. Instead, where explicit permission from the user has been previously obtained, security data is used to determine whether data received from the user attempting to unlock the phone matches stored profile data representing the user who has set up security measures on the user device 102. In an example, an embedding generated from a captured NIR image of the user's face is a numerical vector representation of the facial features of the user 114 and is used during a facial authentication attempt to compare to previously stored embeddings to locate a match.
[0026] The facial recognition module 206 may invoke a machine learning model to use NIR images (e.g., images 116, 118, 120, 122) captured by the NIR camera system 104 to generate authentication data that may be used to verify against the stored security data in order to authenticate the user 114 and unlock the user device 102. The computer-readable medium 204 includes a stored profile that is not accessible by processes or applications in user space.
[0027] User device 102 may also include network interface 208. User device 102 may use network interface 208 to transmit data over a wired, wireless, or optical network. By way of example and not limitation, network interface 208 may transmit data over a local area network (LAN), a wireless local area network (WLAN), a personal area network (PAN), a wide area network (WAN), an intranet, the Internet, a peer-to-peer network, a point-to-point network, or a mesh network.
[0028] The user device 102 may also include an authentication system 210. Various embodiments of the authentication system 210 may include a system on a chip (SoC), one or more integrated circuits (ICs), a processor with embedded processor instructions or configured to access processor instructions stored in a memory, hardware with embedded firmware, a printed circuit board with various hardware components, or any combination thereof. As described in further detail below, the authentication system 210 may implement the facial recognition module 206 in a secure mode to compare the authentication data received from the user 114 with the security data stored in the secure storage for authenticating the user 114 to unlock the user device 102. In some aspects, the authentication system 210 uses the image data obtained from the NIR camera system 104 to generate the authentication data and provide the authentication data to the secure storage so that the secure storage can compare the authentication data with the stored security data and determine whether there is a match.
[0029] The NIR camera system 104 is implemented to capture NIR image data that can be used to generate a three-dimensional depth map of an object, such as a user's face. Figure 3 The NIR camera system 104 is described in more detail in FIG. Figure 3 , the NIR camera system 104 includes the NIR illuminator 108, which includes a dot projector 302 and a flood illuminator 304. The NIR camera system 104 also includes a left camera sensor 308, a right camera sensor 306, an image processor unit 310, and a camera driver system 312.
[0030] The NIR illuminator 108 projects light in the near infrared spectrum and includes a dot projector 302 and a flood illuminator 304. The flood illuminator 304 illuminates the object with a dedicated flash of NIR light. The right camera sensor 306 and the left camera sensor 308 (e.g., the first NIR camera 110 and the second NIR camera 112, respectively) both capture an image of the object (referred to herein as a flood image) based on the NIR light output by the flood illuminator 304 by capturing photons of NIR light reflected from the object. The dot projector 302 projects a plurality of (e.g., hundreds, thousands) NIR dots onto the object, and the right camera sensor 306 and the left camera sensor 308 both capture an image of the resulting dot pattern (referred to herein as a dot image). The image processor unit 310 reads the image and the dot pattern and generates a three-dimensional facial map. Because multiple (e.g., two) cameras are used, the image processor unit 310 calculates the difference between matching points on different captured images, which provides a depth measurement for corresponding pixels that can be used to generate a three-dimensional facial map.
[0031] The camera driver system 312 enables communication between the NIR camera system 104 and other components of the user device 102, such as the computer processor 202, the automatic calibration module 106, and the authentication system 210. The camera driver system 312 can be initiated by any suitable trigger, such as a user input received via an actuation control or a button press, or a signal received from one or more sensors 212. In an example, a proximity sensor can transmit a signal indicating that the user 114 is close to the user device 102 and is likely to attempt to unlock the user device 102 using facial authentication due to the proximity. The camera driver system 312 also controls settings of the right camera sensor 306 and the left camera sensor 308, including calibration parameters.
[0032] Return to reference Figure 2As mentioned above and further described below, the auto-calibration module 106 may use the captured images from the NIR camera system 104 to determine calibration updates for the right camera sensor 306 and the left camera sensor 308. When the point image pairs and / or flood image pairs captured by the right camera sensor 306 and the left camera sensor 308 fail to produce a sufficient number of correspondences (e.g., matching features between the left point image and the right point image and / or between the left flood image and the right flood image) to achieve depth measurement, the auto-calibration module 106 may perform calibration updates at any suitable time, such as during a face authentication attempt. In one aspect, the auto-calibration module 106 sends the point image pairs through the point pipeline and separately sends the flood image pairs through the flood pipeline. The output of the flood pipeline is combined with the output of the point pipeline to disambiguate the point features in the point images. Subsequently, the auto-calibration module 106 determines calibration updates for the right camera sensor 306 and the left camera sensor 308 using an iterative solver, such as a linear solver.
[0033] Alternatively, the auto-calibration module 106 may use the point image pairs as separate inputs to the point pipeline and the flood pipeline. By processing local regions of the point images in the flood pipeline, rather than processing the point images globally, the flood pipeline may produce point features and descriptors that may be used to disambiguate the output of the point pipeline. The auto-calibration module 106 may then use an iterative solver, such as a nonlinear solver, to determine calibration updates for the right camera sensor 306 and the left camera sensor 308.
[0034] The sensor 212 of the user device 102 may include any of a variety of sensors, such as an audio sensor (e.g., a microphone), a touch input sensor (e.g., a touch screen), an image capture device (e.g., a camera or video camera), a proximity sensor (e.g., a capacitive sensor), or an ambient light sensor (e.g., a photodetector). In at least some embodiments, the user device 102 may include a radar system (not shown) to detect proximity of the user 114 to the user device 102, and based on the proximity, initiate one or more components and / or functions, such as initiating the NIR camera system 104 and the authentication system 210 to initiate a facial authentication attempt.
[0035] The user device 102 may also include a display device, such as a display 214. The display 214 may include any suitable display device, such as a touch screen, a liquid crystal display (LCD), a thin film transistor (TFT) LCD, an in-place switching (IPS) LCD, a capacitive touch screen display, an organic light emitting diode (OLED) display, an active matrix organic light emitting diode (AMOLED) display, a super AMOLED display, etc.
[0036] These and other capabilities and configurations are described in more detail below and Figures 1 to 3 The way entities act and interact with each other. These entities can be further divided, combined, etc. Figure 1 Environment 100 and Figures 2 to 9 The detailed description illustrates some of the many possible environments and devices in which the described techniques can be employed.
[0037] Figure 4 Illustrated is a diagram of a method operable to calibrate multiple cameras using near infrared illuminators. Figure 1 An example schematic diagram of an automatic calibration module 400. The automatic calibration module 406 is from relative to Figures 1 to 3 The described NIR camera system 104 receives NIR images 402. The NIR images 402 as described above include a point image pair and a flood image pair. The point image pair may include a left point image 122 and a right point image 118. The flood image pair may include a left flood image 120 and a right flood image 116. As further described below, in some aspects, the NIR images 402 may include a point image pair without a flood image. Each pair of images 402 may include a left image and a right image captured by a left camera sensor 308 and a right camera sensor 306, respectively.
[0038] The automatic calibration module 106 includes a local contrast normalizer 404 that normalizes the contrast of the NIR image 402 in limited areas. For example, the local contrast normalizer 404 can perform histogram equalization in each of a plurality of limited areas of the image. These areas can be of any suitable size, including NxM pixels. In one example, N and M are equal. As the local contrast normalizer 404 sweeps the NxM filter across the image, it performs an equalizing change. This change normalizes the bright and dark portions of the image to similar brightness, essentially graying those portions and leaving the edges in the image. The local contrast normalizer 404 normalizes the contrast of each point image and each flood image.
[0039] The normalized point image is passed to a point pipeline 406, which includes a point feature detector 408. The normalized flood image is passed to a flood pipeline 410, which includes a corner detector 412, a feature descriptor 414, and a flood feature matcher 416. Point feature detector 408 is a matched filter that identifies the locations of points projected by point projector 302 and captured by right camera sensor 306 and left camera sensor 308 in each of left point image 118 and right point image 122.
[0040] Corner detector 412 locates feature points (eg, points of interest) in the normalized flood image and provides corresponding XY positions. Any suitable corner detector may be used to locate feature points. An example corner detector includes the Features from Accelerated Segment Testing (FAST) corner detector.
[0041] Feature descriptor 414 encodes the location of feature points detected by corner detector 412. Feature points may be encoded using any suitable feature descriptor, which provides surrounding context for the feature points. One example feature descriptor is the Binary Robust Independent Elementary Features (BRIEF) algorithm. BRIEF provides a bit vector that encodes information about the local area around the XY location detected by corner detector 412 in both the left normalized flood image and the right normalized flood image.
[0042] Flood feature matcher 416 matches feature locations between the left flood image and the right flood image and provides a flood feature match. A flood feature refers to a combination of a detected corner point and a corresponding descriptor at the location of the detected corner point. In one example, the flood feature is a combination of a FAST detected corner point and a BRIEF descriptor at the location of the corner point. Flood feature matcher 416 compares the flood features identified in the two flood images and generates a matched correspondence. Several distance metrics can be used to determine the quality of the matched correspondence. In one example, feature locations are matched between the left flood image and the right flood image based on the Hamming distance, which is a calculation of the cross-correlation between features. Other correspondence quality checks can also be implemented, including expectation-maximization-like algorithms or other algorithms for filtering the correspondence of outliers to improve quality.
[0043] Combined feature matcher 418 uses the output from the flood pipeline to disambiguate or filter the output from the point pipeline. More specifically, combined feature matcher 418 determines the correspondence between the two point images by using the flood feature matches and the output of flood feature matcher 416 to resolve the ambiguity between points in left point image 122 and points in right point image 118. The flood feature matches from the flood image are used as anchor points in the point image to match additional points around the anchor points.
[0044] Solver 420 uses the matched correspondence between the right point image and the left point image to generate updated calibration parameters 422 for the right camera sensor 306 and the left camera sensor 308. Solver 420 can be a linear solver, a nonlinear solver, or a solver with a combination of linear and nonlinear aspects. Figure 5 and Figure 6These and other capabilities and configurations are described in greater detail.These components and aspects of the auto-calibration module 106 may be further divided, combined, etc.
[0045] Figure 5 The diagram is further detailed based on the use of a dot projector and a flood illuminator. Figure 4 500. In the example schematic diagram 500, a point image 502 and a flood image 504 captured by the NIR camera system 104 are received by the auto-calibration module 106 in an image space 506. The point image 502 includes a point image pair (e.g., a left point image and a right point image captured by the left camera sensor 308 and the right camera sensor 306, respectively). The auto-calibration module 106 includes a rotator 508 that rotates each of the point image 502 and the flood image 504. Subsequently, the auto-calibration module 106 passes the rotated point image to a first local contrast normalizer (LCN1 510) and passes the rotated flood image to a second local contrast normalizer (LCN2 512), both of which are included in a rotated image space 514. LCN1 510 and LCN2 512 are both Figure 4 An example of the local contrast normalizer 404 in FIG.
[0046] As described above, LCN1 510 normalizes the rotated point image and provides the normalized point image to the point detector 516. The point detector 516 detects the position of the point in each of the normalized point images. The point detector 516 is a matched filter (e.g., a linear filter) that compares the segmentation of the input image with a known template of the point pattern from the point projector 302. In the example, the point pattern is a five-point quincunx grid that includes four points forming a square and one point in the center of the square. The point detector 516 scans the pattern over the image and marks the position corresponding to the match (e.g., five points aligned with the five points of the filter). The point detector 516 outputs the numerical position of the point in the image.
[0047] Normalization of the contrast of the point image normalizes the response of the point detector 516 to effectively enable a single threshold (e.g., an intensity threshold) to be used to determine what is considered a detected point (e.g., a set of matching responses) when analyzing the output of the linear filter in the point detector 516. If the intensity of the matching response is above the threshold, then the matching response is determined to be a detected point. Without normalization of contrast, the degree of response of the point detector 516 varies across the image field of view based on various depths and various reflectances in the scene. These differences can cause some points to not be detected because they are overexposed or underexposed. For example, if the image includes a user with fair skin and wearing a dark coat, the point detector 516 may detect bright spots on the user's face (e.g., a strong response) and dark spots on the coat (e.g., a weak response), and determine that the dark spots may not be considered detected points. Normalizing the contrast first makes the bright spots and dark spots have the same intensity and makes the response of the point detector 516 relatively the same across the image, which makes the points more easily identifiable by the filter.
[0048] Using the normalized image as input to the point detector 516 enables a single threshold to identify the location of the point pattern in the image. Without normalization, darker areas may not trigger the filter threshold.
[0049] Point detector 516 provides output to point connector 518 and rectifier 520. Point connector 518 calculates neighboring points given a starting point. The starting point is output by point detector 516, and point connector 518 identifies the closest neighbor of each starting point on each point image. The results of point connector 518 are passed to point matcher 522 in rectified image space 524.
[0050] Corrector 520 is a feature point corrector that undistorts and transforms feature points to canonical space before they are run through a matching algorithm to find correspondences between images (e.g., representing correspondences between cameras). Corrector 520 uses the original calibration of the camera sensors to correct the feature points. For example, if the two camera sensors 306, 308 are slightly out of position relative to each other, then corrector 520 can transform the coordinate system of the XY position of the points in the left point image and the right point image to canonical space, which enables the points to be matched more accurately. Corrector 520 provides the output of the corrected points to point matcher 522.
[0051] LCN2 512 normalizes the rotated flood image and provides the normalized flood image to flood detector 526. Flood detector 526 includes Figure 4504. Flood detector 526 provides feature point locations and feature descriptors from each of flood images 504 to rectifier 520. Rectifier 520 transforms feature point locations from flood images 504 to the canonical space described above. Rectifier 520 then outputs the rectified flood feature points to point matcher 522, flood matcher 528, flood inlier 530, and match stitcher 532.
[0052] Flood matcher 528 (e.g., flood feature matcher 416) attempts to find matches between the XY locations and the descriptors of the rectified flood feature points in the left and right flood images. In various aspects, flood matcher 528 uses a cost function, such as the Hamming distance. Flood matcher 528 passes the set of detected matches to flood inlier 530 to filter the detected matching outliers. In various aspects, flood matcher 528 and flood inlier 530 are implemented together as Figure 4 528 and the flood feature matcher 416. Alternatively, the flood matcher 528 and the flood inlier 530 may be implemented separately.
[0053] Flood inlier 530 takes as input an initial set of matches between the left and right flood images and filters it to produce a subset of matches that has fewer false matches than the initial set of matches. In an example, flood inlier 530 implements a method similar to that of an expectation-maximization algorithm that determines which of the matches are correct matches and which are false (e.g., an exo-pointer). In some instances, flood images 504 may include noise that produces false feature matches between flood images 504. For example, if flood images 504 include a wall that does not have a feature point of interest, then several random matches that do not correspond to the same point may be identified due to the noise in flood images 504. Therefore, flood inlier 530 removes false matches from the initial set of matches and outputs the correct matches.
[0054] In addition, the flood inlier 530 may include a first to second matched correspondence filter to further filter the exlier matches. When matching features from left to right in an image, there may be many candidates to match a particular feature in another image. The filter determines the cost of the best candidate and the cost of the second best candidate. If the cost ratio between two candidates is above a threshold (e.g., a similarity threshold), then both candidates are considered to be low confidence matches and therefore not matched to the particular feature in the other image. If the cost ratio is below a threshold, then the best candidate may be considered to be a match to the particular feature in the other image, and the second best candidate may be considered a false match.
[0055] Point matcher 522 uses as input the rectified points, the rectified flood feature points, the output of flood inpointer 530, and the output of point connector 518 to determine matches between points in left point image 502 and right point image 502. It can be assumed that substantially all points in point image 502 look the same. Therefore, information from the flood image is used to match points between left point image 502 and right point image 502. In an example, point matcher 522 uses the locations of matches from flood image 504 as anchor points in the point image to find correspondences between left point image 502 and right point image 502. Specifically, point matcher 522 uses the XY locations and the anchor point locations to identify additional matches in the point image, such as points around the anchor point. In a simple example, imagine a flood feature match that anchors two points together, one in the left point image 122 and one in the right point image 118. Using this information, point matcher 522 can also match points near the anchor point. The point matcher 522 uses a combination of the anchor point and point proximity information to provide an output comprising an index pair (I, J) indicating which feature (eg, point) in the left point image 122 matches which feature in the right point image 118 .
[0056] The point matcher 522 provides the output of the point correspondence to the matching stitcher 532. In various aspects, both the point matcher 522 and the matching stitcher 532 are Figure 4 4. Alternatively, the point matcher 522 and the matching stitcher 532 may be implemented separately.
[0057] Match stitcher 532 uses as input the corrected points (output by corrector 520), point correspondences (output by point matcher 522), and correct flood feature matches (output by flood inlier 530) to combine the matches found as part of the flood pipeline with those found in the point pipeline. The results are output to linear solver 534.
[0058] The linear solver 534 includes an iterative solver such as an L1 solver or an L2 solver, which generates a camera calibration pattern that minimizes the correction error between the correspondences from the left point image and the right point image. In essence, the linear solver 534 performs a linear approximation of a nonlinear problem. For example, the points from the left image are warped into the right image to measure the misalignment between these points (e.g., y-axis phase difference). This enables the stereo algorithm for depth calculation to run on the same scan line in the image pair. The linear solver 534 attempts to fit the position of the correspondence matched by the matching stitcher 532 to the new calibration parameters. In various aspects, the linear solver 534 optimizes the focal length, roll (relative to the two camera sensors 306, 308) and vertical shift (relative to the two camera sensors 306, 308). In order to improve robustness, especially when there is a mismatch, the linear solver 534 can use the L1 error instead of the square error. The L1 error refers to the minimum absolute deviation. The linear solver 534 outputs an update to the calibration parameters that minimizes the correction error between the left-point image and the right-point image.
[0059] Based on the output of the linear solver 534, the original calibration parameters of the camera sensors 306, 308 are calibrated and updated 536 to provide a new calibration for the camera sensors 306, 308. The auto-calibration module 106 passes the new calibration parameters to the camera driver system 312 to recalibrate the right camera sensor 306 and the left camera sensor 308 to capture additional point images and flood images.
[0060] Figure 6 An example schematic diagram 600 of automatic calibration of multiple cameras using a dot projector (without a flood illuminator) is illustrated. Many of the components illustrated in schematic diagram 600 are similar to those of FIG. Figure 5 600 , and are therefore described in further detail above. Due to hardware limitations, the spot image is typically captured at a separate time from the flood image, which in some cases can lead to aliasing when matching features between the images. Therefore, the example schematic 600 provides an alternative to using only the spot image 502 (and raw camera calibration parameters) as input to auto-calibrate multiple cameras. The flood image 504 is not used in this schematic 600 for auto-calibrating multiple cameras.
[0061] In the illustrated example, a pair of point images 502 is used as input to the auto-calibration module 106 to calibrate the camera sensors 306, 308. In the image space 506, the point images 502 are rotated and passed to LCN1 510 in the rotated image space. As described above, LCN1 510 normalizes the contrast in each of the point images. LCN1 510 then outputs the same normalized point image as a separate instance to both the flood pipeline and the point pipeline, as described in reference to FIG. Figure 4 For example, LCN1 510 passes the normalized point images to a point detector 516, which detects the location of points in each of the normalized point images. LCN1 510 also provides the normalized point images as output to a flood detector 526.
[0062] The point detector 516 passes the output to the point connector 518, the corrector 520, and also to the matching stitcher 532. The corrector 520 corrects the detected points output by the point detector 516 to the canonical space based on the original calibration parameters of the camera sensors 306, 308, and outputs the corrected point positions to the point matcher 522 in the corrected image space 524. The point connector 518 obtains the point positions from the point detector 516 and calculates the neighboring points of each point. Subsequently, the point connector 518 passes the output to the point matcher 522.
[0063] Using the normalized point images, the flood detector 526 provides the XY location and descriptors of the flood feature to each of the rectifier 520 and the match stitcher 532. The flood detector 526 determines the XY location of the point by searching a limited area (e.g., a narrow segment on the y-axis, and also a limited length on the x-axis) of each of the point images 502. Limiting the search area reduces ambiguity between points and provides sufficient information (e.g., texture) in the images to faithfully match the points between the left point image 502 and the right point image 502. The rectifier 520 undistorts and rectifies the XY location and descriptors to provide rectified flood features to each of the point matcher 522 and the flood matcher 528 in the rectified image space 524. The flood matcher 528 (e.g., the flood feature matcher 416) attempts to find a match between the XY location and the descriptors of the rectified flood feature points in the left flood image 504 and the right flood image 504. Flood matcher 528 passes detected matches to flood inlier 530 to filter detected matched outliers. Flood inlier 530 filters matches within a similarity threshold of another match to effectively remove false matches and maintain correct matches. Flood inlier 530 outputs the results to point matcher 522 and match stitcher 532.
[0064] As described above, the point matcher uses as input the rectified points, the rectified bloom feature points, the output of the bloom inlier 530, and the output of the point connector 518 to determine matches between points in the left point image 502 and the right point image 502. The point matcher 522 outputs the point correspondences between the two point images 502 to the matching stitcher 532 for use as anchor points in the point images 502. The matching stitcher 532 uses the outputs from the bloom detector 526, the point detector 516, the point matcher 522, and the bloom inlier 530 to determine additional matching points around the anchor points. The results of the matching stitcher 532 are then passed to the nonlinear solver 602.
[0065] The nonlinear solver 602 is configured to optimize the six degrees of freedom (6-DOF) pose of one camera relative to another camera to effectively minimize a nonlinear objective function. The objective function includes extra nuisance and / or latent variables representing the positions of N three-dimensional (3D) points in space, which can plausibly explain the set of N matching detections in the two images. Thus, the objective function is a weighted combination of two terms. The first term robustly measures the reprojection error, which is obtained by projecting the current estimate of the N 3D points in the two views and comparing it with the detection. The second term penalizes 6-DOF pose estimates that are fitted to a low confidence region in a prior distribution of an empirical data set of possible 6-DOF poses.
[0066] In various aspects, the nonlinear solver 602 attempts to find the 6-DOF pose of one camera relative to another camera by solving for the relative translation and rotation of the cameras. To this end, the nonlinear solver 602 optimizes the reprojection error. The reprojection error is a geometric error corresponding to the image distance between the projected point and the measured point. The reprojection error is used to quantify how close the estimate of the 3D point is to recreate the observed detection of the point in two views. Compared with the linear solver 534 that warps the point from the left point image 122 to the right point image 118, the nonlinear solver 602 optimizes on a hypothetical point cloud. For example, a point cloud is generated based on a nonlinear solver that optimizes the 3D point corresponding to each match found. Subsequently, using the current estimate of the calibration parameters, the point cloud is projected into each of the point images 502, and the nonlinear solver 602 measures the phase difference in the x-direction and the y-direction. The optimizer iteratively attempts to update both the calibration parameters and the position of the points to minimize the measured phase difference. As a result, new calibration parameters for the camera sensors 306, 308 are obtained.
[0067] To improve robustness, the nonlinear solver 602 measures the L1 error instead of the squared error. In an example, the nonlinear solver 602 uses a Cauchy error metric or a truncated quadratic metric that is more robust than the squared error. For example, if the reprojection error of the match is greater than two or three pixels, then the inlier match is prioritized to weaken the impact of the match.
[0068] In addition to measuring reprojection error, the optimizer may also be penalized for moving too far from the (empirically determined) distribution of possible poses. Thus, even if a 6-DOF estimate that is far outside the expected range (e.g., a pose that is empirically determined to be unlikely) produces a low reprojection error, the nonlinear solver 602 may be discouraged from selecting that 6-DOF estimate, and instead prioritize more likely poses that produce slightly higher reprojection errors.
[0069] Example Method
[0070] Figure 7 and Figure 8 Example methods 700 and 800 for automatic calibration of multiple cameras using near infrared illuminators are depicted. Methods 700 and 800 may be performed by a user device 102 that uses an automatic calibration module 106 to automatically calibrate multiple cameras using NIR illuminators. Method 700 uses a spot image and a flood image as input for automatic calibration. Method 800 uses a spot image (without a flood image) as input for automatic calibration.
[0071] Methods 700 and 800 are shown as a collection of blocks that specify the operations performed, but are not necessarily limited to the order or combination shown for performing the operations by the corresponding blocks. In addition, any one or more of the operations may be repeated, combined, reorganized, or linked to provide a wide array of additional and / or alternative methods. In the following discussion, reference may be made to Figure 1 Example operating environment 100 or reference Figures 2 to 6 References to entities or processes described in detail in the foregoing are by way of example only. The technology is not limited to being performed by one entity or multiple entities operating on one device.
[0072] At 702, the auto-calibration module receives a plurality of images from the stereo near infrared camera system for calibration. For example, the auto-calibration module 106 receives the spot images 502 (eg, left spot image 122 and right spot image 118) along with the flood images 504 (eg, left flood image 120 and right flood image 116).
[0073] At 704, the auto-calibration module normalizes the contrast of the images to provide a normalized point image and a normalized flood image. For example, the local contrast normalizer 404 normalizes the contrast of each of the images 502, 504 and passes the normalized point image to the point pipeline 406 and passes the normalized flood image to the flood pipeline 410.
[0074] At 706 , the auto-calibration module detects point features in each normalized point image. For example, the point feature detector 408 in the point pipeline 406 identifies the locations of the points in the normalized point image 502 .
[0075] At 708, the auto-calibration module detects flood features in each normalized flood image by identifying locations of feature points in the normalized flood image and determining surrounding contexts of the feature points. For example, corner detector 412 identifies feature point locations in the normalized flood image, and feature descriptor 414 encodes the feature point locations along with surrounding contexts of the feature points.
[0076] At 710, the auto-calibration module determines a flood feature match based on a comparison of the flood features between the normalized flood images. For example, the flood feature matcher 416 matches feature locations between the left and right flood images based on feature point locations and descriptors to provide a flood feature match.
[0077] In 712, the auto-calibration module uses the flood feature matches to disambiguate the detected point features and determine point feature correspondence between the normalized point images. In the example, the combined feature matcher 418 uses the flood feature matches from the flood feature matcher 416 to filter the detected points from the point feature detector 408. More specifically, the flood feature matches are used as anchor points in the point images to identify additional matches.
[0078] At 714 , the auto-calibration module determines updated calibration parameters for the stereo near infrared camera system based on the point feature correspondences. In an example, solver 420 , such as linear solver 534 , generates updated calibration parameters for NIR camera system 104 using the matched correspondences between the left point image and the right point image.
[0079] Figure 8 An example method 800 for automatic calibration of multiple cameras using a near infrared illuminator, specifically a dot projector, is illustrated. In 802, the automatic calibration module receives a pair of point images from a stereo near infrared camera system. Here, the pair of point images 502 (e.g., left point image 122 and right point image 118) is used for automatic calibration purposes without any flood image. The same pair of point images 502 is used as a separate input to the point pipeline 406 and the flood pipeline 410 after being normalized by the local contrast normalizer 404.
[0080] At 804 , the auto-calibration module normalizes the contrast of each point image to provide a normalized point image pair. As described above, the local contrast normalizer 404 normalizes the contrast of the point images 502 .
[0081] At 806 , the auto-calibration module detects point features in each of the normalized point images in the point pipeline. As described above, the point feature detector 408 in the point pipeline 406 identifies the locations of points in the normalized point images 502 .
[0082] At 808, the auto-calibration module detects a flood feature (e.g., a point) in a local area of each of the normalized point images in the flood pipeline. Rather than using the flood image in the flood pipeline 410, the local contrast normalizer 404 passes the normalized point image to the flood pipeline 410 to determine the flood feature in the normalized point image. The flood pipeline may analyze the normalized point image in a limited area to determine the flood feature.
[0083] At 810, the auto-calibration module determines a bloom feature match based on a comparison of the bloom features between the normalized point images. For example, the bloom feature matcher 416 matches feature locations between the left point image and the right point image based on feature point locations and descriptors of the bloom feature.
[0084] At 812, the auto-calibration module uses flood feature matching to disambiguate the detected point features and determine point feature correspondence between the normalized point images. For example, the combined feature matcher 418 uses the point XY position from the point feature detector 408 and the anchor point position from the flood feature matcher 416 to provide additional matches in the point image 502.
[0085] At 814, the auto-calibration module determines updated calibration parameters for stereo near infrared camera system 104 based on the point feature correspondences. In an example, solver 420, such as nonlinear solver 602, optimizes a 6-DOF pose of first NIR camera 110 (e.g., right camera sensor 306) relative to second NIR camera 112 (e.g., left camera sensor 308).
[0086] Generally, any of the components, modules, methods, and operations described herein may be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Some operations of the example methods may be described in the general context of executable instructions stored on a computer-readable storage memory located locally and / or remotely from a computer processing system, and implementations may include software applications, programs, functions, and the like. Alternatively or in addition, any functionality described herein may be performed, at least in part, by one or more hardware logic components, such as, but not limited to, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip (SoC), a complex programmable logic device (CPLD), and the like.
[0087] Example computing system
[0088] Fig. 9 900, which may be implemented as described with reference to the previous Figures 1 to 8 Any type of client, server, and / or electronic device described may implement techniques for automatic calibration of multiple cameras using near infrared illuminators.
[0089] The computing system 900 includes a communication device 902 that enables wireless and / or wired communication of device data 904 (e.g., radar data, authentication data, reference data, received data, data being received, data scheduled for broadcast, and data packets of data). The device data 904 or other device content may include configuration settings for the device, media content stored on the device, and / or information associated with a user of the device (e.g., the identity of a person within a radar field or customized air gesture data). The media content stored on the computing system 900 may include any type of radar data, biometric data, audio data, video data, and / or image data. The computing system 900 includes one or more data inputs 906, through which any type of data, media content, and / or input may be received, such as human speech, interaction with a radar field, touch input, user-selectable input or interaction (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and / or image data received from any content and / or data source.
[0090] The computing system 900 also includes a communication interface 908, which can be implemented as any one or more of a serial and / or parallel interface, a wireless interface, any type of network interface, a modem, and any other type of communication interface. The communication interface 908 provides a connection and / or communication link between the computing system 900 and a communication network through which other electronic, computing, and communication devices communicate data with the computing system 900.
[0091] The computing system 900 includes one or more processors 910 (e.g., any of a microprocessor, controller, or other controller) that can process various computer executable instructions to control the operation of the computing system 900 and enable techniques for automatic calibration of multiple cameras using near-infrared illuminators, or in which automatic calibration of multiple cameras using near-infrared illuminators can be implemented. Alternatively or in addition, the computing system 900 can be implemented with any one or combination of hardware, firmware, or fixed logic circuit systems implemented in conjunction with processing and control circuits (which are generally identified at 912). Although not shown, the computing system 900 may include a system bus or data delivery system that couples various components within the device. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any one of a variety of bus architectures.
[0092] The computing system 900 also includes computer-readable media 914, such as one or more memory devices that enable persistent and / or non-transitory data storage (as opposed to mere signal transmission), examples of which include random access memory (RAM), non-volatile memory (e.g., any one or more of read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), and disk storage devices. The disk storage device may be implemented as any type of magnetic or optical storage device, such as a hard drive, a recordable and / or rewritable compact disk (CD), any type of digital versatile disk (DVD), etc. The computing system 900 may also include a mass storage media device (storage media) 916.
[0093] The computer-readable medium 914 provides a data storage mechanism for storing device data 904, as well as various device applications 918 and any other type of information and / or data related to the operational aspects of the computing system 900. For example, an operating system 920 may be maintained as a computer application with the computer-readable medium 914 and executed on the processor 910. The device applications 918 may include device managers, such as any form of control applications, software applications, signal processing and control modules, code inherent to a specific device, abstract modules, air gesture recognition modules, and other modules. The device applications 918 may also include system components, engines, modules, or managers to implement automatic calibration of multiple cameras using near infrared illuminators, such as the NIR camera system 104 and the automatic calibration module 106. The computing system 900 may also include one or more machine learning systems or have access to one or more machine learning systems.
[0094] In the following, some examples are given.
[0095] Example 1: A method for automatic calibration of multiple cameras in a user device, the method comprising: receiving multiple images from a stereo near-infrared camera system, the multiple images including point images and flood images; normalizing the contrast of the images to provide normalized point images and normalized flood images; detecting point features in each normalized point image; detecting flood features in each normalized flood image by identifying the location of the feature points in the normalized flood image and determining the surrounding context of the feature points; determining flood feature matching based on comparison of the flood features between the normalized flood images; using flood feature matching to disambiguate the detected point features to determine point feature correspondence between the normalized point images; and determining updated calibration parameters of the stereo near-infrared camera system based on the point feature correspondence.
[0096] Example 2: The method of claim 1 further comprises: before determining the flood feature match, correcting the feature points based on original calibration parameters of the stereo near-infrared camera system.
[0097] Example 3: A method according to any one of Examples 1 or 2, wherein determining updated calibration parameters of the stereo near-infrared camera system includes: applying an iterative solver that minimizes the correction error between point feature correspondences in the normalized point image.
[0098] Example 4: A method according to Example 3, wherein the iterative solver optimizes the focal length, roll and / or vertical shift of the first camera of the stereo near-infrared camera system relative to the second camera of the stereo near-infrared camera system.
[0099] Example 5: The method according to any of the preceding examples further comprises: filtering the outliers of the flood feature match before using the flood feature match to disambiguate the detected point features.
[0100] Example 6: A method according to any one of the preceding examples, wherein using the floodlight feature matching to disambiguate the detected point features includes: using the floodlight feature matching as an anchor point in a normalized point image; and identifying additional point features around the anchor point to resolve ambiguity in the detected point features of each normalized point image.
[0101] Example 7: A method according to any one of the preceding examples, wherein the normalized contrast comprises: normalization of the contrast of the point image and / or the flood image in a limited area.
[0102] Example 8: A method according to any of the preceding examples, wherein detecting the point features uses a matched filter.
[0103] Example 9: The method according to any of the preceding examples further comprises: rotating each of the point image and the flood image before normalizing the contrast of the image.
[0104] Example 10: The method according to any of the preceding examples further comprises: before determining the bloom feature match, undistorting the feature points and transforming the feature points to a canonical space.
[0105] Example 11: A user device comprising: a stereo near-infrared camera system, the stereo near-infrared camera system being configured to capture multiple images by using multiple near-infrared illuminators, the multiple images including point images and flood images; and a processor and memory system, the processor and memory system being operable to execute stored instructions to implement an automatic calibration module, the automatic calibration module being configured to perform the method according to any one of the preceding examples.
[0106] Example 12: A method for automatic calibration of multiple cameras using near-infrared illuminators in a user device, the method comprising: receiving a point image pair from a stereo near-infrared camera system; normalizing the contrast of each point image to provide a normalized point image pair; in a point pipeline, detecting a point feature in each of the normalized point images; in a flood pipeline: detecting a flood feature in a local area of each of the normalized point images, the flood feature corresponding to a descriptor of a feature point location and a surrounding context of the feature point location; and determining a flood feature match based on a comparison of the flood feature between the normalized point image pairs; using the flood feature match to disambiguate the detected point features and determine point feature correspondence between the normalized point image pairs; and determining updated calibration parameters for the stereo near-infrared camera system based on the point feature correspondence.
[0107] Example 13: The method according to Example 12 further includes: after detecting the flood feature and before determining the flood feature match, correcting the feature point position to the canonical space based on the original calibration parameters of the stereo near-infrared camera system.
[0108] Example 14: A method according to any one of Example 12 or Example 13, wherein: a stereoscopic near-infrared camera system comprises: a first near-infrared camera and a second near-infrared camera; and determining the updated calibration parameters comprises: applying a nonlinear solver to the point feature correspondence to optimize the six-degree-of-freedom posture of the first near-infrared camera relative to the second near-infrared camera.
[0109] Example 15: The method of Example 14, wherein the nonlinear solver minimizes the reprojection error.
[0110] Example 16: The method of any one of Examples 12 to 15 further comprises: filtering the outliers of the flood feature matching before using the flood feature matching to disambiguate the detected point features.
[0111] Example 17: A method according to any one of Examples 12 to 16, wherein disambiguating the detected point features using flood feature matching includes: filtering the detected point features using the flood feature matching to provide an anchor point; and associating additional point features around the anchor point to resolve ambiguity in the detected point features between each normalized point image.
[0112] Example 18: A method according to any one of Examples 12 to 17, wherein determining the flood feature match includes: filtering the flood feature match based on a similarity threshold between a first best candidate and a second best candidate for the flood feature match between the normalized point image pair.
[0113] Example 19: A method according to any one of Examples 12 to 18, wherein normalizing the contrast comprises normalizing the contrast of each point image in a limited area.
[0114] Example 20: A method according to any one of Examples 12 to 19, wherein detecting the point features uses a matched filter.
[0115] Example 21: The method according to any one of Examples 12 to 20 further includes: rotating each point image before normalizing the contrast of each point image.
[0116] Example 22: The method according to any one of Examples 12 to 21 further comprises: before determining the bloom feature match, undistorting the feature points and transforming the feature points to a canonical space.
[0117] Example 23: A user device comprising: a stereoscopic near-infrared camera system, the stereoscopic near-infrared camera system being configured to use a dot projector to capture near-infrared point image pairs; and a processor and a memory system, the processor and the memory system being operable to execute stored instructions to implement an automatic calibration module, the automatic calibration module being configured to perform the method described in any one of Examples 12 to 21.
[0118] in conclusion
[0119] Although embodiments of techniques and apparatus for implementing automatic calibration of multiple cameras using near infrared illuminators have been described in language specific to features and / or methods, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, specific features and methods are disclosed as example embodiments for automatic calibration of multiple cameras using near infrared illuminators.
Claims
1. A method for automatic calibration of multiple cameras in a user device using a near infrared illuminator, the method comprising: receiving a pair of point images from a stereoscopic near-infrared (NIR) camera system, each point image in the pair of point images comprising a point pattern corresponding to a plurality of NIR points projected onto an object by a point projector and captured by the stereoscopic NIR camera system; normalizing the contrast of each point image to provide a normalized point image pair; outputting a first instance of the normalized point image pair to a point pipeline configured to process point images having a pattern of NIR points, and outputting a second instance of the normalized point image pair to a flood pipeline configured to process a flood image of an object illuminated with a dedicated flash of NIR light; In the point pipeline, detecting point features in each of the normalized point images of a first instance of the normalized point image pair; In the floodlight pipeline: detecting a bloom feature in a local region of each of the normalized point images of a second instance of the normalized point image pair, the bloom feature corresponding to a descriptor of a feature point location and a surrounding context of the feature point location; as well as determining a bloom feature match based on a comparison of the bloom features between the normalized point images of a second instance of the normalized point image pair; disambiguating point features detected from the point pipeline and determining point feature correspondence between the normalized point image pairs using the flood feature matches determined from the flood pipeline; as well as Updated calibration parameters of the stereo near-infrared camera system are determined based on the point feature correspondences.
2. The method according to claim 1, further comprising: After the flood feature is detected and before determining a flood feature match, the feature point positions are corrected to a canonical space based on original calibration parameters of the stereo near-infrared camera system.
3. The method according to claim 1, wherein: The stereo near-infrared camera system comprises: a first near-infrared camera and a second near-infrared camera; and Determining the updated calibration parameters includes applying a nonlinear solver to the point feature correspondences to optimize a six-degree-of-freedom pose of the first near infrared camera relative to the second near infrared camera.
4. The method according to claim 3, wherein: The nonlinear solver minimizes the reprojection error.
5. The method according to claim 1, further comprising: Prior to using the determined bloom feature matches to disambiguate the detected point features, outliers of the bloom feature matches are filtered.
6. The method according to claim 1, wherein: Disambiguating the detected point features using the determined bloom feature matches includes: filtering the detected point features using the determined bloom feature matches to provide anchor points; and Additional point features around the anchor point are associated to resolve ambiguity in the detected point features between each normalized point image in the pair of normalized point images.
7. The method according to claim 1, wherein: Determining the flood feature match includes filtering the flood feature match based on a similarity threshold between a first best candidate and a second best candidate for a flood feature match between the pair of normalized point images.
8. The method according to claim 1, further comprising: Prior to determining the flood feature match, the feature point positions are undistorted and transformed into a canonical space.
9. A user equipment, comprising: a stereoscopic near-infrared (NIR) camera system configured to use a dot projector to capture a pair of NIR dot images, each dot image in the pair of NIR dot images comprising a dot pattern corresponding to a plurality of NIR dots projected onto an object by the dot projector and captured by the stereoscopic NIR camera system; as well as A processor and memory system operable to execute stored instructions to implement an automatic calibration module, the automatic calibration module being configured to: normalizing the contrast of each point image to provide a normalized point image pair; outputting a first instance of the normalized point image pair through a dot pipeline configured to process a dot image having a pattern of NIR dots, and outputting a second instance of the normalized point image pair through a flood pipeline configured to process a flood image of an object illuminated with a dedicated flash of NIR light; In the point pipeline, detecting point features in each of the normalized point images of a first instance of the normalized point image pair; In the floodlight pipeline: detecting a bloom feature in a local region of each of the normalized point images of a second instance of the normalized point image pair, the bloom feature corresponding to a descriptor of a feature point location and a surrounding context of the feature point location; as well as determining a bloom feature match based on a comparison of the bloom features between the normalized point images of a second instance of the normalized point image pair; disambiguating point features detected from the point pipeline using the flood feature matches from the flood pipeline to provide disambiguated point features; determining point feature correspondence between the normalized point image pairs using the disambiguated point features; as well as Updated calibration parameters of the stereo near-infrared camera system are determined based on the point feature correspondences.
10. The user equipment according to claim 9, wherein: The automatic calibration module is further configured to: after detecting the flood feature and before determining the flood feature match, correct the feature point position to a canonical space based on original calibration parameters of the stereo near-infrared camera system.
11. The user equipment according to claim 9, wherein: The stereo near-infrared camera system comprises: a first near-infrared camera and a second near-infrared camera; and The updated calibration parameters are determined based on applying a nonlinear solver to the point feature correspondences to optimize a six-degree-of-freedom pose of the first near-infrared camera relative to the second near-infrared camera.
12. The user equipment according to claim 11, wherein: The nonlinear solver minimizes the reprojection error.
13. The user equipment according to claim 9, wherein: The auto-calibration module is further configured to filter outliers of the determined flood feature matches before using the determined flood feature matches to disambiguate the detected point features.
14. The user equipment according to claim 9, wherein: The automatic calibration module is further configured to: filtering the detected point features using the determined flood feature matches to provide anchor points; as well as Additional point features around the anchor point are associated to resolve ambiguity in the detected point features between each normalized point image in the pair of normalized point images.
15. The user equipment according to claim 9, wherein: The auto-calibration module is further configured to filter the bloom feature matches based on a similarity threshold between a first best candidate and a second best candidate for a bloom feature match between the pair of normalized point images.
16. The user equipment according to claim 9, wherein: The automatic calibration module is further configured to: undistort the feature point positions and transform the feature point positions to a canonical space before determining the flood feature match.
17. One or more computer-readable media having instructions, which in response to execution by a computer processor cause the computer processor to implement an automatic calibration module, the automatic calibration module being configured to: receiving a pair of point images from a stereoscopic near-infrared (NIR) camera system, each point image in the pair of point images comprising a point pattern corresponding to a plurality of NIR points projected onto an object by a point projector and captured by the stereoscopic NIR camera system; normalizing the contrast of each point image to provide a normalized point image pair; outputting a first instance of the normalized point image pair through a dot pipeline configured to process a dot image having a pattern of NIR dots, and outputting a second instance of the normalized point image pair through a flood pipeline configured to process a flood image of an object illuminated with a dedicated flash of NIR light; In the point pipeline, detecting point features in each of the normalized point images of a first instance of the normalized point image pair; In the floodlight pipeline: detecting a bloom feature in a local region of each of the normalized point images of a second instance of the normalized point image pair, the bloom feature corresponding to a descriptor of a feature point location and a surrounding context of the feature point location; as well as determining a bloom feature match based on a comparison of the bloom features between the normalized point images of a second instance of the normalized point image pair; disambiguating point features detected from the point pipeline and determining point feature correspondence between the normalized point image pairs using the flood feature matches determined from the flood pipeline; as well as Updated calibration parameters of the stereo near-infrared camera system are determined based on the point feature correspondences.
18. One or more computer-readable media according to claim 17, wherein: The stereo near-infrared camera system comprises: a first near-infrared camera and a second near-infrared camera; and Determining the updated calibration parameters includes applying a nonlinear solver to the point feature correspondences to optimize a six-degree-of-freedom pose of the first near infrared camera relative to the second near infrared camera.
19. One or more computer-readable media according to claim 18, wherein: The nonlinear solver minimizes the reprojection error.
20. The one or more computer-readable media of claim 17, wherein: Disambiguating the detected point features using the determined bloom feature matches includes: filtering the detected point features using the determined bloom feature matches to provide anchor points; and Additional point features around the anchor point are associated to resolve ambiguity in the detected point features between each normalized point image in the pair of normalized point images.
Citation Information
Patent Citations
Real-time registration of a stereo depth camera array
CN105432079A
Face detection, pose estimation, and distance from camera estimation using single network
CN109325393A
