Calibration for real-time blind registration of different video image streams

Abstract

The present disclosure provides calibration for real-time blind registration of different video image streams. A first calibration image of a calibration target is received and resized to provide a resized first calibration image. Pixel coordinates of the resized first calibration image are mapped to pixel coordinates of the first calibration image to provide a resized map. The resized first calibration image is distortion corrected to provide a distortion corrected, resized first calibration image. Pixel coordinates of the distortion corrected, resized first calibration image are mapped to pixel coordinates of the resized first calibration image to provide a distortion correction map. The distortion corrected, resized first calibration image is resampled to provide a resampled, distortion corrected, resized first calibration image. Pixel coordinates of the resampled, distortion corrected, resized calibration image are mapped to pixel coordinates of the distortion corrected, resized calibration image to provide a resampled map.

Description

Technical Field

[0001] The present disclosure relates to real-time blind registration of different video image streams in a vehicle. Background Art

[0002] A vehicle may be equipped with two or more cameras that simultaneously capture images of the same scene in the environment surrounding the vehicle. It may be desirable to equip a vehicle with spectrally different cameras. When one camera is configured to capture images in a first spectral band but not a second spectral band, and another camera is configured to capture images in a second spectral band but not a first spectral band, the two cameras are spectrally different. Images captured by spectrally different cameras are referred to herein as spectrally different images.

[0003] Two spectrally different images can be combined to form a single multispectral image. Multispectral images can help humans visualize imaged objects by providing spectral information that would be missing in a single spectral image. For example, some objects may be more visible in infrared (IR) images due to their thermal properties, while other objects may have a more unique visual appearance in red, green, and blue (RGB) images. Multispectral images can make both objects visible.

[0004] Two cameras are spatially distinct when they are separated longitudinally and / or laterally by a distance that introduces a parallax effect into the captured image. The farther apart the cameras are, the more pronounced the parallax effect. Vehicle structural layout, visibility, aesthetics, and other considerations can make camera placement challenging. These considerations can determine the spatial separation between the cameras that creates the parallax effect in the captured images of the cameras.

[0005] To combine images of the same scene captured by two different cameras, the images are first registered. Registering images captured by spatially or spectrally distinct cameras is computationally complex and, therefore, resource-intensive and time-consuming. Particularly for video images, registering images captured by both spatially and spectrally distinct cameras introduces further computational complexity.

[0006] Real-time image registration refers to the ability to align images as they are captured or processed without significant latency or time-consuming computations. The present disclosure provides apparatus and methods for real-time registration of spatially and spectrally diverse video images. Summary of the invention

[0007] Multispectral video images can be generated by combining video images captured by spectrally different cameras. Images captured by spectrally different cameras are referred to as spectrally different images in this article. In order to combine spectrally different images (including spectrally different video images), first register the images. Image registration is the process of aligning two or more images taken from different viewing angles or using different modalities. Two commonly used image registration techniques are feature-based registration and intensity-based registration. Feature-based registration involves identifying and matching unique features or key points in the image to establish feature or point correspondences. Unique features include corners, edges, spots or other prominent points. Intensity-based registration relies on the difference in pixel intensity in the image to estimate registration parameters. Due to the difference in spectral information, feature-based and intensity-based registration techniques may be invalid for spectrally different images, thereby complicating the registration process.

[0008] Images captured by spatially different cameras may suffer from parallax effects. The parallax effect is the apparent displacement of an object when viewed from different locations. Objects at different distances will have different relative positions in the two images. The farther apart the images or cameras are, the more pronounced the parallax effect. This makes it difficult to establish correspondences to effectively register the images. Furthermore, correction techniques used to register images captured by widely spaced cameras involve complex geometric transformations and warping to overcome the parallax effect. These transformations require significant computational resources.

[0009] Blind image registration refers to the process of registering two or more images in the absence of any knowledge or information about their features, their pixel intensities, or about the objects in the images. Real-time image registration refers to the ability to register images as they are acquired or processed without significant waiting time or time-consuming computations.

[0010] Apparatus and methods for real-time blind registration of spatially and spectrally distinct video images are disclosed herein.

[0011] A device includes a processor and a memory coupled to the processor, the memory including processor-executable instructions, which, when executed by the processor, configure the processor to: determine first intrinsic camera parameters of a first camera; determine second intrinsic camera parameters of a second camera; calculate a scaling factor for resizing a first calibration image and a second calibration image based on a relationship between the first intrinsic camera parameter and the second intrinsic camera parameter; and resize the first calibration image and the second calibration image according to the scaling factor to provide a resized first calibration image and a resized second calibration image.

[0012] The processor may be further configured to map pixel coordinates of the resized first calibration image to pixel coordinates of the first calibration image to generate a resizing map.

[0013] The processor may also be configured to: receive a distortion correction matrix for the first camera; and apply the distortion correction matrix to the resized first calibration image to provide a distortion corrected, resized first calibration image.

[0014] The processor may be further configured to map pixel coordinates of the distortion corrected, resized first calibration image to pixel coordinates of the resized first calibration image to generate a distortion correction map.

[0015] The processor may be configured to: resample the distortion-corrected, resized first calibration image to provide a resampled, distortion-corrected, resized first calibration image; and map pixel coordinates of the resampled, distortion-corrected, resized first calibration image to pixel coordinates of the distortion-corrected, resized first calibration image to provide a resampled map.

[0016] The processor may also be configured to resample the distortion-corrected, resized first calibration image by: correcting the distortion-corrected, resized first calibration image to provide a corrected, distortion-corrected, resized first calibration image; warping the corrected, distortion-corrected, resized first calibration image to fit the corrected, distortion-corrected, resized second calibration image; and performing inverse correction on the corrected, distortion-corrected, resized first calibration image to provide the resampled, distortion-corrected, resized first calibration image.

[0017] The processor can also be configured to perform correction by: detecting a calibration target in the distortion-corrected, resized first calibration image; detecting the calibration target in a corresponding distortion-corrected, resized second calibration image; identifying a correspondence between the first distortion-corrected, resized calibration image and the second distortion-corrected, resized calibration image; calculating a basic matrix based on the correspondence; calculating a first corrective transformation of the distortion-corrected, resized first calibration image based on the basic matrix; and calculating a second corrective transformation of the distortion-corrected, resized second calibration image based on the basic matrix.

[0018] The processor may also be configured to: apply the first corrective transform to the distortion-corrected, resized first calibration image to provide a rectified, distortion-corrected, resized first calibration image; and map pixel coordinates of the rectified, distortion-corrected, resized first calibration image to pixel coordinates of the distortion-corrected, resized first calibration image to provide a corrective map.

[0019] The processor may also be configured to: determine a best fit polynomial function to fit the rectified, distortion corrected, resized first calibration image to the rectified, distortion corrected, resized second calibration image; determine a warp matrix based on the best fit polynomial function; and apply the warp matrix to the rectified, distortion corrected, resized first calibration image to provide a warped, rectified, distortion corrected, resized first calibration image.

[0020] The processor may be further configured to map pixel coordinates of the warped, rectified, distortion-corrected, resized first calibration image to pixel coordinates of the rectified, distortion-corrected, resized first calibration image to generate a warp map.

[0021] A method includes determining first intrinsic camera parameters of a first camera; determining second intrinsic camera parameters of a second camera; calculating a scaling factor for resizing a first calibration image based on a relationship between the first intrinsic camera parameters and the second intrinsic camera parameters; and resizing the first calibration image and the second calibration image according to the scaling factor to provide a resized first calibration image and a resized second calibration image.

[0022] The method may further include mapping pixel coordinates of the resized first calibration image to pixel coordinates of the first calibration image to generate a resizing map.

[0023] The method may also include receiving a distortion correction matrix for the first camera; and applying the distortion correction matrix to the resized first calibration image to provide a distortion corrected, resized first calibration image.

[0024] The method may further include mapping pixel coordinates of the distortion corrected, resized first calibration image to pixel coordinates of the resized first calibration image to generate a distortion correction map.

[0025] The method may also include: resampling the distortion-corrected, resized first calibration image to provide a resampled, distortion-corrected, resized first calibration image; and mapping pixel coordinates of the resampled, distortion-corrected, resized first calibration image to pixel coordinates of the distortion-corrected, resized first calibration image to provide a resampled map.

[0026] The method may include performing resampling by: rectifying the distortion-corrected, resized first calibration image to provide a rectified, distortion-corrected, resized first calibration image; warping the rectified, distortion-corrected, resized first calibration image to fit a rectified, distortion-corrected, resized second calibration image; and performing inverse rectification on the rectified, distortion-corrected, resized first calibration image to provide the resampled, distortion-corrected, resized first calibration image.

[0027] The method may also include: detecting a calibration target in the distortion-corrected, resized first calibration image; detecting the calibration target in a corresponding distortion-corrected, resized second calibration image; identifying a correspondence between the first distortion-corrected, resized calibration image and the second distortion-corrected, resized calibration image; and calculating a basic matrix based on the correspondence; calculating a first corrective transformation of the distortion-corrected, resized first calibration image based on the basic matrix; and calculating a second corrective transformation of the distortion-corrected, resized second calibration image based on the basic matrix.

[0028] The method may also include: applying the first corrective transform to the distortion-corrected, resized first calibration image to provide a rectified, distortion-corrected, resized first calibration image; and mapping pixel coordinates of the rectified, distortion-corrected, resized first calibration image to pixel coordinates of the distortion-corrected, resized first calibration image to provide a corrective map.

[0029] The method may also include: determining a best fit polynomial function to fit the rectified, distortion-corrected, resized first calibration image to the rectified, distortion-corrected, resized second calibration image; determining a warp matrix based on the best fit polynomial function; and applying the warp matrix to the rectified, distortion-corrected, resized first calibration image to provide a warped, rectified, distortion-corrected, resized first calibration image.

[0030] The method may further include mapping pixel coordinates of the warped, rectified, distortion-corrected, resized first calibration image to pixel coordinates of the rectified, distortion-corrected, resized first calibration image to generate a warp map. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a block diagram of an exemplary vehicle system suitable for implementing the disclosed apparatus and methods;

[0032] Figure 2 is a diagram of an exemplary vehicle equipped with an imaging sensor suitable for implementing the disclosed apparatus and methods;

[0033] Figure 3 is a block diagram of an apparatus for registering different images;

[0034] Figure 4 yes Figure 3 A block diagram of an alternative implementation of the resampling LUT shown in ;

[0035] Figure 5 yes Figure 3 A block diagram of an exemplary embodiment of the device shown in;

[0036] Figure 6 yes Figure 3 A block diagram of an alternative embodiment of the device shown in;

[0037] Figure 7 is a flow chart of a method for real-time registration of images;

[0038] Fig. 8A is Figure 7 Flowchart of the sizing method used in the method shown in ;

[0039] Figure 8B is Figure 7 A flow chart of a distortion correction method used in the method shown in;

[0040] Fig. 9 is Figure 7 Flowchart of the online resampling method used in the method shown in ;

[0041] Fig.10 is a flow chart of the online image registration process;

[0042] Fig.11 is a flow chart of a method for generating a resizing map;

[0043] Fig.12 is a flow chart of a method for generating a distortion correction map;

[0044] Fig.13A and Fig. 13B Flowcharts of methods for generating resampled maps, rectified maps, inverse rectified maps, warped maps, and registered maps are provided;

[0045] FIG. 14A to FIG. 14D Figuratively illustrating resizing of a first calibration image;

[0046] Fig.15 is a flow chart of a method for providing a multispectral image stream; and

[0047] Fig.16 is a flow chart of a method for adjusting a gamma curve for displaying a multispectral image. DETAILED DESCRIPTION

[0048] Figure 1 1 is a block diagram of an exemplary vehicle 10 including a vehicle system 100 suitable for implementing the disclosed apparatus and methods. The vehicle system 100 may include a vehicle 10 equipped with sensors 108, including one or more imaging sensors ( Figure 2 ). Sensors 108 may also include one or more ranging sensors (not separately depicted).

[0049] The vehicle 10 includes a vehicle computer 104, which may include one or more electronic control units (ECUs) disposed within the vehicle 10 to control the operation of various devices and components of the vehicle 10. The computer 104 may be programmed to send and / or receive messages via a vehicle network 114 to communicate with other devices disposed within the vehicle (e.g., sensors 108, actuators, components 110, communication modules, human machine interfaces (HMIs) 112, etc.).

[0050] The vehicle system 100 may include one or more vehicle networks 114 through which various electronic control units (ECUs) including the computer 104 may communicate with each other to send commands and / or exchange, send or receive data. The vehicle network 114 may include, but is not limited to, a controller area network (CAN), a local interconnect network (LIN), Ethernet, In some embodiments, the vehicle computer 104 and / or other components of the vehicle system 100 may be configured to communicate with a remote server 118 via a wide area network (WAN) 116 .

[0051] The computer 104 includes a processor and memory such as are known. In addition, the memory includes one or more forms of computer-readable media and stores instructions that can be executed by the processor to perform various operations including as disclosed herein. For example, the computing device 104 may include programming to operate one or more of vehicle braking, propulsion (i.e., controlling acceleration in the vehicle 10 by controlling one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior lights and / or exterior lights, etc., and determine whether and when the computing device 104 (rather than a human operator) controls such operations.

[0052] The computing device 104 may include more than one computing device (i.e., controllers included in the vehicle 10 for monitoring and / or controlling various vehicle components (i.e., propulsion, braking, steering, etc.), or be communicatively coupled to the more than one computing device via a vehicle communication bus as further described below. The computing device 104 is arranged to communicate via a vehicle communication network 114 (i.e., including a bus in the vehicle 10, such as a controller area network (CAN), etc.); the vehicle 10 network 114 may additionally or alternatively include, for example, known wired or wireless communication mechanisms, i.e., Ethernet or other communication protocols.

[0053] The computing device 104 may transmit and / or receive messages to and from various devices in the vehicle (i.e., controllers, actuators, sensors (including sensor 108), etc.) via the vehicle network 114. Alternatively or additionally, where the computing device 104 includes multiple devices, the vehicle communication network 114 may be used to communicate between the devices represented as computing devices 104 in this disclosure. Furthermore, as mentioned below, various controllers or sensing elements (such as sensors 108) may provide data to the computing device 104 via the vehicle communication network 114.

[0054] Additionally, computing device 104 may be configured to communicate with a remote server computer 118 (i.e., a cloud server) via network 116 through a vehicle-to-infrastructure (V2X) interface, as described below, which includes hardware, firmware, and software that allows computing device 104 to communicate with a remote server computer 118 (i.e., a cloud server) via a vehicle-to-infrastructure (V2X) interface, such as a wireless Internet connection. The V2X interface may thus include a network 116 configured to communicate with a remote server computer 118 using a variety of wired and / or wireless networking technologies (i.e., cellular, The computing device 104 may be configured to communicate with other vehicles 10 through a V2X (vehicle-to-everything) interface using a vehicle-to-vehicle (V-to-V) network formed between neighboring vehicles on the basis of a mobile ad hoc network or through an infrastructure-based network (i.e., based on cellular communication (C-V2X) wireless communication cellular, dedicated short-range communication (DSRC), etc.). The computing device 104 also includes a non-volatile memory such as known. The computing device 104 may record data by storing the data in a non-volatile memory for later retrieval and transmission to a server computer 118 or a user mobile device via the vehicle communication network 114 and the vehicle-to-infrastructure (V2X) interface.

[0055] As already mentioned, programming for operating one or more vehicle 10 components (i.e., braking, steering, propulsion, etc.) without human operator intervention is included in instructions stored in memory and executable by a processor of the computing device 104. Using data received in the computing device 104 (i.e., sensor data from sensors 108, server computer 118, etc.), the computing device 104 can make various determinations and / or control various vehicle 10 components and / or operations. For example, the computing device 104 can include programming to adjust vehicle 10 operating behavior (i.e., the physical manifestation of vehicle 10 operation), such as speed, acceleration, deceleration, steering, etc., as well as strategic behavior (i.e., generally controlling operating behavior in a manner intended to achieve efficient traversal of a route), such as the distance between vehicles and / or the amount of time between vehicles, lane changes, minimum gaps between vehicles, left turn crossing path minimums, arrival times at specific locations, and the minimum time from arrival at an intersection to crossing an intersection (without a signal light).

[0056] The sensors 108 may include a variety of known imaging devices, such as cameras and / or other devices, to provide data via the vehicle communication bus. For example, a radar fixed to the front bumper (not shown) of the vehicle 10 may provide the distance from the vehicle 10 to the next vehicle in front of the vehicle 10, or a global positioning system (GPS) sensor disposed in the vehicle 10 may provide the geographic coordinates of the vehicle 10. For example, the distance provided by the radar and / or other sensors 108 and / or the geographic coordinates provided by the GPS sensor may be used by the computing device 104 to autonomously or semi-autonomously operate the vehicle 10, such as to control one or more of propulsion, steering, and / or braking.

[0057] Vehicles and cameras

[0058] Figure 21 is a front elevation view of a vehicle 10 equipped with a first camera 250 and a second camera 260 mounted in a fixed relationship on a suitable exterior and / or interior structure of the vehicle 10. The camera 250 can be a camera having any suitable sensor array size (e.g., 320 x 240 sensing elements). In an exemplary embodiment, the image captured by the camera 250 is an eight-bit grayscale image. In an exemplary embodiment disclosed herein, the camera 250 includes a long wave infrared (LWIR) band imager that is sensitive to energy in the thermal infrared wavelength band. The LWIR band spans wavelengths roughly between 8 and 15 microns (μm). Therefore, the LWIR camera 250 can detect and capture thermal radiation emitted by objects and surfaces.

[0059] Thermal images allow visualization of thermal signatures, temperature variations, and heat patterns that are invisible to the human eye and not captured by other imaging sensors. Thermal images can be used for low-light or no-light applications, night vision, surveillance, thermal analysis and remote sensing, as well as applications requiring imaging in challenging atmospheric environments such as smoke, fog, and mist.

[0060] In other exemplary embodiments, the camera 250 may include a near infrared (NIR) imaging sensor. The NIR band is a portion of the electromagnetic spectrum extending from approximately 700 nanometers (nm) to 2500 nm, but the exact boundaries may vary depending on the context and application. The NIR band is adjacent to the visible spectrum, which covers wavelengths visible to the human eye.

[0061] In other embodiments, the camera 250 may include a short wave infrared (SWIR) imaging sensor. SWIR is a portion of the electromagnetic spectrum that spans wavelengths between approximately 700 and approximately 3000 nanometers (nm). SWIR radiation has better atmospheric transmission than the longer wavelength infrared bands. It can penetrate fog, smoke, and mist better than LWIR (long wave infrared) radiation. SWIR imaging is suitable for applications where visibility is limited due to atmospheric interference. SWIR radiation can penetrate water to some extent. This property enables SWIR cameras to detect underwater objects.

[0062] The second camera 260 may be a red, green, blue (RGB) camera. An RGB camera typically has a sensor with three separate sensing elements per pixel, each sensing element being sensitive to one of these primary colors. The second camera 260 may be an RGB camera including a sensor including an array of sensing elements (sensel), such as a complementary metal oxide semiconductor (CMOS), a charge coupled device (CCD), or an array of other suitable light sensing elements. In some embodiments, the camera 260 may be a complementary metal oxide semiconductor (CMOS) sensor that provides a red, green, blue, and white (RGBC) light imager that adds clear (unfiltered) pixels to the RGB filter array. In an exemplary embodiment, the camera 260 may be a 2 megapixel (MP) imager having a resolution of 1980X 1080, having a physical pixel size of three microns, having a focal length between 3.14 millimeters (mm) and 2.42 millimeters, and having a horizontal field of view (HFOV) of 85 degrees to 100 degrees or more.

[0063] The above examples are illustrative only. The present disclosure is not limited by the wavelength bands of the first camera 250 and the second camera 260. Likewise, the present disclosure is not limited to the registration of spectrally or spatially different images. Although the examples provided herein are described in the context of images acquired by a vehicle-borne camera, the disclosed apparatus and methods are not limited to the registration of images captured by a vehicle-borne camera. It should be understood that the image registration apparatus and methods described herein will be found in a variety of practical applications, including maritime applications, search and rescue applications, and augmented reality video games.

[0064] In the disclosed example, cameras 250 and 260 are digital cameras, each of which includes an imaging sensor that includes a 2D array of light sensing elements (sensels) that form a 2D likeness (image) of a scene. A digital camera can be modeled by a 3X4 camera projection matrix and a 4x4 matrix projection that encode the camera's intrinsic parameters. The intrinsic parameters specify the camera's internal geometric properties, by which point coordinates in a 3D world coordinate system are mapped to point coordinates in a 3D camera view coordinate system. The origin of the camera view coordinate system is in the image plane (the plane of its sensor array), and the Z axis is perpendicular to the image plane.

[0065] The intrinsic parameters are the transformation or matrix that converts point coordinates from the camera 3D coordinate system to the camera's 2D pixel coordinate system. An array of pixel intensity values ​​defines an image. Each pixel intensity value has a position in the array. The position is specified by a pixel coordinate. The pixel coordinates of an image can be determined by dividing the image coordinates by the pixel width and height, which are camera parameters specified in meters per pixel.

[0066] The matrix used to transform the image from one of the cameras 250, 260 into the 2D pixel coordinate system of the other camera 250, 260 may be determined when the vehicle 10 is manufactured or when the cameras 250, 260 are installed in the vehicle 10. The matrix used to transform the image may be determined by determining the same set of feature points from the scene viewed by both cameras 250, 260. Determining the same set of feature points may be aided by placing one or more fiducial markers (e.g., a checkerboard pattern) at various locations in the scene that may be viewed by both cameras 250, 260.

[0067] The matrix used to transform one of the cameras 250, 260 to the 2D pixel coordinate system of the other camera 250, 260 is determined by first determining the matrices T1, T2 that transform the images I1, I2 into the rectified space. The transformed images I1, I2 are indicated by T1(I1) and T2(I2). Transforming the images I1, I2 into the rectified space transforms feature points indicating the same position in the scene to the same horizontal line in the rectified space. After transforming the images I1, I2 into the rectified space, a warp interpolation matrix W is determined to warp the pixels of one of the images (e.g., image I1) to best match the pixels of the other image (e.g., I2). For example, the warp interpolation matrix W can be determined based on moving matching feature points that appear on the same horizontal line to the same position (e.g., ).

[0068] When the matrix determined by the matrix T1 and W is determined, the image I1 acquired by the first camera 250 may be transformed into a 2D pixel coordinate system of the second image I2 acquired by the second camera 260 by the following formula:

[0069]

[0070] in is image I1 transformed into the 2D pixel coordinate system of image I2, and is the inverse transformation from the rectified space to the 2D pixel coordinate system.

[0071] The transformation from formula (1) is a series of linear matrix operations and can be implemented as a lookup table (LUT) operation. LUT in this article means a data structure, such as an array, that receives a first value as an input and uses the first value as an index to select a second value as the output of the LUT. By calculating formula (1) for each pixel position in a first image (e.g., image I1), it can be determined that the pixel position in image I1 will be transformed into image At runtime, the pixel values ​​from the first image I1 can be transferred to the image Once the LUT is determined, performing image transformations using the LUT can be more efficient and use fewer computational resources than computing matrix operations.

[0072] exist Figure 2 In an exemplary embodiment of the present invention, camera 250 is positioned on the top surface 11 of vehicle 10, while camera 260 may be disposed within vehicle 10, such as behind or near a rearview mirror. In any given embodiment, the relative positions of cameras 250 and 260 may vary. When physical constraints are considered in the installation, camera 250 and camera 260 may be separated by a distance that may produce an undesirable geometric parallax effect. Parallax is the displacement or difference in the apparent position of an object viewed along two different lines of sight. The greater the distance that the cameras are separated from each other, the greater the geometric parallax.

[0073] The degree to which parallax affects the apparent position of an object in a captured image is also determined by the distance that separates the object from the camera at the time of capture. Generally speaking, objects that are closer to the camera at the time of image capture will show larger differences in apparent position in the captured image. This effect can be seen in Figure 2 , because the lines of sight d1 and d2 are not parallel but converge with distance from the cameras 250, 260. The close relationship between parallax and the distance of an object relative to the camera poses a challenge when aligning two-dimensional (2D) images, because 2D images provide little depth information to determine the distance of an object from the camera.

[0074] Image Registration Equipment

[0075] Figure 3 3 is a block diagram of an image registration apparatus 300 for real-time blind registration of spatially and spectrally different video images captured by the above-mentioned spectrally and spatially cameras 250 and 260. The apparatus 300 includes a processor 310, a memory 320 and an image registration device 370.

[0076] The image registration device 370 includes LUTs 330, 317, and 315. The LUTs 330, 317, and 315 are configured by an offline calibration process in which the processor 310 executes processor executable instructions to encode the corresponding image registration function as a functional relationship established by the corresponding output (or vice versa) and input pixel coordinate mapping map. The LUTs 330, 317, and 315 are configured by loading them with their corresponding output and input pixel coordinate mapping maps.

[0077] In the present disclosure, an 'offline process' refers to a process that is performed only when the vehicle is not in regular operational use (e.g., during vehicle production or maintenance, or during installation of the cameras 250, 260, or during installation of the image registration device 300, or the like). In an offline image registration process, the cameras 250, 260 only capture and register calibration images. The calibration image is an image of a special object (e.g., a calibration target having a special pattern such as a checkerboard pattern).

[0078] Once the offline calibration process is complete, the functional relationships encoded in the pixel coordinate map are static, i.e., fixed. They do not change during the online registration process. The image registration device 370 can then be used 'online' to register non-calibrated images captured by the cameras 250, 260 (or their equivalents). The non-calibrated images are images of scenes in the environment surrounding the vehicle 10 and are captured by the cameras 250, 260 during regular use (online use) of the vehicle 10.

[0079] Cameras 250 and 260 may be used in the offline calibration process as well as the online image registration process as long as a fixed spatial relationship between camera 250 and camera 260 is maintained. The offline calibration process may have to be repeated if one or both of cameras 250, 260 are replaced. The cameras 250, 260 used for calibration may be different cameras from those used for online image registration, as long as the cameras used online have the same imaging modality as the cameras used offline, have the same fixed spatial relationship as the cameras used offline, and have the same intrinsic camera parameters and extrinsic camera parameters as the cameras used offline.

[0080] As described above, LUT 330, LUT 317, and LUT 315 are static devices. Their respective pixel maps do not change during online use. During the online registration process, they provide the functional relationship encoded in their maps for each non-calibrated image they receive at their respective inputs, regardless of the non-calibrated image content. Thus, processor 310 performs indexing operations based on the maps to register the non-calibrated images based on the encoded functions.

[0081] In this sense, the online image registration process is a 'blind' image registration process. The online process is performed entirely in the absence of feedback or knowledge of the features or other content of the image I1 received at its input and in the absence of knowledge of the features or content of the image I2 captured by the camera 260 to which the image I1 is to be registered.

[0082] The image registration device 370 includes a distortion correction transform lookup table (LUT) 317, a resizing LUT 330, and a resampling LUT 315. LUTs 330, 317, 315 are static devices. They all perform their respective functions in the same way for each image they receive, regardless of the image content. They are also blind because they perform their respective functions without feedback or knowledge of the features or other content of the images they receive. LUTs 330, 317, 315 can be implemented in software as a data structure in a memory, or in hardware as a solid-state memory device including, for example, a memory cell array having address lines configured as input terminals and data lines configured as output terminals. Suitable structures include field programmable gate arrays (FPGAs), graphics processing units (GPUs), application specific integrated circuits ("ASICs"), read-only memories (ROMs), electrically programmable read-only memories (EPROMs), random access memories (RAMs), erasable PROMs (EEPROMs), and similar devices.

[0083] The corresponding mapping encodes the corresponding corresponding function by functionally relating inputs to outputs (eg, input image pixels to output image pixels). The mapping is generated in an offline calibration procedure (see the method illustrated in FIG. 8 ).

[0084] The resizing LUT 330 is configured with a resizing map that is loaded into the LUT 330 during the offline calibration process. The resizing map encodes a resizing function that is executed by the processor 310 during the offline calibration process to generate a resized calibration image. The resizing map encodes the resizing function as a functional relationship established between corresponding source image pixel coordinates and corresponding corresponding output pixel coordinates.

[0085] The functional relationship is preserved when the processor 310 configures the resizing LUT 330 with the resizing map. Once the LUT 330 is configured with the resizing map, the mapping does not change regardless of any changes to the input image.

[0086] In the online image registration process, the resizing LUT 330 uses the mapping map generated in the offline process to map the corresponding output pixel coordinates of the resized source image to the corresponding corresponding input pixel coordinates of the source image, thereby preserving the functional relationship encoded in the mapping map. In the online process, the resizing map does not change as the source image changes. However, when the source image changes in the online process, the pixel intensity values ​​of the resized source image do change. The pixel intensity values ​​of the resized source image constructed in the online process correspond to the pixel intensity values ​​of the image I1 captured in the online process, and do not correspond to the pixel intensity values ​​of the calibration image I11 used to encode the functional relationship in the resizing map.

[0087] Therefore, in the online image registration process, the processor 310 determines the input pixel coordinates corresponding to the output pixel coordinates provided by the resizing map of the LUT 330. The processor 310 copies the pixel intensity values ​​from the input pixel coordinates of the image I1 to the corresponding corresponding output pixel coordinates provided by the resizing map according to the input pixel coordinates determined by the processor 310 with reference to the resizing map, thereby constructing a resized image I1".

[0088] This relieves the processor of the processing burden of executing processor executable instructions to resize images captured during the online image registration process.

[0089] The distortion correction LUT 317 is also configured to correct distortion using the distortion correction map during the offline calibration process. During the online image registration process, the distortion correction LUT 317 maps corresponding pixel coordinates of the source image to corresponding corresponding output pixel coordinates provided by the map, thereby establishing a functional relationship that defines a distortion-corrected, resized source image, regardless of the image content of the source image.

[0090] Therefore, in the online image registration process, the processor 310 determines the output pixel coordinates corresponding to the input pixel coordinates provided by the distortion correction map of the LUT 317. The processor 310 copies the pixel intensity values ​​from the input pixel coordinates of the image I1' provided by the resizing map to the corresponding corresponding output pixel coordinates provided by the resizing map, thereby constructing the distortion corrected, resized image I1".

[0091] In some embodiments, the resizing LUT 330 and the distortion correction LUT 317 may be arranged such that the mapped image I1 is distortion corrected prior to resizing.

[0092] exist Figure 3In the example of FIG. 3 , during the offline calibration process, the resampling LUT 315 is also configured with the resampling map. During the online image registration process, the resampling LUT 315 maps the corresponding pixel coordinates of the source image to the corresponding corresponding output pixel coordinates provided by the sampling map, thereby establishing a functional relationship that defines the resampled, distortion-corrected, and resized source image, regardless of the image content of the source image.

[0093] Therefore, in the online image registration process, the processor 310 determines the output pixel coordinates corresponding to the input pixel coordinates provided by the resampling map of the resampling LUT 315. The processor 310 copies the pixel intensity values ​​from the input pixel coordinates of the image I1" provided by the distortion correction map of the LUT 317 to the corresponding corresponding output pixel coordinates provided by the resampling map, thereby constructing the resampled, distortion corrected, resized image I1'".

[0094] Once the LUTs constituting the image registration device 370 are loaded with their respective maps, they execute the functions encoded in the maps in the same way for each source image. In this sense, the image registration device 370 performs a blind registration process. In other words, the registration is performed without knowledge of the features of the image I1, without mapping the features, and without any other knowledge of the content of the images I1 and / or I2. In fact, the image registration device 370 can register the image I1 (or the image sequence I1) with the image I2 (or the image sequence I2) without processing any I2 images.

[0095] The pixel coordinate mapping operation may involve interpolation of at least some pixel intensity values.Thus, whenever an input image is mapped to an output image, some degree of fidelity loss may occur in the output image.

[0096] Therefore, in an alternative embodiment of the image registration device 370, the maps of the LUTs 330, 317, 315 are combined into a single map, and the single map is loaded into one LUT. This reduces the number of interpolations from three to no more than one. The single LUT configured with the combined map provides a higher quality image and incurs less computation than would be incurred by performing three separate mapping operations.

[0097] In addition to the image registration device 370 , the apparatus 300 also includes a memory 320 , which may be a non-transitory computer-readable storage medium storing processor-executable instructions that configure the processor 310 to perform the functions, processes, and methods described herein.

[0098] In an exemplary vehicle embodiment, the processor 310 may be included in the network 114 ( Figure 1 ) is communicatively coupled to the memory 320 and the image registration device 370 by a common electrical communication bus in the display 106. In other embodiments, the device 300 may be provided as a stand-alone device that may be disposed within a suitable housing equipped with input / output ports for cooperative intercommunication between components of the device 300 (including, for example, the memory 320) and components of the vehicle 10 that receive images from the frame buffers 350 and 360 and provide multispectral images for rendering on the display device 378. In those embodiments, the electrical communication bus 106 may include printed circuit board interconnects and / or wired or wireless interconnects between components of the device 300 enclosed by the housing.

[0099] In some embodiments, one or more components and / or functions of the device 300 may be distributed among components of the vehicle system 100 and configured for cooperative intercommunication via the network 114. The components of the device 300 may include communication ports for sending and receiving data and / or commands to and from external devices (such as the display device 378, the frame buffers 350, 360, the components 110 of the vehicle system 100, and the cameras 250 and 260) via the network 114.

[0100] Frame buffers 350 and 360 include memory units arranged to store arrays of respective intensity values ​​as frames, the arrays of respective intensity values ​​corresponding to respective sensing elements (sensels) of cameras 250, 260, respectively. For example, as described above, the pixels of the array corresponding to camera 250 may visually represent infrared scene energy by eight-bit Gray code values. The pixel position in the array is specified by an x, y coordinate in the camera coordinate system.

[0101] As used herein, the term 'frame buffer' refers to a physical area of ​​memory, such as random access memory (RAM), configured to store arrays of pixel intensity values ​​provided by corresponding cameras. The term 'buffering' refers to the use of memory to temporarily store or 'buffer' the arrays of pixel intensity values ​​as the corresponding cameras provide the arrays of pixel intensity values ​​in real time. Processor 310 is configured to read frame buffers 350 and 360 to obtain successive corresponding frames of pixel values ​​or rasters provided by cameras 250 and 260, respectively, for example as a stream of video frames.

[0102] In an exemplary use of the device 300, the image registration device 370 receives a frame of a non-calibrated image provided by whichever camera has the lowest resolution among the cameras 250 and 260. In the exemplary embodiment described herein, the camera with the lowest resolution is the camera 250. Therefore, the frame stored in the frame buffer 350 of the camera 250 is provided to the image registration device 370. The image registration device 370 operates on the input frame without reference to the image captured by the camera 260 and stored in the frame buffer 360.

[0103] In some embodiments, the device 300 also includes a blending LUT 313. The blending LUT 313 may be loaded with a blending map encoding a blending function that fuses the registered image I1'' to the corresponding image I2 to form a multispectral image. The output of the blending LUT 313 may be a stream of multispectral images I3.

[0104] In some embodiments, the image registration device 370 provides real-time image processing capabilities to the device 300. In the context of the present disclosure, 'real-time' means within the frame refresh rate, or synchronized or corresponding to the frame refresh rate of the two imagers. Therefore, the processor 310 can register the images within at least one frame refresh cycle of the cameras 250, 260.

[0105] The real-time registration capability is achieved at least in part by encoding the transformations and functions in the mapping and loading the mapping into the LUTs 330, 315, and 317. As described above, encoding the mapping functions in the LUTs relieves the processor 310 from the burden of executing processor-executable instructions to perform these functions. The processor 310 can register the images by providing the frames from the frame buffer 350 as input to the LUTs. The LUTs map the input frames to the output frames to achieve the same function results as those provided by the processor 310 executing the processor-executable instructions to perform the mapping functions on the input frames.

[0106] In an exemplary embodiment, camera 250 provides an array (raster) to frame buffer 350 at a frame refresh rate of 30 Hz or 30 frames per second. The raster corresponds to an array of pixel values ​​provided by the sensing element array of the camera's sensor. In an exemplary embodiment, frame buffer 350 is refreshed synchronously with the refresh of frame buffer 360. In an exemplary embodiment, the refresh rate of frame buffers 350 and 360 is 30 Hz or 30 frames per second. In an exemplary embodiment, frame buffer 350 and frame buffer 360 are synchronized to refresh within + / - 17 milliseconds (ms) of each other.

[0107] In some embodiments, the display device 378 is a transparent display (TD) that includes a portion of the windshield 295 of the vehicle 10 (in Figure 2). For example, the display device 378 may include an augmented reality (AR) display or a heads-up display (HUD) that projects information onto the windshield 295 for viewing by vehicle occupants. In other embodiments, the display device 378 may include a display device disposed in, for example, a console of the vehicle 10.

[0108] Resampling LUT

[0109] Figure 4 yes Figure 3 315 ′ is a block diagram of an alternative embodiment of the resampling LUT 315 shown in FIG. Figure 4 In the example of FIG. 4 , the resampling LUT 315′ includes a correction LUT 442, a distortion LUT 444, and an inverse correction LUT 446. The correction LUT 442 is loaded with Fig.13A The correction map generated at blocks 1319 and 1320 of the offline method 1300 shown in FIG. The correction map configures the correction LUT 442 to map the Cartesian coordinates of the pixels of the distortion-corrected, resized image I1″ provided by the distortion correction LUT 317 to polar coordinates, thereby providing the corrected, distortion-corrected, resized image I1″ at the output of the correction LUT. R .

[0110] The distortion LUT 444 is loaded with Fig.13A 1332, 1334 in the offline calibration method 1300 shown in FIG. The warp map encodes a best fit polynomial function that is applied during the offline calibration method described below to determine a pixel offset for each pixel of the rectified first calibration image to warp the first calibration image to the second calibration image. The warp map generated in the offline process configures the LUT 444 to apply the same pixel offsets to the rectified, distortion corrected, resized non-calibrated image I1 in the online process. R Thus, the warp LUT 444 is configured to provide a warped, rectified, distortion-corrected, resized image at its output. I1W .

[0111] The inverse rectification LUT 446 is loaded with an inverse transformation map that encodes the inverse rectification transform T2' generated at 1338, 1340 in the offline calibration method 1300 shown in Figure 13. The inverse transformation map configures the inverse rectification LUT 446 to convert the polar pixel coordinates of the warped, rectified, distortion-corrected, resized image I1W' to the Cartesian coordinate system of the image I2 captured by the camera 260, thereby providing a warped, distortion-corrected, resized image I1'' registered with the image I2.

[0112] Selectable resampling LUT

[0113] The fidelity is determined by the particular calibration data selected to determine the resampled map, as well as by the distance to the calibration target and the location of the target in the field of view (FOV) of the cameras 250, 260. The non-calibrated images captured by the cameras 250, 260 will be best registered when the distance separating the vehicle 10 from the scene objects is approximately the same as the distance separating the cameras from the calibration targets in the offline registration process. This means that there may be perceptible differences in the registration errors in the images registered in the online process, depending on the distance separating the vehicle 10 from the various scene objects. To compensate for this effect in an exemplary embodiment, the device 300 may be configured as described in the example embodiment. Figure 5 More than one resampling LUT is shown.

[0114] Figure 5 is a block diagram of another exemplary embodiment of the apparatus 300. Figure 5 In the exemplary configuration shown in , the device 300 also includes at least one additional resampling LUT, in this case three additional resampling LUTs: a resampling LUT d1 indicated at 404 , a resampling LUT d2 indicated at 406 , and a resampling LUT d3 indicated at 408 .

[0115] The device 300 also includes a switch 422 coupled to the processor 310. Other embodiments may have two resampling LUTs, or more than three resampling LUTs. The switch 422 may be, for example, a single-pole triple-throw switch or equivalent, which is operable to indicate to the processor 310 a selection of one of the LUT d1 404, LUT d2 406, and LUT d3 408.

[0116] In response to receiving the selection indication from the switch 422, the processor 310 activates a corresponding one of the switches S1, S2, and S3 to couple the output of the distortion correction LUT 317 to a corresponding one of the resampling LUTs 404, 406, and 408. The selected resampling LUT includes the resampling LUT 315″ of the image registration device 370. The output of the selected resampling LUT is the resampled, distortion-corrected, resized image I1″′, as shown. Figure 3 as shown in .

[0117] In some embodiments, switch 422 includes a user interface through which a user (e.g., a vehicle occupant) can select one of the multiple resampling LUTs based on user criteria (such as a perception of average distance or a user preference that one LUT performs better than another).

[0118] In some embodiments, the processor 310 is configured to obtain the distance measurement sensor 318 (in Figure 3 The processor 310 is configured to calculate the average distance between the vehicle 10 and objects in the environment surrounding the vehicle 10 based on the data received from the ranging sensor 318.

[0119] The processor 310 is configured to select one of the resampling LUTs 404, 406, and 408 based on the distance information to perform a resampling function on the resampling LUT 315. For example, when the average distance separating the vehicle 10 from objects in the environment surrounding the vehicle 10 is about 5 meters or less, the resampling LUT d1 404 may be selected. For example, when the average distance separating the vehicle 10 from objects in the environment surrounding the vehicle 10 is between about 5 meters and about 15 meters, the resampling LUT d2 406 may be selected. For example, when the average distance separating the vehicle 10 from objects in the environment surrounding the vehicle 10 is greater than about 15 meters, the resampling LUT d3 408 may be selected.

[0120] By comparing the sensed average distance from the vehicle 10 to objects in the environment with different distances and / or angles corresponding to corresponding ones of the resampling LUTs, a LUT corresponding to the current environment of the vehicle 10 may be selected from a plurality of different resampling LUTs. The LUT having the closest match of the distances and / or angles to the sensed distances and / or angles may be selected from the plurality of resampling LUTs to perform the resampling block of the method 800. The LUT selection may be automatic (i.e., programmed) or manual (i.e., selected by a user).

[0121] The sensed distance to an object in the environment may be determined by the ranging sensor 318 ( Figure 3 ), which may be a LIDAR sensor of the vehicle 10 perception system. The sensed distance may be provided by other vehicle sensors, such as those implemented by a GPS system, a cellular network, etc. For example, a GPS system may determine the location of the vehicle 10, and the processor 310 may use available mapping data to locate the vehicle 10. The mapping data may be stored on the vehicle 10, or the processor 310 may access the vehicle 10 via the WAN 116 ( Figure 1 ) obtains mapping data from a cloud-based server. The processor 310 may determine which of the three LUTs to select based on the environment (eg, urban, rural) indicated by the mapping data.

[0122] In other embodiments, a monocular depth estimation using machine learning and unsupervised training (e.g., Monocular Depth 2) may be used to generate a pixel-by-pixel range map for images acquired by either or both of the cameras 250, 260. The distance information collected by the monocular depth algorithm may be provided to the processor 310. The processor 310 may be configured to select one of a plurality of LUTs for use after taking into account the distance information.

[0123] Correction of display gamma curve

[0124] Figure 6 yes Figure 3 A block diagram of another exemplary embodiment of the device shown in FIG. Figure 6 In an embodiment, the image registration device 370' of the apparatus 300 includes an image registration LUT 670. The image registration LUT 670 is configured with a single image registration map, the single image registration map being composed of Figure 3 6. The image registration LUT 670 may be used to provide a mapping provided by the LUTs 330, 317, and 315 in the exemplary image registration apparatus 370 shown in . The image registration LUT may perform a pixel intensity value interpolation process when copying pixel intensity values ​​from pixel coordinates of the input non-calibrated image I1 to output pixel coordinates to provide a registered output image I1''. Because only one pixel coordinate mapping is performed to align the non-calibrated input image I1 with the non-calibrated image I2, the pixel intensity value interpolation process is performed no more than once. The registered image provided by the LUT 670 may suffer less image quality degradation compared to an embodiment in which more than one pixel intensity value interpolation process is performed using more than one LUT mapping to align images.

[0125] The apparatus 300 further comprises an encoder 340. The display device 378 comprises a corresponding decoder 345 comprising a switch 373 and two gamma curve correction LUTs 371,372.

[0126] The encoder 340 compresses the multispectral image streams provided at the output of the hybrid LUT 313 so that they use less vehicle network bandwidth. The encoder 340 encodes the multispectral video stream in real time according to the H.264 video coding standard. In other embodiments, the encoder 340 may encode the video stream according to other standards (e.g., H.265, VP9, ​​and AV1).

[0127] The encoder 340 is configured to identify regions of the video image that contain IR spectral content and regions of the video image that contain RGB spectral content. The encoder 340 encodes the IR spectral content at a higher encoding bit rate than it encodes the RGB spectral content. When the encoded image stream is decoded by the decoder 345 of the display device 378, the IR region is rendered at a higher resolution than the RGB region.

[0128] In some embodiments, the encoder 340 includes an ambient light sensor. In other embodiments, the encoder 340 is configured to receive an ambient lighting signal from one or both of the cameras 250, 260. In either method, the encoder 340 is configured to encode a region of a multispectral image stream representing IR imager spectral content at a different encoding bit rate than the encoding bit rate used to encode a region of a multispectral image stream representing RGB imager spectral content. Which spectral content is encoded at a higher rate is based on ambient lighting conditions. In an exemplary embodiment, under high ambient light conditions, the encoder 340 uses a higher bit stream to encode a region of an image representing IR spectral content at a higher bit rate than the bit rate at which the region of the image representing RGB spectral content is encoded. This is because RGB sensors perform better under high ambient lighting conditions than IR sensors. Under low ambient lighting conditions, the encoder 340 is configured to encode a region representing RGB spectral content at a higher bit rate than the bit rate at which the region representing IR spectral content is encoded. This is because RGB sensors and IR sensors do not perform under low ambient lighting conditions.

[0129] In another exemplary embodiment, the encoder 340 is configured to encode a video stream captured during high ambient light conditions at a lower bit rate than the bit rate it uses to encode a video stream captured during low ambient lighting conditions. In another exemplary embodiment, the encoder 340 is configured to control the bit rate based on the time of day. For example, the encoder 340 calculates the overall average bit rate over a twenty-four hour period as the bit rate budget. The encoder 340 is configured to allocate the bit rate budget on an hourly basis. The bit rate during daytime hours is lower than the bit rate during nighttime hours, while the average bit rate over the twenty-four hour period remains constant.

[0130] In one embodiment, the apparatus 300 is configured to cooperate with the display device 378 to apply a first gamma correction to the IR image spectral content and a second gamma correction to the RGB image spectral content to enhance visibility of the IR image spectral content in the multispectral image.

[0131] The gamma adjustment LUT is provided in the display device 378 and is arranged to cooperate with a display processor (not separately depicted) to adjust the display gamma according to the gamma adjustment specified by the gamma adjustment LUT.

[0132] The processor 310 is configured to provide a gamma LUT identifier corresponding to the gamma adjustment LUT to the encoder 340. The encoder 340 is configured to encode the gamma LUT identifier in the multispectral image frame stream provided to the decoder 345 of the display device 378. The display device 378 may be provided with more than one gamma curve correction LUT, for example, a gamma LUT 371 and a gamma LUT 372.

[0133] The decoder 345 decodes the encoded stream, recovers the gamma LUT identifier, and converts the encoded stream of the multispectral image frame into a display format such as HDMI for rendering on a display screen 346 of a display device 378. The decoder 345 provides the recovered gamma curve correction identifier to a processor of the display device 378. The processor of the display device 378 uses the gamma LUT identified by the identifier to select one of the gamma LUT 371 and the gamma LUT 372 to adjust the gamma curve of the display device 378. Therefore, the IR portion of the multispectral image is rendered according to a gamma different from the gamma used to render the RGB portion of the multispectral image.

[0134] Real-time online image registration

[0135] Figure 7 is a flow chart of an online method 700 for real-time registration of (non-calibrated) images captured by spatially and spectrally different cameras 250, 260. Figure 7 In the exemplary method shown in , camera 250 is a LWIR camera having a lower pixel resolution than camera 260. In this example, camera 260 is an RGB camera. In the exemplary application, cameras 250, 260 (in Figure 2 and Figure 3 ) is mounted to a vehicle, such as vehicle 10, as shown in Figure 1 and Figure 2The cameras are positioned and oriented to simultaneously capture non-calibrated images of the same scene within the cameras' respective fields of view (FoV) of the environment around the vehicle 10 during normal driving operation of the vehicle 10. The camera 250 provides captured images I1 of frames including pixel intensity values ​​to the frame buffer 350, where the images are buffered for processing according to the method 700.

[0136] At block 702, the processor 310, for example, retrieves a frame from the frame buffer 350 ( Figure 3 ) receives a first image I1 of a first frame including pixel values.

[0137] At block 704, processor 310 provides pixel coordinates of image I1 to resizing LUT 330. Resizing LUT 330 is configured with a resizing map that encodes a resizing function that scales the input image according to the following relationship:

[0138] S(x, y) = IFoV calibration image I12 / IFoV calibration image I11 (2)

[0139] Where the instantaneous field of view (IFoV) of the calibration image is given by the pixel pitch (center-to-center distance between sensing elements on the camera imager) / focal length. This is equivalent to the center-to-center distance of the squares on the calibration target / the viewing distance of the imaged calibration target from the camera. Therefore, the IFoV of the resized output image I1' will be the same as the IFOV of the corresponding image I2 provided by the camera 260.

[0140] The resizing map relates output pixel coordinates defining a resized calibration image captured by the camera 250 during an offline calibration process to corresponding source input pixel coordinates.

[0141] At block 706 , the processor 310 constructs a resized image I1 ′ based on the pixel intensity values ​​of the image I1 and the output pixel coordinates provided by the resizing LUT 330 .

[0142] At block 708, the processor 310 provides the pixel coordinates of the resized image I1' to the distortion correction LUT 317. The distortion correction LUT 317 is configured with a distortion correction map that relates the pixel coordinates of the source image (i.e., the resized image I1') to the pixel coordinates of the distortion-corrected, resized image I1".

[0143] At block 710 , the processor 310 constructs a distortion-corrected, resized image I1 ″ based on the pixel intensity values ​​of the resized image I1 ′ and the distortion correction map provided by the LUT 330 .

[0144] At block 712, the processor 310 provides pixel coordinates of the distortion-corrected, resized image I1″ to the resampling LUT 315. The resampling LUT 315 is configured with a single resampling map that encodes the corrective transform, the warping function, and the inverse corrective transform.

[0145] At box 714, the processor 310 constructs a resampled, distortion-corrected, resized image I'" based on the pixel intensity values ​​of the distortion-corrected, resized image I". The resampled, distortion-corrected, resized image I1'" is registered with the image I2.

[0146] At box 716 , the processor 310 provides the resampled, distortion-corrected, resized image I1 ′″ at the output of the registration device 370 , and the method ends.

[0147] Fig. 8A is a flow chart of a method 800 for constructing a resized image I1 ′ representing block 706 of method 700 .

[0148] At block 802 , the processor 310 provides pixel coordinates of an input image I1 to the resizing LUT 330 .

[0149] At block 804 , for the respective output pixel coordinates given by the resizing LUT 330 , the processor 310 identifies the corresponding respective pixel coordinates of the image I1 .

[0150] At block 806 , processor 310 copies the corresponding pixel intensity values ​​from the corresponding corresponding pixel coordinates of I1 to the corresponding corresponding output pixel coordinates given by resizing LUT 330 , thereby constructing a resized image I1 ′, and the method ends.

[0151] The processor 310 may also be configured to correct distortion in the resized image I1′ to provide a distortion-corrected, resized image I1″, such as Figure 8B as shown in the flow chart.

[0152] Figure 8B is a flow chart of a method 850 for constructing a distortion-corrected, resized image I1″ for use in block 710 of method 700 .

[0153] At block 814 , the processor 310 provides the pixel coordinates of the resized input image I1 ′ to the distortion correction LUT 317 .

[0154] At block 816 , for respective pixel coordinates of the resized image I1 ′, the processor 310 determines corresponding respective output pixel coordinates provided by the distortion correction map of the distortion correction LUT 317 .

[0155] At block 818, processor 310 copies the pixel intensity values ​​from the corresponding input pixel coordinates of resized image I1 ′ to the corresponding corresponding output pixel coordinates provided by the distortion correction map of LUT 317, and the method ends.

[0156] After performing method 850, the intensity values ​​of the output pixel coordinates comprise the distortion-corrected, resized input image I1'''.

[0157] Fig. 9 is a flow chart of a method 900 for resampling a distortion-corrected, resized image I1″.

[0158] At block 902, the processor 310 provides pixel coordinates of the distortion-corrected, resized image I1″ to the resampling LUT 315. The resampling LUT 315 is configured with a resampling map generated in an offline process described below, and the resampling LUT 315 is configured with a resampling map in the same offline calibration process.

[0159] At block 904, for respective pixel coordinates of the distortion-corrected, resized image I1″, the processor 310 determines the corresponding respective output pixel coordinates given by the resampling map.

[0160] At box 906, the processor 310 copies the corresponding pixel intensity values ​​from the corresponding pixel coordinates of the distortion-corrected, resized image I1" to the corresponding corresponding output pixel coordinates provided by the resampling LUT 315, thereby constructing a resampled, distortion-corrected, resized image I1'". At this box of the method, the resampled, distortion-corrected, resized image I1'" is aligned with the image I2 captured by the camera 260. The method can be performed in real time online without processing any I2 images.

[0161] At block 908 , the processor 310 provides the resampled, distortion-corrected, resized image I1′″ at the output of the image registration device 370 , and the method ends.

[0162] Fig.10 Is used by Fig.13A and Fig. 13B Flowchart of online registration method 1000 performed using a single registration LUT configured with an offline calibration method (e.g., blocks 1346, 1348).

[0163] At block 1002, processor 310 receives image I1 and provides pixel coordinates of image I1 to a registration LUT (see, e.g., Figure 6 The registration LUT is configured with a registration map that provides a mapping of output pixel coordinates established by a registered calibration image generated in an offline calibration process to pixel coordinates of a source image to be registered (eg, source image I1).

[0164] At block 1004 , for the corresponding corresponding output coordinates provided by the registration LUT, the processor 310 determines the corresponding corresponding input pixel coordinates of the image I1 by looking up the corresponding pixel coordinates in the LUT.

[0165] At block 1006 , for corresponding pixel coordinates of the first image, the processor 310 copies the corresponding corresponding pixel intensity values ​​to the corresponding corresponding output pixel coordinates provided by the registration LUT, thereby constructing a registered image I1′″.

[0166] At block 1008 , processor 310 provides the registered image I1 ′″ to an output of device 300 , and the method ends.

[0167] Offline calibration process

[0168] Figure 11 to Figure 1 3 is a flow chart of an offline calibration method executed to configure the image registration device 370 to perform online image registration. The processor 310 uses the cameras 250 and 260 to perform the offline image registration method to capture a first calibration image sequence I11 and a second calibration image sequence I12. The calibration image is an image of a calibration target (e.g., a structure showing black and white squares in a checkerboard pattern) (see, e.g., FIG. 14A to FIG. 14D ). A checkerboard pattern is used for calibration because its regular grid structure provides a unique feature that can be easily detected and matched in the captured image.

[0169] The black and white squares of the checkerboard calibration target have corners (feature points) with known geometric relationships. Processor 310 detects the feature points in calibration images I11 and I12 and establishes correspondences in calibration images I11 and I12 based on the feature points. Processor 310 can estimate intrinsic parameters and extrinsic parameters of cameras 250 and 260 based on the correspondences. Intrinsic camera modes and extrinsic camera modes are used, for example, to determine the size adjustment map used to generate the resizing map described below with respect to FIG. 8 and to generate lens distortion corrections to generate the resizing map described below with respect to FIG. Fig. 9 Describes the resizing factor of the distortion correction map.

[0170] In an exemplary embodiment of the offline calibration process, camera 250 and camera 260 are mounted in a fixed relationship to each other on a vehicle (eg, Figure 1 and Figure 2 The vehicle 10 shown in FIG. 10 is used to perform the offline calibration process. The vehicle used to perform the offline calibration process and the camera can be used to perform the calibration according to FIG. Figure 7 An example of an online registration method to register non-calibrated images of the same car and camera.

[0171] The vehicle used to capture and process the calibration images I11, I12 in the offline process described herein is not necessarily a car. Suitable vehicles include various types of vehicles. For example, the vehicle can be any movable platform or cart on which the cameras 250 and 260 are mounted in a fixed relationship to perform the offline calibration process. After the calibration is performed using a movable platform in the offline process, the cameras 250 and 260 (or their equivalents) can be mounted on a different vehicle (e.g., a car) in the same fixed relationship.

[0172] Before performing the offline calibration, attach the checkerboard calibration target to a support structure. Suitable support structures include, for example, the interior walls of a building. This would be suitable for embodiments where a cart or movable platform would be used to carry the cameras 250, 260. Suitable support structures may also include the outside of a building. This would be suitable for embodiments where a car would be used to carry the cameras 250 and 260.

[0173] The vehicle carrying cameras 250 and 260 moves from a far side (far) position relative to the calibration target to a near side (near) position relative to the calibration target. The near side distance and far side distance from the calibration target may vary depending on the application. In an exemplary application, the far side (far) distance is about 15 meters and the near side (near) distance is about 5 meters. The vehicle can move toward the calibration target and away from the calibration target, and the movement can be repeated any number of times. When the vehicle moves, cameras 250, 260 capture image sequences I11, I12 of the calibration target, respectively. In one example, cameras 250, 260 capture images at a motion video frame rate (e.g., 30Hz).

[0174] The calibration image sequences I11 and I12 captured in this process may be stored or buffered in a memory and subsequently used by the processor 310 in an offline calibration process to register the calibration images including the calibration image sequence I11 with the calibration images including the calibration image sequence I12. The goal of performing image registration on the images I11 and I12 in the offline calibration process is not to register the images themselves. The goal is to generate a pixel coordinate map that is used to configure the image registration device 370 to register non-calibrated images in a real-time online image registration process, such as Figure 7In the online registration process, the image registration device 370 replaces the computationally complex and time-consuming functions performed by the processor 310 executing processor-executable instructions to register the calibration images with one or more simpler and less complex array indexing operations that can be performed using the image registration device 370 in the real-time online image registration process.

[0175] Fig.11 is a flow chart of an offline method 800 for generating a resizing map for configuring the resizing LUT 330. The method 800 is performed offline. In the method 800, the cameras 250 and 260 capture a sequence of images of a calibration target to provide calibration image sequences I11 and I12, respectively.

[0176] At block 1102 , processor 310 executes instructions to determine intrinsic parameters of camera 250 .

[0177] At block 1104, the processor 310 executes instructions to determine intrinsic parameters of the camera 260. The intrinsic parameters may include field of view (FoV), instantaneous field of view (IFoV), spatial range, spatial resolution, and pixel density or spacing. These may be specified by the camera manufacturer and stored in memory for use by the processor 310 when executing method 800. Alternatively, the intrinsic camera parameters may be determined by performing a camera calibration process and storing the intrinsic parameters prior to performing the offline registration calibration process described herein. Suitable camera calibration processes are well known and are therefore not described in detail herein.

[0178] At block 1106, processor 310 executes instructions to calculate a scaling factor for resizing calibration image I11. The scaling factor is given by:

[0179] S(x,y)=IFoV calibration image I12 / IFoV calibration image I11(1)

[0180] Where the IFoV of the calibration image is given by the pixel pitch (center-to-center distance between sensing elements on the camera imager) / focal length. This is equivalent to the center-to-center distance of the squares on the calibration target / the viewing distance of the imaged calibration target from the camera. At box 1108, the processor 310 executes instructions to resize the input calibration image I11' according to the scaling factor to provide a resized calibration image I11' at the output. The IFoV of the resized calibration image I11' will be the same as the IFoV of the calibration image I12. (See, e.g. FIG. 14A to FIG. 14D ). In exemplary embodiments, 'same' means within about 1%. In other embodiments, 'same' means within about 2%.

[0181] At block 1110, the processor 310 executes the instructions to calculate the input (source) pixel coordinates of each pixel of the output (resized) calibration image I11' based on the scaling factor to generate a resizing map. The resizing map is an output pixel coordinate to input pixel coordinate mapping map that encodes the resizing function performed by the processor 310 by executing the instructions. The mapping map encodes the function as a functional relationship between the output pixel coordinates and the input pixel coordinates.

[0182] At block 1112 , processor 310 executes processor-executable instructions to load the resizing map into resizing LUT 330 , thereby configuring LUT 330 to provide an output image whose pixel coordinates have a functional relationship to pixel coordinates of the input image defined by the resizing map.

[0183] Thereafter, during the online image registration process, the LUT 330 processor 310 (or another suitable processor) may provide the non-calibrated image captured by the camera 250 to the input of the resizing LUT 330. The processor 310 then iterates through each output pixel coordinate of the resizing map and looks up the corresponding input pixel coordinate in the map. The processor 310 identifies the input pixel coordinate in the input non-calibrated image and copies the pixel intensity value from the pixel coordinate to the output pixel coordinate, thereby constructing a resized calibration image I11' at the output of the LUT 330. In this manner, the resizing LUT 330 replaces the resizing function performed by the processor 310 executing processor executable instructions during the offline image registration process with a less complex and faster array indexing operation during the online image registration process.

[0184] Fig.12 is a flow chart of an offline method 1200 for generating a distortion correction map for configuring the distortion correction LUT 317 .

[0185] At block 1202 , processor 310 receives a distortion correction matrix for camera 250 .

[0186] At block 1204 , the processor 310 executes processor-executable instructions to apply the distortion correction matrix to the resized calibration image I11 ′ to provide a distortion-corrected, resized calibration image I11 ″.

[0187] At block 1206 , the processor 310 maps pixel coordinates of the resized calibration image I11 ′ to corresponding respective pixel coordinates of the distortion-corrected, resized calibration image I11 ″ to generate a distortion correction map.

[0188] At block 1208 , the processor 310 loads the distortion correction map into the distortion correction LUT 317 , and the method ends.

[0189] Fig.13A and Fig. 13B A flow chart of an offline method 1300 for generating a resampling map and for configuring the LUT 315 using the resampling map is provided.

[0190] At block 1302 , the processor 310 receives a distortion-corrected, resized calibration image sequence I11 ″.

[0191] At block 1304 , the processor 310 receives a corresponding distortion-corrected, resized calibration image sequence I12 ″.

[0192] At block 1306 , processor 310 detects a calibration target in the distortion-corrected, resized calibration image I11 ″.

[0193] At block 1308 , processor 310 detects a calibration target in the corresponding distortion-corrected, resized calibration image I 12 ″.

[0194] At box 1310, the processor 310 identifies correspondences between points or features of the calibration target detected in the distortion-corrected, resized calibration image sequence I11" and the distortion-corrected, resized calibration image sequence I12". Identifying correspondences is the process of establishing a match between points in a pair of images.

[0195] At block 1312 , the processor 310 calculates a fundamental matrix (FM) based on the correspondences identified in block 1310 .

[0196] At block 1314 , the processor 310 computes a polar rectification transform T1 of the distortion-corrected, resized calibration image sequence I11 ″ based on the FM computed in block 1112 .

[0197] At block 1316 , the processor 310 computes a polar rectification transform T2 of the distortion-corrected, resized calibration image sequence I12 ″ based on the FM computed in block 1112 .

[0198] At block 1318, the processor 310 applies the polar coordinate correction transformation T1 to the corresponding relationships in the distortion-corrected, resized image sequence I11″ to provide a corresponding distortion-corrected, resized, corrected image sequence I11 R .

[0199] At block 1319, the processor 310 may map pixels of the distortion-corrected, resized image I11″ to the distortion-corrected, resized, rectified image I11R At block 1320, the processor 310 may load the T1 polar coordinate correction map into the correction LUT 442 for use in the replacement resampling LUT 315′ ( Figure 4 ).

[0200] At block 1322, the processor 310 applies the polar coordinate correction transform T2 to the corresponding relationships in the distortion-corrected, resized image sequence I12″ to provide a corresponding distortion-corrected, resized, corrected image sequence I12 R The flow chart is in Fig. 13B Continue in.

[0201] At block 1323 , the processor 310 computes the inverse T2 ′ of the polar rectification transform T2 .

[0202] In block 1324 ( Fig. 13B ), the processor 310 identifies the corrected image pair I11 R and I12 R If image I11 R and I12 R are captured simultaneously, ie, they have the same timestamp, then they are a 'pair'. Distant correspondences are correspondence points that have relatively large differences with respect to the image plane.

[0203] At block 1326 , processor 310 calculates the polar column horizontal offsets for the far correspondences.

[0204] At block 1328, the processor 310 generates a signal based on the image I11. R 、I12 R The horizontal offset in the image pair determines the best fitting polynomial function that converts the distortion corrected, resized, rectified image I11 R The pixels of are fitted to the rectified image I12 R The best fitting polynomial function is the function that provides the coefficients of the warp matrix when applied to the distortion-corrected, resized, rectified image I11 R The function gives each distortion-corrected, resized, rectified calibration image I11 when R Each pixel x,y is transformed from its initial position x,y before the distortion to the distortion-corrected, resized, rectified, warped image I11 WThe horizontal displacement of the new position (x+W(x,y),y) in the pixel, where x and y specify the pixel position coordinates and x represents I11 R The warping function consists of a pair of two-dimensional mapping functions u(x,y) and v(x,y) that transform the image I11 into R The pixel position (x,y) of the pixel array is mapped to the image I11 W The new pixel position (u,v) in the pixel array.

[0205] At block 1330, processor 310 applies the best fit polynomial function to image I11 R , to convert image I11 R The pixels of image I12 R The corresponding pixels in are horizontally shifted, thereby making the corrected image I11 R Warped to match the rectified image I12 R Alignment, thereby providing a distortion-corrected, resized, warped, rectified image I11 W .

[0206] In some embodiments, at block 1332, the processor 310 may convert the distortion-corrected, resized, rectified image I11 into a R The pixel coordinates of are mapped to the distorted, rectified image I11 W The pixel coordinates of the image are used to construct the distortion map.

[0207] In some embodiments, at block 1334, the processor 310 may load the warp map into the warp LUT 444 ( Figure 4 Thereafter, without the processor 310 executing processor executable instructions to determine and / or apply the best fit polynomial function in the online image registration process, the processor 310 may use the warp LUT 444 to transform the distortion corrected, resized, rectified image I1 according to the best fit polynomial function. R Mapping to the distortion corrected, resized, rectified, warped non-calibrated image I11 W .

[0208] In any embodiment ( Figure 3 or Figure 4 ), at block 1336, the processor 310 applies T2′ (calculated in block 1322) to the distortion-corrected, resized, warped, rectified calibration image I11 W , to provide a registered calibration image I11 Reg .

[0209] At block 1338, processor 310 may process the distortion-corrected, resized, rectified, warped calibration image I11. W The pixels of the image are mapped to the registered calibration image I11. Reg .

[0210] At block 1340, the processor 310 may load the T2′ map into the inverse correction LUT 446 to include an alternative to the resampling LUT 315′ ( Figure 4 Thereafter, in an online (run-time) application, the processor 310 may provide the LUT 446 with the distortion-corrected, resized, rectified, warped, non-calibrated image I1. W to the distortion corrected, resized, rectified, warped non-calibrated image I1 W The inverse rectification is performed to provide a registered non-calibrated image I1 without the processor 310 executing processor executable instructions for performing the inverse rectification transformation T2′ and without the processor 310 processing any image I2 in the sequence of images I2. REG .

[0211] In either embodiment, at block 1342, processor 310 may map pixel coordinates of the distortion-corrected, resized calibration image I11″ to the registered calibration image I11 REG The pixel coordinates of the image are loaded into the LUT 315 to provide an image resampling map, thereby configuring the LUT 315 for use in online resampling.

[0212] At block 1344, the processor 310 loads the image resampling map into the image resampling LUT 315 ( Figure 3 ). Thereafter, in an online (runtime) application, the processor 310 may perform resampling on the distortion corrected, resized non-calibrated image I1″ by providing the image I1″ to the LUT 315, thereby configuring the LUT to resample the image I1″ when the processor 310 does not execute processor executable instructions for performing the online resampling function and the processor 310 does not process any image in the image sequence I2.

[0213] like Figure 5As shown in , in some embodiments, the device 300 may include more than one resampling LUT. For example, the processor 310 may provide a resampling LUT d1 that is 'focused' on nearby objects, a resampling LUT d2 that is focused on intermediate objects, and a resampling LUT d3 that is focused on far objects. In this context, 'near' may be within about 5 meters from the camera 250, 260. 'Far' may be about 15 meters or more from the camera 250, 260, and 'intermediate' may be anywhere between about 5 meters and 15 meters from the camera 250, 260.

[0214] In order to provide a resampled LUT d1 that is configured to better register nearby objects (e.g., within about 5 meters from the cameras 250, 260) than objects at intermediate or far distances from the cameras 250, 260, the processor 310 selects at least three rectified calibration images in the captured image sequence I11 and three corresponding rectified calibration images in the captured image sequence I12 that were captured when the cameras 250, 260 were within about 5 meters of the calibration target. The processor 310 uses those captured images to perform the method 700.

[0215] Likewise, to provide a resampled LUT d2 configured to register objects at 'intermediate' distances (e.g., distances between approximately 5 meters and 15 meters from the cameras 250, 260), the processor 310 selects calibration images I11 and I12 captured when the cameras 250, 260 were between approximately 5 meters and 15 meters. The processor 310 performs the method 700 on those intermediate images.

[0216] To provide a resampled LUT d3 configured to register objects at 'far' distances (e.g., about 15 meters from cameras 250, 260), processor 310 selects calibration images captured when cameras 250, 260 were farther than 15 meters from the calibration target. Processor 310 performs method 700 on those 'far' images.

[0217] The LUTs configured with the mappings generated by the processor 310 executing the method 1300 are 'fixed' in that they do not change when the cameras 250, 260 are subsequently used to capture non-calibrated images. During subsequent regular (non-calibrated) use of the cameras to register non-calibrated images, the LUTs have no information about the scene or image content in the image frames captured by the cameras 250, 260. Therefore, they do not respond to changes in the scene or image content in any captured frame. In this sense, the LUTs are 'blind', that is, they blindly perform their mapping functions in the same manner for any image they receive without identifying features and without any information about or reference to the image content. Furthermore, in an online (runtime) application, the processor 310 does not process any image I2 in the image sequence I2 to register image I1 with image I2.

[0218] In this exemplary embodiment, cameras 250 and 260 are spaced apart so that their image planes do not coincide. Therefore, in this example, processor 310 applies a polar coordinate correction scheme, and the corrected view space is a polar coordinate system. Therefore, processor 310 maps the pixel coordinates of calibration image I11 from coordinates in a Cartesian coordinate system to polar coordinates. As described above, in embodiments where the cameras are closely spaced apart, the corrected coordinate system can be a planar coordinate system or a cylindrical coordinate system.

[0219] Fig.14A Depicted is the resizing of an exemplary 6x6 pixel array 1402 that includes a calibration image I12 of a calibration target captured by the second camera 260. In this example, one square 1406 corresponds to one pixel. Fig. 14B Depicted is an exemplary 2 x 2 pixel array 1404 comprising an input calibration image I11 of the same calibration target captured by the first camera 250. One square 1408 corresponds to one pixel. Fig. 14C Depicts a clipped area equal to Fig. 14B The cropped calibration image I12′ includes a 4×4 pixel array 1412. Squares 1414 correspond to pixels.

[0220] Fig.14D The output image I11' is depicted after being resized by the processor 310. The resized image I11' comprises a 4X4 array 1422, which is Fig. 14CThe array size of the cropped image I12 in is the same as that of the input image I11. The processor 310 maps each pixel of the input image I11 to four pixels in the output image I11' (one pixel is indicated at pixel 1420). For white checkerboard squares, the processor 310 maps 4 pixels with white intensity values. For black checkerboard squares, the processor 310 maps 4 pixels with black intensity values. Fig.14D The IFoV of the resized image I11' in Fig. 14C The IFoV of the cropped image I12 ′ in is the same.

[0221] The processor 310 creates a mapping between pixels of the input calibration image I11 and pixels of the output calibration image I11' to provide a resizing map. The processor 310 loads the resizing map into the resizing LUT 330. The resizing LUT 330 can then be used to resize the image captured by the first camera 250 in a manner such as Figure 7 The non-calibrated images captured during the online image registration process shown in are resized.

[0222] Fig.15 is a flow chart of a method 1500 for adjusting a display device gamma curve of a multi-spectral video stream. At block 1502, the processor 310 provides a multi-spectral image stream and a gamma LUT identifier to the encoder 340.

[0223] At block 1504 , the processor 310 encodes a stream including a gamma LUT identifier.

[0224] At block 1506 , the encoder 340 provides the encoded stream of the multispectral image including the gamma LUT identifier to the decoder 345 , and the method ends.

[0225] Fig.16 is a flow chart of a method 1600 for adjusting a display device gamma curve for a multi-spectral video stream.

[0226] At block 1602 , the decoder 345 receives an encoded stream of multispectral images from the encoder 340 .

[0227] At block 1604, the decoder 345 decodes the stream and recovers the gamma LUT identifier in the decoded video stream.

[0228] At block 1606, the decoder 345 selects the gamma LUT identified by the gamma LUT identifier. In one embodiment, the decoder 345 selects one of the gamma LUT1 371 and the gamma LUT2 372 based on the identifier.

[0229] At block 1608, the display device 378 adjusts the display device gamma curve according to the selected gamma adjustment LUT, and the method ends.

[0230] At block 1610 , the display device 378 renders the decoded gamma-adjusted video stream on the display screen 346 , and the method ends.

[0231] Computing devices such as those discussed herein typically each include instructions that can be executed by one or more computing devices (such as those identified above) and used to perform the blocks or steps of the processes described above. For example, the process blocks discussed above may be embodied as computer-executable instructions.

[0232] The computer executable instructions may be compiled or interpreted by a computer program created using a variety of programming languages ​​and / or technologies, including, but not limited to, Java, TM , C, C++, VisualBasic, Java Script, Perl, HTML, etc. In general, a processor (e.g., a microprocessor) receives instructions, such as from a memory, a computer-readable medium, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using a variety of computer-readable media. Files in a computing device are typically a collection of data stored on a computer-readable medium such as a storage medium, a random access memory, etc.

[0233] Computer-readable media include any media that participate in providing data (e.g., instructions) that can be read by a computer. Such media can take many forms, including but not limited to non-volatile media, volatile media, etc. Non-volatile media include, for example, optical disks or disks and other persistent memories. Volatile media include dynamic random access memory (DRAM), which usually constitutes main memory. Common forms of computer-readable media include, for example, floppy disks, floppy disks, hard disks, tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media with hole patterns, RAM, PROMs, EPROMs, flash EEPROMs, any other memory chips or cassettes, or any other non-temporary storage media from which a computer can read stored data or instructions.

[0234] In the accompanying drawings, the same reference numerals indicate the same elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring in a certain ordered sequence, such processes may be practiced by performing the steps described in an order different from the order described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the description of the process herein is provided for the purpose of illustrating certain embodiments and should never be interpreted as limiting the claims claimed for protection.

[0235] Therefore, it should be understood that the above description is intended to be illustrative rather than restrictive. After reading the above description, many embodiments and applications other than the examples provided will be apparent to those skilled in the art. The scope of the present invention should not be determined with reference to the above description, but with reference to the appended claims together with the full scope of equivalents to which such claims are assigned. It is contemplated and expected that future developments will occur in the technology discussed herein, and the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that the present invention is capable of modification and variation and is limited only by the appended claims.

[0236] All terms used in the claims are intended to be given their broadest reasonable interpretations and their ordinary meanings as understood by those skilled in the art, unless an explicit indication to the contrary is made herein. Specifically, unless a claim recites an explicit limitation to the contrary, use of singular articles such as "a," "an," "the," and "said" should be construed as reciting one or more of the indicated elements.

[0237] According to the present invention, a device is provided, the device comprising: a processor; a memory coupled to the processor, the memory comprising processor-executable instructions, the processor-executable instructions, when executed by the processor, configuring the processor to: determine first intrinsic camera parameters of a first camera; determine second intrinsic camera parameters of a second camera; calculate a scaling factor for resizing a first calibration image and a second calibration image based on a relationship between the first intrinsic camera parameters and the second intrinsic camera parameters; and resize the first calibration image and the second calibration image according to the scaling factor to provide a resized first calibration image and a resized second calibration image.

[0238] According to one embodiment, the processor is further configured to map pixel coordinates of the resized first calibration image to pixel coordinates of the first calibration image to generate a resizing map.

[0239] According to one embodiment, the processor is further configured to: receive a distortion correction matrix of the first camera; and apply the distortion correction matrix to the resized first calibration image to provide a distortion corrected, resized first calibration image.

[0240] According to one embodiment, the processor is further configured to map pixel coordinates of the distortion corrected, resized first calibration image to pixel coordinates of the resized first calibration image to generate a distortion correction map.

[0241] According to one embodiment, the processor is configured to: resample the distortion-corrected, resized first calibration image to provide a resampled, distortion-corrected, resized first calibration image; and map pixel coordinates of the resampled, distortion-corrected, resized first calibration image to pixel coordinates of the distortion-corrected, resized first calibration image to provide a resampled map.

[0242] According to one embodiment, the processor is also configured to resample the distortion-corrected, resized first calibration image by: correcting the distortion-corrected, resized first calibration image to provide a corrected, distortion-corrected, resized first calibration image; warping the corrected, distortion-corrected, resized first calibration image to fit the corrected, distortion-corrected, resized second calibration image; and performing inverse correction on the corrected, distortion-corrected, resized first calibration image to provide the resampled, distortion-corrected, resized first calibration image.

[0243] According to one embodiment, the processor is also configured to: detect a calibration target in the distortion-corrected, resized first calibration image; detect the calibration target in the corresponding distortion-corrected, resized second calibration image; identify a correspondence between the first distortion-corrected, resized calibration image and the second distortion-corrected, resized calibration image; calculate a basic matrix based on the correspondence; calculate a first corrective transformation of the distortion-corrected, resized first calibration image based on the basic matrix; and calculate a second corrective transformation of the distortion-corrected, resized second calibration image based on the basic matrix.

[0244] According to one embodiment, the processor is also configured to: apply the first corrective transform to the distortion-corrected, resized first calibration image to provide a rectified, distortion-corrected, resized first calibration image; and map pixel coordinates of the rectified, distortion-corrected, resized first calibration image to pixel coordinates of the distortion-corrected, resized first calibration image to provide a corrective map.

[0245] According to one embodiment, the processor is also configured to: determine a best fit polynomial function to fit the rectified, distortion corrected, resized first calibration image to the rectified, distortion corrected, resized second calibration image; determine a warp matrix based on the best fit polynomial function; and apply the warp matrix to the rectified, distortion corrected, resized first calibration image to provide a warped, rectified, distortion corrected, resized first calibration image.

[0246] According to one embodiment, the processor is further configured to map pixel coordinates of the warped, rectified, distortion-corrected, resized first calibration image to pixel coordinates of the rectified, distortion-corrected, resized first calibration image to generate a warp map.

[0247] According to the present invention, a method includes: determining first intrinsic camera parameters of a first camera; determining second intrinsic camera parameters of a second camera; calculating a scaling factor for resizing a first calibration image based on a relationship between the first intrinsic camera parameters and the second intrinsic camera parameters; and resizing the first calibration image and the second calibration image according to the scaling factor to provide a resized first calibration image and a resized second calibration image.

[0248] In one aspect of the invention, the method comprises mapping pixel coordinates of the resized first calibration image to pixel coordinates of the first calibration image to generate a resizing map.

[0249] In one aspect of the invention, the method comprises: receiving a distortion correction matrix for the first camera; and applying the distortion correction matrix to the resized first calibration image to provide a distortion corrected, resized first calibration image.

[0250] In one aspect of the invention, the method comprises mapping pixel coordinates of the distortion corrected, resized first calibration image to pixel coordinates of the resized first calibration image to generate a distortion correction map.

[0251] In one aspect of the present invention, the method includes: resampling the distortion-corrected, resized first calibration image to provide a resampled, distortion-corrected, resized first calibration image; and mapping pixel coordinates of the resampled, distortion-corrected, resized first calibration image to pixel coordinates of the distortion-corrected, resized first calibration image to provide a resampled map.

[0252] The method of claim 15, wherein resampling is performed by: rectifying the distortion-corrected, resized first calibration image to provide a rectified, distortion-corrected, resized first calibration image; warping the rectified, distortion-corrected, resized first calibration image to fit a rectified, distortion-corrected, resized second calibration image; and performing inverse rectification on the rectified, distortion-corrected, resized first calibration image to provide the resampled, distortion-corrected, resized first calibration image.

[0253] In one aspect of the present invention, the method includes: detecting a calibration target in the distortion-corrected, resized first calibration image; detecting the calibration target in the corresponding distortion-corrected, resized second calibration image; identifying a correspondence between the first distortion-corrected, resized calibration image and the second distortion-corrected, resized calibration image; and calculating a basic matrix based on the correspondence; calculating a first corrective transformation of the distortion-corrected, resized first calibration image based on the basic matrix; and calculating a second corrective transformation of the distortion-corrected, resized second calibration image based on the basic matrix.

[0254] In one aspect of the present invention, the method includes: applying the first corrective transform to the distortion-corrected, resized first calibration image to provide a rectified, distortion-corrected, resized first calibration image; and mapping pixel coordinates of the rectified, distortion-corrected, resized first calibration image to pixel coordinates of the distortion-corrected, resized first calibration image to provide a corrective map.

[0255] In one aspect of the invention, the method comprises: determining a best fit polynomial function to fit the rectified, distortion corrected, resized first calibration image to the rectified, distortion corrected, resized second calibration image; determining a warp matrix based on the best fit polynomial function; and applying the warp matrix to the rectified, distortion corrected, resized first calibration image to provide a warped, rectified, distortion corrected, resized first calibration image.

[0256] In one aspect of the invention, the method comprises mapping pixel coordinates of the warped, rectified, distortion-corrected, resized first calibration image to pixel coordinates of the rectified, distortion-corrected, resized first calibration image to generate a warp map.

Claims

1. A method comprising: determining first intrinsic camera parameters of the first camera; determining second intrinsic camera parameters of the second camera; calculating a scaling factor for resizing a first calibration image based on a relationship between the first intrinsic camera parameter and the second intrinsic camera parameter; as well as The first calibration image and the second calibration image are resized according to the scaling factor to provide a resized first calibration image and a resized second calibration image.

2. The method of claim 1, further comprising: Pixel coordinates of the resized first calibration image are mapped to pixel coordinates of the first calibration image to generate a resizing map.

3. The method of claim 1, further comprising: receiving a distortion correction matrix of the first camera; as well as The distortion correction matrix is ​​applied to the resized first calibration image to provide a distortion corrected, resized first calibration image.

4. The method of claim 3, further comprising: Pixel coordinates of the distortion corrected, resized first calibration image are mapped to pixel coordinates of the resized first calibration image to generate a distortion correction map.

5. The method of claim 3, further comprising: resampling the distortion-corrected, resized first calibration image to provide a resampled, distortion-corrected, resized first calibration image; Pixel coordinates of the resampled, distortion-corrected, resized first calibration image are mapped to pixel coordinates of the distortion-corrected, resized first calibration image to provide a resampled map.

6. The method of claim 5, wherein resampling is performed by: rectifying the distortion-corrected, resized first calibration image to provide a rectified, distortion-corrected, resized first calibration image; warping the rectified, distortion-corrected, resized first calibration image to fit the rectified, distortion-corrected, resized second calibration image; and An inverse rectification is performed on the rectified, distortion-corrected, resized first calibration image to provide the resampled, distortion-corrected, resized first calibration image.

7. The method of claim 6, further comprising: detecting a calibration target in the distortion-corrected, resized first calibration image; detecting the calibration target in a corresponding distortion-corrected, resized second calibration image; identifying a corresponding relationship in the first distortion-corrected, resized calibration image and the second distortion-corrected, resized calibration image; as well as Calculating a basic matrix based on the corresponding relationship; computing a first rectification transform of the distortion-corrected, resized first calibration image based on the fundamental matrix; as well as A second rectification transform of the distortion corrected, resized second calibration image is calculated based on the fundamental matrix.

8. The method of claim 7, further comprising: applying the first corrective transform to the distortion-corrected, resized first calibration image to provide a rectified, distortion-corrected, resized first calibration image; as well as Pixel coordinates of the rectified, distortion-corrected, resized first calibration image are mapped to pixel coordinates of the distortion-corrected, resized first calibration image to provide a rectification map.

9. The method according to claim 8, further comprising: determining a best fit polynomial function to fit the rectified, distortion corrected, resized first calibration image to the rectified, distortion corrected, resized second calibration image; determining a warp matrix based on the best fitting polynomial function; as well as The warp matrix is ​​applied to the rectified, distortion-corrected, resized first calibration image to provide a warped, rectified, distortion-corrected, resized first calibration image.

10. The method of claim 9, further comprising: Pixel coordinates of the warped, rectified, distortion-corrected, resized first calibration image are mapped to pixel coordinates of the rectified, distortion-corrected, resized first calibration image to generate a warp map.

11. A computer programmed to perform the method of any one of claims 1 to 10.

Images