Multi-camera processor with feature matching

Abstract

Embodiments of a semiconductor package apparatus can include techniques for capturing two or more concurrent images of a scene by two or more cameras, detecting a feature in a first image from a first camera of the two or more cameras, matching a feature in a second image from a second camera of the two or more cameras, and performing photometric calibration between the first camera and the second camera based on a portion of the first image corresponding to the detected feature and a portion of the second image corresponding to the matched feature. Other embodiments are disclosed and claimed.

Description

Technical Field

[0001] The embodiments generally relate to multi-camera systems. More specifically, the embodiments relate to multi-camera processors with feature matching. Background Technology

[0002] Multi-camera systems enable functions such as depth extraction, 3D reconstruction, and panoramic imaging. They can process images from multiple cameras concurrently to provide these capabilities. Attached Figure Description

[0003] By reading the following specification and appended claims and referring to the following drawings, those skilled in the art will understand the various advantages of the embodiments, wherein:

[0004] Figure 1 This is a block diagram of an example electronic processing system according to an embodiment;

[0005] Figure 2 This is a block diagram of an example of a semiconductor packaging apparatus according to an embodiment;

[0006] Figures 3A to 3C This is a flowchart illustrating an example of a method for calibrating multiple cameras according to an embodiment.

[0007] Figures 4A to 4B These are illustrative diagrams illustrating examples of images from multiple cameras according to an embodiment;

[0008] Figure 5 This is a block diagram illustrating an example of a single camera according to an embodiment;

[0009] Figure 6 These are illustrative diagrams illustrating an example of a processing flow for a multi-camera system according to an embodiment;

[0010] Figure 7 This is an illustrative diagram of another example of a processing flow for a multi-camera system according to an embodiment;

[0011] Figure 8 This is an illustrative diagram of another example of a processing flow for a multi-camera system according to an embodiment;

[0012] Figure 9 This is a block diagram of an example system with a navigation controller according to an embodiment; and

[0013] Figure 10 This is a block diagram of an example system with a small number of shapes according to an embodiment. Detailed Implementation

[0014] Now refer to Figure 1Embodiments of the electronic processing system 10 may include: a processor 11; two or more cameras 12 (e.g., cameras C1 to C2). N The system comprises: N>1), which is communicatively coupled to processor 11 to capture two or more concurrent images of a scene; and logic 13, which is communicatively coupled to processor 11, for: detecting features in a first image from a first camera C1 of two or more cameras 12; matching features in a second image from a second camera C2 of two or more cameras 12; and performing photometric calibration between the first camera C1 and the second camera C2 based on the portion of the first image corresponding to the detected features and the portion of the second image corresponding to the matched features. In some embodiments, logic 13 may be configured to: define a first region of interest (ROI) of the first camera C1 based on the detected features in the first image; define a second ROI of the second camera C2 based on the matched features in the second image; and perform photometric calibration between the first camera C1 and the second camera C2 based on the first ROI and the second ROI. For example, logic 13 may be configured to: define a first region of interest (ROI) and a second region of interest (ROI) based on the corresponding positions and sizes of features in the first image and the second image.

[0015] In some embodiments of system 10, logic 13 may additionally or alternatively be configured to: define a first ROI for the first camera C1; detect features in a first image based on the first ROI; define a second ROI for the second camera C2 based on matched features in a second image; and perform photometric calibration between the first camera C1 and the second camera C2 based on the first and second ROIs. For example, logic 13 may be configured to: define the second ROI based on the position and size of matched features in the second image. In any embodiment herein, logic 13 may also be configured to: determine one or more parameter values ​​for each of two or more cameras based on photometric calibration. For example, photometric calibration may determine parameter values ​​for one or more of focus, color correction, gamma correction, sharpness, exposure, black level, noise, white balance, and contrast.

[0016] Embodiments of each of the processor 11, camera 12, logic 13, and other system components described above can be implemented in hardware, software, or any suitable combination thereof. For example, hardware implementations may include configurable logic, such as a programmable logic array (PLA), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), or fixed-function logic hardware using circuit technologies (e.g., application-specific integrated circuit (ASIC), complementary metal-oxide-semiconductor (CMOS), or transistor-transistor (TTL) technology), or any combination thereof.

[0017] Alternatively or additionally, all or part of these components may be implemented in one or more modules as a set of logical instructions stored in a machine- or computer-readable storage medium (e.g., random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc.) for execution by a processor or computing device. For example, computer program code for performing component operations may be written in any combination of one or more operating system (OS)-applicable / appropriate programming languages, including object-oriented programming languages ​​(e.g., PYTHON, PERL, JAVA, SMALLTALK, C++, C#, etc.) and traditional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). For example, main memory, permanent storage media, or other system memory may store a set of instructions that, when executed by processor 11, cause system 10 to implement one or more components, features, or aspects of system 10 (e.g., logic 13, detecting features in a first image, matching features in a second image, performing photometric calibration, etc.).

[0018] Now refer to Figure 2 Embodiments of the semiconductor packaging device 20 may include one or more substrates 21 and logic 22 coupled to the one or more substrates 21, wherein the logic 22 is at least partially implemented in one or more of configurable logic and fixed-function hardware logic. The logic 22 coupled to the one or more substrates may be configured to: capture two or more concurrent images of a scene using two or more cameras; detect features in a first image from a first camera among the two or more cameras; match features in a second image from a second camera among the two or more cameras; and perform photometric calibration between the first and second cameras based on portions of the first image corresponding to the detected features and portions of the second image corresponding to the matched features. In some embodiments, the logic 22 may be configured to: define a first ROI for the first camera based on the detected features in the first image; define a second ROI for the second camera based on the matched features in the second image; and perform photometric calibration between the first and second cameras based on the first and second ROIs. For example, the logic 22 may be configured to: define the first and second ROIs based on the corresponding positions and sizes of features in the first and second images.

[0019] In some embodiments, logic 22 may be additionally or alternatively configured to: define a first ROI for a first camera; detect features in a first image based on the first ROI; define a second ROI for a second camera based on matched features in a second image; and perform photometric calibration between the first and second cameras based on the first and second ROIs. For example, logic 22 may be configured to: define the second ROI based on the position and size of matched features in the second image. In any embodiment herein, logic 22 may also be configured to: determine one or more parameter values ​​for each of two or more cameras based on photometric calibration. For example, photometric calibration may determine parameter values ​​for one or more of focus, color correction, gamma correction, sharpness, exposure, black level, noise, white balance, and contrast.

[0020] Embodiments of logic 22 and other components of device 20 may be implemented in hardware, software, or any combination thereof, including at least a partial implementation in hardware. For example, a hardware implementation may include configurable logic (e.g., PLA, FPGA, CPLD, or fixed-function logic hardware using circuitry techniques (e.g., ASIC, CMOS, or TTL technology), or any combination thereof). Additionally, portions of these components may be implemented in one or more modules as a set of logic instructions stored in a machine- or computer-readable storage medium (e.g., RAM, ROM, PROM, firmware, flash memory, etc.) for execution by a processor or computing device. For example, computer program code for performing component operations may be written in any combination of one or more OS-applicable / appropriate programming languages, including object-oriented programming languages ​​(e.g., PYTHON, PERL, JAVA, SMALLTALK, C++, C#, etc.) and traditional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages).

[0021] Now refer to Figures 3A to 3CEmbodiments of the method 30 for calibrating multiple cameras may include: at block 31, capturing two or more concurrent images of a scene using two or more cameras; at block 32, detecting features in a first image from a first camera among the two or more cameras; at block 33, matching features in a second image from a second camera among the two or more cameras; and at block 34, performing photometric calibration between the first and second cameras based on portions of the first image corresponding to the detected features and portions of the second image corresponding to the matched features. Some embodiments of method 30 may also include: at block 35, defining a first Region of Interest (ROI) for the first camera based on the detected features in the first image; at block 36, defining a second ROI for the second camera based on the matched features in the second image; and at block 37, performing photometric calibration between the first and second cameras based on the first and second ROIs. For example, method 30 may also include: at block 38, defining the first and second ROIs based on the corresponding positions and sizes of features in the first and second images.

[0022] Some embodiments of method 30 may additionally or alternatively include: defining a first ROI for a first camera at block 39; detecting features in a first image based on the first ROI at block 40; defining a second ROI for a second camera based on matched features in a second image at block 41; and performing photometric calibration between the first and second cameras based on the first and second ROIs at block 42. For example, method 30 may further include: defining a second ROI based on the position and size of matched features in the second image at block 43. For any embodiment herein, method 30 may further include: determining one or more parameter values ​​for each of two or more cameras based on photometric calibration at block 44. Non-limiting examples of parameter values ​​at block 45 may include focus, color correction, gamma correction, sharpness, exposure, black level, noise, white balance, contrast, etc.

[0023] Embodiments of method 30 can be implemented in systems, apparatuses, computers, devices, etc., such as those described herein. More specifically, hardware implementations of method 30 may include configurable logic (e.g., PLA, FPGA, CPLD, or fixed-function logic hardware using circuitry techniques (e.g., ASIC, CMOS, or TTL technology), or any combination thereof). Alternatively or additionally, method 30 may be implemented in one or more modules as a set of logic instructions stored in a machine- or computer-readable storage medium (e.g., RAM, ROM, PROM, firmware, flash memory, etc.) for execution by a processor or computing device. The computer program code for performing component operations can be written in any combination of one or more OS-applicable / appropriate programming languages, including object-oriented programming languages ​​(e.g., PYTHON, PERL, JAVA, SMALLTALK, C++, C#, etc.) and traditional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages).

[0024] For example, method 30 can be implemented on a computer-readable medium in conjunction with the examples 19 through 24 below. Embodiments or portions of method 30 can be implemented in firmware, applications (e.g., via application programming interfaces (APIs)) or driver software running on an operating system (OS).

[0025] Some embodiments can advantageously provide a multi-camera image signal processor (ISP) utilizing feature detection and matching. Multi-camera arrays can include various computational imaging capabilities, such as (e.g., with a dual-camera arrangement) depth extraction, 3D reconstruction, panoramic imaging, 360-degree video, etc. (e.g., more capabilities possible than in the case of a single camera). For accurate functionality of a multi-camera array, calibration at three levels—including geometry (e.g., relative pose), photometric (e.g., similar image quality), and temporality (e.g., time-synchronized capture)—may be important. Some embodiments can improve photometric calibration by providing better photometric similarity across multi-camera images capturing the same scene.

[0026] The photometric properties of an image can include the exposure, focus, white balance, gamma correction, and sharpness settings used to capture the image. These parameters can form different blocks for the ISP (Image Signal Processor) of a camera. Because each camera in a multi-camera array can have its own ISP, the cameras can generate images with photometric variations. For photometric calibration, all or most of these parameters can be locally adapted for each camera so that the output images after each component ISP block look similar. Successful calibration can produce a final set of color images from each camera that appears more consistent with all these parameters. Some other systems can utilize a master-slave technique, where one camera is selected as the master camera, and the other cameras in the multi-camera system are configured to have the same parameters as the master. However, some parameters do not conform to the master-slave technique. For example, focus may not work well for this technique because the ROI (Region of Interest) focused by the master can change with the camera viewpoint, and using the same ROI for the slaves may result in focusing on completely different objects that may also be at different distances. Some embodiments can provide photometric calibration techniques that advantageously allow dynamic ROIs and can utilize optimization methods for parameter estimation.

[0027] Some embodiments can advantageously utilize feature detection and feature matching techniques to dynamically define Regions of Interest (ROIs) in an image for photometric calibration. Any suitable feature detection / matching technique can be used, including Scale Invariant Feature Transform (SIFT), Oriented Fast and Rotated BRIEF (ORB), Speed-Up Robust Feature Transform (SURF), KAZE, AKAZE, etc. Some embodiments can leverage unique features and feature matching between multi-camera images obtained after each ISP block to define dynamic ROIs. For example, an ROI may correspond to a list of matching two-dimensional (2D) pixel locations and a descriptor of the region surrounding each 2D pixel location. For some ISP parameters (e.g., color correction), the defined ROI may be a function of the location of detected matching features. For other parameters (e.g., focus), a specific ROI can be defined in the master camera, and features within the ROI can then be tracked across the image. In some embodiments, both scenarios can advantageously allow dynamic ROIs.

[0028] In some embodiments, the properties of the matched features (e.g., intensity, sharpness, etc. within the window) can be utilized as observations. For example, an error function parameterized by the ISP parameters can be formulated based on the observations. The error function can then be solved as an optimization problem to achieve improved or optimal multi-camera ISP parameters in an automated manner. Advantageously, some embodiments can provide techniques for improving the tuning and utilization of multi-camera systems. Some embodiments can determine improved or optimized parameter values ​​for any block of the camera ISP, including but not limited to focus, color correction, gamma correction, sharpness, exposure, black level, noise, white balance, and contrast.

[0029] Some other multi-camera systems may require homogeneous cameras with very similar characteristics to maintain effective photometric calibration. Advantageously, some embodiments may utilize heterogeneous camera systems. In some embodiments, some photometric variations between heterogeneous cameras can be effectively calibrated by determining appropriate ISP parameters to account for variations. Due to the use of feature points in some embodiments, any heterogeneity across cameras that is not an ISP parameter can be advantageously encoded in the feature descriptor. Some embodiments may provide feature-based optimization techniques to determine multi-camera setup and / or operating parameters. In some embodiments, individual camera parameters may be extended or optimized in a systemic manner for the multi-camera system.

[0030] Now refer to Figures 4A to 4B Four corresponding cameras C1 to C4 (not shown) can concurrently capture a set of four images I1 to I4. The captured images can have significant color temperature differences, such as different shadow patterns (e.g., image I1 can be warmer, while image I2 can have a much cooler hue). For example, even in a homogeneous camera array, inconsistencies can exist between cameras regarding various camera parameters (e.g., exposure, noise, color, sharpness, etc.). This inconsistency can lead to various problems in applications that utilize multiple images (e.g., computer vision). For example, luminance variations can cause visual artifacts in view interpolation and image stitching. Some embodiments can provide global optimization of the color and gamma parameters of individual cameras C1 to C4 to provide a final image that is more similar in luminance. For example, feature-based color and gamma correction can be applied to the captured images I1 to I4 to provide a corrected image I... 1C to I 4C (For example, see) Figure 4B Feature-based color and gamma correction can achieve similar color temperatures, allowing a complete set of four corrected images I1 to I4 to achieve color temperature similarity as represented by the same shading pattern in all four images. Corrected image I 1C To I 4C This can help mitigate some of these artifacts or improve the operation of downstream image processing applications.

[0031] Some embodiments may use feature detection and feature matching computed across multiple captured images to dynamically change the ROI depending on the camera viewpoint definition. An optimization function can then be formulated, which can be used to locally refine individual ISP parameter blocks to produce photometrically similar images (e.g., with respect to exposure, color, gamma, sharpness, etc.). The optimization function may depend on the ISP parameter blocks being optimized, and some embodiments may achieve similarity in the output image after each parameterized ISP block. Some embodiments may apply this feature-based photometric calibration to multiple ISP parameter blocks, making the final images obtained from the ISP pipeline more similar in several aspects.

[0032] Some other multi-camera systems can use a master-slave configuration, where master parameters are calculated to represent image regions in the master image, and then broadcast to slave cameras. Each slave camera either uses the master parameters directly (e.g., focus parameters) or adapts the master parameters independently of other slave cameras to optimize its own image criteria. Some other multi-camera systems can have feature-based approaches that primarily focus on color correction without applying any global optimization. Some embodiments can leverage feature detection and feature matching to sample data for ISP parameters and can also optimize ISP parameters considering all cameras. For example, some embodiments can simultaneously optimize all ISP parameters, either globally or with respect to a specified master camera, for each camera in the multi-camera array.

[0033] In some embodiments, feature matching can track similar features across images to define dynamically sized ROIs for different camera images. Dynamically sized ROIs can be particularly advantageous for multi-camera arrays, where each camera may have a different viewpoint. Some embodiments may use feature matching for multi-camera ISP optimization.

[0034] Some implementations may utilize feature descriptors that are invariant to image properties not labeled by the ISP block. For example, an invariant feature descriptor encoding gradient information can be scaled around feature points in two images at different resolutions to determine sharpness. The scaling around the feature points may be independent of the resolution of the two images. Some other techniques may use a square window as the ROI around the feature points, and may first have to determine the relative image scale to know the window size for calculating the image gradient to determine the focus.

[0035] Some other multi-camera systems can post-process the final image after the ISP pipeline stage. Some embodiments can process intermediate ISP block images to achieve image similarity regarding that particular ISP block. Intermediate processing can advantageously allow for greater flexibility in ISP pipeline optimization and avoid loss of technical information about the techniques used to work on the final ISP image.

[0036] Some embodiments may use feature detection and feature matching techniques to generate variable-size Regions of Interest (ROIs) in each of multiple camera images. An ROI may correspond to a list of 2D pixel locations being tracked across multiple camera images. Depending on the ISP parameters being optimized, the ROI may be the entire image (e.g., in the case of color and gamma correction), or a pre-selected ROI may be found in one of the images (e.g., all feature points within a rectangular region in one of the images). These points can then be tracked in all other images (e.g., in the case of focus settings). Finally, an optimization error function dependent on the ISP parameters can be defined in the tracked ROI.

[0037] Now refer to Figure 5 An embodiment of a single camera 50 may include an ISP pipeline with various different ISP parameters / blocks. Camera 50 may include an image sensor 51 with a pixel sensor and a color filter array (CFA) (e.g., or a color filter mosaic (CFM)) placed on the pixel sensor of image sensor 51 to capture color information. Lens 52 may be appropriately positioned relative to image sensor 51. Focus control block 53 may control the focus used for lens 52, and exposure control block 54 may control the exposure of the image captured by image sensor 51. Sensor defect correction block 55 may be coupled between image sensor 51 and focus control block 53. Image data from image sensor 51 can be processed by an ISP pipeline, which may include serially coupled ISP blocks, such as a black level adjustment block 56a, a noise reduction block 56b, a white balance block 56c, a CFA interpolation block 56d, a color correction matrix (CCM) block 56e, a gamma correction block 56f, a red-green-blue (RGB) to luminance-blue difference-red difference (YCC) conversion block 56g, a sharpness block 56h, a contrast enhancement block 56i, and an error chromaticity suppression block 56j. In some embodiments, the ISP pipeline may include more or fewer blocks, and may include different ISP blocks and / or different orders of blocks. As a result of processing the image captured by image sensor 51 through the ISP pipeline, camera 50 can provide an output image 57.

[0038] Many parameters can affect how various control blocks and ISP blocks of camera 50 operate. According to some embodiments, camera 50 may include a parameter adjuster 58 to adjust one or more parameters of camera 50 based on feature detection, feature matching techniques, and / or dynamically defined ROIs for photometric calibration with respect to one or more other cameras in a multi-camera system. For example, as part of a multi-camera system, camera 50 may be coupled wired or wirelessly to other cameras or a centralized multi-camera processor to dynamically define ROIs in an image using feature detection and feature matching techniques for photometric calibration. Camera 50 may communicate with other cameras about information to exchange data and parameters from camera 50, data and parameters from other cameras, and / or calculations / adjustments to improve photometric calibration across multiple cameras.

[0039] Now refer to Figure 6 Embodiments of the processing flow 60 for a multi-camera system may include photometric calibration for multi-camera focus control. Some embodiments of the system may allow some or all cameras in the multi-camera system to focus on the same depth or the same features. Advantageously, some embodiments may help remove out-of-focus blur in captured images. For example, some embodiments may determine an improved or ideal focus motor position for each of the cameras with autofocus capabilities in the multi-camera grid. Features detected in the autofocus ROI from the master camera can be tracked in all other cameras, and a feature-matching ROI can be determined for all other images. Individual cameras can then use their feature-matching ROIs to determine optimal focus positions and can accordingly activate the focus motors.

[0040] Processing flow 60 may include: main camera C m and two or more additional cameras C1 to C N Each of (N>1) captures raw data in BAYER format in boxes 61a to 61c, and fills in defective pixels in boxes 62a to 62c. Main camera C m In box 63, a 2D autofocus ROI can be selected in the scene where the camera wants to focus. An autofocus ROI can be selected manually or automatically (e.g., as the most prominent area in the image). An autofocus ROI can be any suitable size and shape (e.g., not necessarily any geometry). For example, an autofocus ROI can be based on the focal point plus and minus Δx and Δy values. (Main camera C) m It can have a fixed focus area and / or size, and the autofocus ROI can correspond to that focus area and size (e.g., adding / subtracting a certain Δx / y). Main camera C mIt can automatically detect features in an image (e.g., faces, objects, etc.), and the autofocus ROI can correspond to the detected feature focus area (e.g., adding / subtracting a certain Δx / y). Main camera C m Users can be allowed to select a focus area (e.g., by touching the display screen to select a focus point / block / face / object, etc.), and the autofocus ROI can correspond to the user-selected focus area (e.g., by adding / subtracting a certain Δx / y).

[0041] In box 64a, the autofocus ROI from the main camera C can be targeted. m Feature detection (e.g., SIFT) can be performed on the images. In boxes 62b to 62c, features from cameras C1 to C2 can be analyzed. N Feature detection is performed on the images in their respective complete captured images. In box 65, feature matching can be performed between features detected in the autofocus ROI and all other images. An autofocus ROI can be selected for the master camera in box 66a, and in boxes 66b-66c, image-dependent feature-matched ROIs can be selected in other images based on the tracked features. Each camera can then compute a focus metric for feature points that are part of the corresponding autofocus ROI and feature-matched ROI in boxes 67a-67c. A focus actuator for each camera can be computed to focus based on the repeated focus metric in boxes 68a-68c, and a corresponding output image can be provided in boxes 69a-69c. Advantageously, the output image can be focused on the same scene point from all cameras in the system. In some embodiments, the multi-camera focus controller can perform some or all of the processing flows 60 including boxes 64-66. The multi-camera focus controller can be a separate processor that can be implemented in one of the cameras (e.g., the master camera C). M They can be located in one camera or distributed among two or more cameras.

[0042] Now refer to Figure 7An embodiment of the processing flow 70 for a multi-camera system may include photometric calibration for color correction matrix and gamma correction. As the multi-camera system operates, each component camera can capture scene statistics, which may include a grayscale histogram, red / green color ratio, and blue / green color ratio for each pixel. These color ratios can determine a reference brightness and thus the color temperature of the scene. To achieve consistent color reproduction across different cameras, a color correction matrix (CCM) can be calculated after white balance. The gamma correction curve for all sensors can also be adjusted to produce similar contrast for all images in the multi-camera array. Some embodiments may advantageously provide multi-camera CCM and gamma correction based on feature detection, matching, and optimization functions. For example, as part of an ISP pipeline, some embodiments may provide a multi-camera color correction matrix C for the i-th camera in the multi-camera array. i * and gamma parameter β m *、β i The calculation of *. Some embodiments may also provide optimized functions for calculating these parameters.

[0043] One of the cameras in a multi-camera system can be designated or selected as the primary camera C. m Other cameras can be designated as camera C. i A single camera ISP block color correction matrix [a1a2…] can be applied within boxes 71a to 71i. {m,1,…,i} And parameter γ can be used in boxes 72a to 72i. {m,1,…,i} A single camera ISP block performs gamma correction to provide the corresponding image I after the single camera ISP. m to I i Feature detection can be performed on each of the full images in boxes 74a to 74i. Feature matching can then be performed on the detected features across all multi-camera images, and stored as f in box 75. The optimization function E(β) can be solved in box 76. m ,β {1,…,i} C {1,…,i} To refine the individual ISP CCM matrix and gamma values, optimized parameters can then be applied in blocks 77a to 77i to obtain the corrected image (e.g., and then the next ISP block is implemented). The gamma correction parameters (β) can be applied to both the primary camera and the remaining cameras. m ,β {1,…,i} The cost E(β) of jointly solving the CCM and gamma optimization is obtained using the relative CCM adjustment matrix C. m ,β {1,…,i} C {1,…,i} ISP parameters can be corrected using the CCM and gamma of individual cameras (e.g., [a1, a2, ...] respectively). [m,1,…,i] and γ [m,1,…,i]This function is initialized. It can be done in image I. m I1, ... I i Optimization is performed on the RGB values ​​of the feature matching position f. In some embodiments, the multi-camera CCM and gamma correction module may perform some or all of the processing flows 70 including blocks 74 to 77. The multi-camera CCM and gamma correction module may be a separate processor and may be implemented in one of the cameras (e.g., the main camera C). M They can be located in one camera or distributed among two or more cameras.

[0044] Now refer to Figure 8 An embodiment of the processing flow 80 for a multi-camera system may include photometric calibration for sharpness. Sharpness mismatch between images from a camera array can produce image pairs with blurred shapes. Many applications of multi-camera arrays on these image pairs (e.g., panoramic stitching, parallax estimation, etc.) may have artifacts. Some embodiments can advantageously set sharpness jointly on the camera array. Spatial frequency response (SPR) or modulation transfer function (MTF) can provide a measure of sharpness, where the index of image sharpness can be spatial frequency, where the MTF is 50% of its low-frequency value (MTF50) or 50% of its peak value (MTF50P). Some other systems can measure the MTF50 of individual cameras based on technical objectives and calculate sharpness parameters to enhance the high frequencies in the MTF. However, for multi-camera arrays, viewpoint changes may cause the ROI on the technical objective to shift for each camera. Accordingly, using a fixed ROI location on all camera images allows non-technical objective areas of the scene to also be analyzed for MTF. Some embodiments can utilize feature detection and matching to track the target ROI of the master camera. Advantageously, the scene / viewpoint dependent ROI can remain the same for each camera, and some embodiments can provide improved sharpness in multi-camera systems. In some embodiments, homography can then be used to register the tracked ROI, and the resulting registered image can be used to compute individual camera-dependent sharpness parameters.

[0045] One of the cameras in a multi-camera system can be designated or selected as the primary camera C. m Other cameras can be designated as cameras C1 to C2. N (N>1). In box 81, data can be obtained from the main camera C. m The ROI is specified on the technical target in the image. After the RGB to YCC conversion block, in boxes 82a to 82c, the feature detector can detect features from the master camera C. m In the images and all other cameras C1 to C N Features of the ROI in the complete image. These can be seen in boxes 83a to 83b at the main camera C. mFeatures detected in the ROI are matched with features from different images. Feature matching can be defined from C1 to C2 of other cameras. N The ROI in the image. In boxes 84a-84b, paired planar homography can be calculated based on at least four matches found. In boxes 85a-85c, the calculated homography can be used to position the master camera C. m The ROI is registered / aligned with the ROIs from cameras C1 and C2. In box 86, the sharpest ROI S can be calculated based on frequency analysis. In boxes 87a to 87c, for each camera image ROI, the corresponding ideal sharpest ROI can be determined, and the formula K = (registered ROI area) * S can be used. -1 The sharpness kernel is computed in the Fourier domain. In some embodiments, homography can be paired, and these pairs can be geometrically adjacent to each other. For example, geometrically adjacent cameras may have significant field-of-view overlap. In some embodiments, the sharpness kernel can be calculated for H. m1 H 12 H 23 H 34 And so on up to H {N-1,N} Calculate homography to generate region R m R1, R2, and so on up to region R N .

[0046] For example, for some images, the main camera C m The image can filter out some high frequencies. The sharpness kernel from box 87a, when applied to the main camera C... m When capturing images, it's possible to generate an MTF that allows for high frequencies and becomes similar to the MTF of another camera. For example, the main camera C can be increased. m The MTF50 is matched more closely to that of the other camera. When optimized to produce better sharpness, the other camera, when combined with another camera, may result in lower correction errors and smoother parallax. For example, in some embodiments of multi-camera systems with sharpness optimization, average correction accuracy can be improved, and parallax can be less noisy.

[0047] In some embodiments, the multi-camera sharpening module may perform some or all of the processing flows 80 including those in blocks 81 to 87. The multi-camera sharpening module may be a separate processor and may be implemented in one of the cameras (e.g., the primary camera C). M In a multi-camera setup, or distributed among two or more cameras. Some embodiments may include two or more multi-camera photometric calibration modules, which include a multi-camera focus controller (…). Figure 6 ), multi-camera CCM and gamma correction module ( Figure 7 ) and multi-camera sharpness module ( Figure 8Two or more of the following. In view of the benefits of this specification and the accompanying drawings, feature detection and matching of ROIs based on dynamic and / or variable-size ROIs in multi-camera systems for photometric calibration applications can be readily applied to other ISP parameter blocks. Figures 6 to 8 The embodiments described herein should be considered illustrative rather than limiting.

[0048] Figure 9 An embodiment of system 700 is shown. In the embodiment, system 700 may be a media system, but system 700 is not limited to this context. For example, system 700 may be incorporated into a personal computer (PC), laptop computer, ultrabook computer, tablet, touchpad, portable computer, handheld computer, PDA, personal digital assistant (PDA), cellular phone, combination of cellular phone and PDA, television, smart device (e.g., smartphone, smart tablet or smart TV), mobile internet device (MID), messaging device, data communication device, etc.

[0049] In one embodiment, system 700 includes a platform 702 coupled to a display 720 that presents visual content. Platform 702 can receive video bitstream content from a content device (e.g., content serving device 730 or content delivery device 740 or other similar content source). A navigation controller 750, including one or more navigation features, can be used to interact with, for example, platform 702 and / or display 720. Each of these components is described in more detail below.

[0050] In an embodiment, platform 702 may include any combination of chipset 705, processor 710, memory 712, storage 714, graphics subsystem 715, application 716, and / or radio device 718 (e.g., network controller). Chipset 705 may provide mutual communication among processor 710, memory 712, storage 714, graphics subsystem 715, application 716, and / or radio device 718. For example, chipset 705 may include a storage adapter (not described) capable of providing mutual communication with storage 714.

[0051] The processor 710 can be implemented as a Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processor, an x86 instruction set compatible processor, a multi-core or any other microprocessor or central processing unit (CPU). In embodiments, the processor 710 may include a dual-core processor, a dual-core mobile processor, etc.

[0052] The memory 712 can be implemented as a volatile memory device, such as, but not limited to, random access memory (RAM), dynamic random access memory (DRAM), or static RAM (SRAM).

[0053] Storage 714 can be implemented as a non-volatile storage device, such as, but not limited to, disk drives, optical disc drives, tape drives, internal storage devices, attached storage devices, flash memory, battery-backed SDRAM (synchronous DRAM), and / or network-accessible storage devices. In embodiments, for example, storage 714 may include techniques for enhancing storage performance protection for valuable digital media when multiple hard drives are included.

[0054] The graphics subsystem 715 can perform image (e.g., still or video) processing for display. For example, the graphics subsystem 715 can be a graphics processing unit (GPU) or a visual processing unit (VPU). Analog or digital interfaces can be used to communicatively couple the graphics subsystem 715 and the display 720. For example, the interface can be any of High Definition Multimedia Interface (HDMI), DisplayPort, Wireless HDMI, and / or Wireless HD Compliant technology. The graphics subsystem 715 can be integrated into a processor 710 or a chipset 705. The graphics subsystem 715 can be a standalone card communicatively coupled to the chipset 705. In one example, the graphics subsystem 715 includes a noise reduction subsystem as described herein.

[0055] The graphics and / or video processing techniques described herein can be implemented in various hardware architectures. For example, graphics and / or video functions can be integrated within a chipset. Alternatively, discrete graphics and / or video processors can be used. As yet another embodiment, graphics and / or video functions can be implemented by a general-purpose processor including a multi-core processor. In another embodiment, the functions can be implemented in a consumer electronic device.

[0056] Radio device 718 may be a network controller comprising one or more radio devices capable of transmitting and receiving signals using various suitable wireless communication technologies. These technologies may involve communication across one or more wireless networks. Exemplary wireless networks include (but are not limited to) wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area networks (WMANs), cellular networks, and satellite networks. In communicating across these networks, radio device 718 may operate according to one or more applicable standards of any version.

[0057] In embodiments, display 720 may include any type of television monitor or display. Display 720 may include, for example, a computer screen, touchscreen display, video monitor, television-like device, and / or television. Display 720 may be digital and / or analog. In embodiments, display 720 may be a holographic display. Furthermore, display 720 may be a transparent surface capable of receiving visual projections. These projections may convey various forms of information, images, and / or objects. For example, these projections may be visual overlays for mobile augmented reality (MAR) applications. Under the control of one or more software applications 716, platform 702 may display user interface 722 on display 720.

[0058] In this embodiment, the content service device 730 may be hosted by any national, international, and / or independent service, and therefore may be accessible via, for example, Internet access platform 702. The content service device 730 may be coupled to platform 702 and / or display 720. Platform 702 and / or content service device 730 may be coupled to network 760 to transmit (e.g., send and / or receive) media information to and / or from network 760. Content delivery device 740 may also be coupled to platform 702 and / or display 720.

[0059] In this embodiment, the content service device 730 may include a cable TV box, a personal computer, a network, a telephone, an internet-enabled device or appliance capable of transmitting digital information and / or content, and any other similar device capable of transmitting content unidirectionally or bidirectionally between the content provider and platform 702 and / or display 720 via network 760 or directly. It should be understood that content can be transmitted unidirectionally and / or bidirectionally via network 760 to and from any component of system 700 and the content provider. Examples of content may include any media information, including, for example, video, music, medical and gaming information.

[0060] Content service device 730 receives content (e.g., media information, digital information, and / or other content such as cable television programs). Examples of content providers may include any cable or satellite television or radio or internet content provider. The examples provided are not intended to limit the embodiments.

[0061] In one embodiment, platform 702 may receive control signals from navigation controller 750 having one or more navigation features. For example, the navigation features of controller 750 may be used to interact with user interface 722. In another embodiment, navigation controller 750 may be a pointing device, which may be a computer hardware component (specifically, a human-machine interface device) that allows users to input spatial (e.g., continuous and multidimensional) data into a computer. Many systems (e.g., graphical user interfaces (GUIs) and televisions and monitors) allow users to control and provide data to a computer or television using physical gestures.

[0062] The movement of navigation features of controller 750 can be reflected on a display (e.g., display 720) by the movement of a pointer, cursor, focus ring, or other visual indicator displayed on the display. For example, under the control of software application 716, navigation features located on navigation controller 750 can be mapped to virtual navigation features displayed on user interface 722. In embodiments, controller 750 may not be a separate component but may be integrated into platform 702 and / or display 720. However, embodiments are not limited to the elements or context shown or described herein.

[0063] In embodiments, for example, a driver (not shown) may include technology enabling a user to instantly turn platform 702 (e.g., a television) on and off via a touch button after initial startup upon activation. When the platform is “off,” program logic may allow platform 702 to stream content to a media adapter or other content service device 730 or content delivery device 740. Furthermore, for example, chipset 705 may include hardware and / or software support for 5.1 surround sound audio and / or high-definition 7.1 surround sound audio. The driver may include a graphics driver for an integrated graphics platform. In embodiments, the graphics driver may include a Peripheral Component Interconnect (PCI) high-speed graphics card.

[0064] In various embodiments, any one or more of the components shown in system 700 may be integrated. For example, platform 702 and content serving device 730 may be integrated, or platform 702 and content delivery device 740 may be integrated, or, for example, platform 702, content serving device 730 and content delivery device 740 may be integrated. In various embodiments, platform 702 and display 720 may be integrated units. For example, display 720 and content serving device 730 may be integrated, or display 720 and content delivery device 740 may be integrated. These examples are not intended to limit the embodiments.

[0065] In various embodiments, system 700 can be implemented as a wireless system, a wired system, or a combination of both. When implemented as a wireless system, system 700 may include components and interfaces suitable for communication via a wireless shared medium (e.g., one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, etc.). Examples of a wireless shared medium may include a portion of a wireless spectrum (e.g., RF spectrum, etc.). When implemented as a wired system, system 700 may include components and interfaces suitable for communication via a wired communication medium (e.g., input / output (I / O) adapters, physical connectors for connecting I / O adapters to corresponding wired communication media, network interface cards (NICs), disk controllers, video controllers, audio controllers, etc.). Examples of a wired communication medium may include leads, cables, metal wires, printed circuit boards (PCBs), backplanes, switch assemblies, semiconductor materials, twisted pairs, coaxial cables, optical fibers, etc.

[0066] Platform 702 can establish one or more logical or physical channels to transmit information. Information may include media information and control information. Media information can refer to any data representing content for a user. Examples of content may include data such as that from voice conversations, video conferencing, streaming video, email (“email”) messages, voicemail messages, alphanumeric symbols, graphics, images, video, text, etc. Data from voice conversations may be, for example, voice information, silence periods, background noise, comfort noise, tone, etc. Control information can refer to any data representing commands, instructions, or control words for an automated system. For example, control information can be used to route media information through the system or to instruct nodes to process media information in a predetermined manner. However, embodiments are not limited to this. Figure 9 The elements or context shown or described in the text.

[0067] As mentioned above, system 700 can be implemented using different physical styles or shapes. Figure 10 An embodiment of a small-scale device 800 that can embody system 700 is shown. In this embodiment, for example, device 800 can be implemented as a mobile computing device with wireless capabilities. A mobile computing device can refer to any device having a processing system and a mobile power source or power source (e.g., one or more batteries).

[0068] As described above, examples of mobile computing devices may include personal computers (PCs), laptop computers, ultrabook computers, tablets, touchpads, portable computers, handheld computers, PDAs, personal digital assistants (PDAs), cellular phones, combinations of cellular phones and PDAs, televisions, smart devices (e.g., smartphones, smart tablets, or smart TVs), mobile internet devices (MIDs), messaging devices, data communication devices, and the like.

[0069] Examples of mobile computing devices may also include computers arranged for human wear (e.g., wrist computers, finger computers, earphone computers, glasses computers, clip-on computers, armband computers, shoe computers, clothing computers, and other wearable computers). In embodiments, for example, a mobile computing device may be implemented as a smartphone capable of performing computer applications as well as voice and / or data communications. While some embodiments are described by way of example using a mobile computing device implemented as a smartphone, it is understood that other embodiments can also be implemented using other wireless mobile computing devices. Embodiments are not limited to this context.

[0070] like Figure 10 As shown, device 800 may include a housing 802, a display 804, an input / output (I / O) device 806, and an antenna 808. Device 800 may also include navigation features 812. Display 804 may include any suitable display unit for displaying information appropriate for the mobile computing device. I / O device 806 may include any suitable I / O device for inputting information into the mobile computing device. Examples of I / O device 806 may include an alphanumeric keypad, numeric keypad, touchpad, input keys, buttons, switches, rocker switches, microphones, speakers, voice recognition devices, and software, etc. Information may also be input into device 800 via a microphone. This information may be digitized by a voice recognition device. The embodiments are not limited to this context.

[0071] In some embodiments, system 700 and / or device 800 may implement one or more aspects of the multi-camera system described herein. For example, system 700 and / or device 800 may include one or more aspects of the following examples.

[0072] Additional notes and examples:

[0073] Example 1 may include an electronic processing system comprising: a processor; two or more cameras communicatively coupled to the processor for capturing two or more concurrent images of a scene; and logic communicatively coupled to the processor for: detecting features in a first image from a first camera of the two or more cameras; matching features in a second image from a second camera of the two or more cameras; and performing photometric calibration between the first camera and the second camera based on portions of the first image corresponding to the detected features and portions of the second image corresponding to the matched features.

[0074] Example 2 may include the system of Example 1, wherein the logic is further configured to: define a first region of interest for the first camera based on detected features in the first image; define a second region of interest for the second camera based on matched features in the second image; and perform photometric calibration between the first camera and the second camera based on the first region of interest and the second region of interest.

[0075] Example 3 may include the system of Example 2, wherein the logic is further configured to: define the first region of interest and the second region of interest based on the corresponding positions and sizes of features in the first image and the second image.

[0076] Example 4 may include the system of Example 1, wherein the logic is further configured to: define a first region of interest (ROI) for the first camera; detect features in the first image based on the first ROI; define a second ROI for the second camera based on the matched features in the second image; and perform photometric calibration between the first camera and the second camera based on the first ROI and the second ROI.

[0077] Example 5 may include the system of Example 4, wherein the logic is further configured to: define the second region of interest based on the position and size of the matched features in the second image.

[0078] Example 6 may include a system of any one of Examples 1 to 5, wherein the logic is further configured to: determine one or more parameter values ​​for each of the two or more cameras based on the photometric calibration.

[0079] Example 7 may include a semiconductor package device comprising: one or more substrates; and logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more of configurable logic and fixed-function hardware logic, the logic coupled to the one or more substrates being configured to: capture two or more concurrent images of a scene via two or more cameras; detect features in a first image from a first camera of the two or more cameras; match features in a second image from a second camera of the two or more cameras; and perform photometric calibration between the first camera and the second camera based on portions of the first image corresponding to the detected features and portions of the second image corresponding to the matched features.

[0080] Example 8 may include the apparatus of Example 7, wherein the logic is further configured to: define a first region of interest for the first camera based on detected features in the first image; define a second region of interest for the second camera based on matched features in the second image; and perform photometric calibration between the first camera and the second camera based on the first region of interest and the second region of interest.

[0081] Example 9 may include the apparatus of Example 8, wherein the logic is further configured to: define the first region of interest and the second region of interest based on the corresponding positions and sizes of features in the first image and the second image.

[0082] Example 10 may include the apparatus of Example 7, wherein the logic is further configured to: define a first region of interest (ROI) of the first camera; detect features in the first image based on the first ROI; define a second ROI of the second camera based on the matched features in the second image; and perform photometric calibration between the first camera and the second camera based on the first ROI and the second ROI.

[0083] Example 11 may include the apparatus of Example 10, wherein the logic is further configured to: define the second region of interest based on the position and size of the matched features in the second image.

[0084] Example 12 may include the apparatus of any of Examples 7 to 11, wherein the logic is further configured to: determine one or more parameter values ​​for each of the two or more cameras based on the photometric calibration.

[0085] Example 13 may include a method for calibrating multiple cameras, comprising: capturing two or more concurrent images of a scene using two or more cameras; detecting features in a first image from a first camera among the two or more cameras; matching features in a second image from a second camera among the two or more cameras; and performing photometric calibration between the first camera and the second camera based on portions of the first image corresponding to the detected features and portions of the second image corresponding to the matched features.

[0086] Example 14 may include the method of Example 13, and further includes: defining a first region of interest for the first camera based on detected features in the first image; defining a second region of interest for the second camera based on matched features in the second image; and performing photometric calibration between the first camera and the second camera based on the first region of interest and the second region of interest.

[0087] Example 15 may include the method of Example 14, and further includes: defining the first region of interest and the second region of interest based on the corresponding positions and sizes of features in the first image and the second image.

[0088] Example 16 may include the method of Example 13, further comprising: defining a first region of interest (ROI) of the first camera; detecting features in the first image based on the first ROI; defining a second ROI of the second camera based on the matched features in the second image; and performing photometric calibration between the first camera and the second camera based on the first ROI and the second ROI.

[0089] Example 17 may include the method of Example 16, and further includes: defining the second region of interest based on the position and size of the matched features in the second image.

[0090] Example 18 may include the method of any one of Examples 13 to 17, and further includes: determining one or more parameter values ​​for each of the two or more cameras based on the photometric calibration.

[0091] Example 19 may include at least one computer-readable medium comprising a set of instructions that, when executed by a computing device, cause the computing device to: capture two or more concurrent images of a scene using two or more cameras; detect features in a first image from a first camera of the two or more cameras; match features in a second image from a second camera of the two or more cameras; and perform photometric calibration between the first camera and the second camera based on portions of the first image corresponding to the detected features and portions of the second image corresponding to the matched features.

[0092] Example 20 may include at least one computer-readable medium of Example 19, including another set of instructions that, when executed by a computing device, cause the computing device to: define a first region of interest for the first camera based on detected features in the first image; define a second region of interest for the second camera based on matched features in the second image; and perform photometric calibration between the first and second cameras based on the first and second regions of interest.

[0093] Example 21 may include at least one computer-readable medium of Example 20, including another set of instructions that, when executed by a computing device, cause the computing device to: define a first region of interest and a second region of interest based on the corresponding positions and sizes of features in the first image and the second image.

[0094] Example 22 may include at least one computer-readable medium of Example 19, including another set of instructions that, when executed by a computing device, cause the computing device to: define a first region of interest (ROI) of the first camera; detect features in the first image based on the first ROI; define a second ROI of the second camera based on matched features in the second image; and perform photometric calibration between the first camera and the second camera based on the first ROI and the second ROI.

[0095] Example 23 may include at least one computer-readable medium of Example 22, including another set of instructions that, when executed by a computing device, cause the computing device to: define the second region of interest based on the position and size of the matched features in the second image.

[0096] Example 24 may include at least one computer-readable medium of any of Examples 19 to 23, including another set of instructions that, when executed by a computing device, cause the computing device to: determine one or more parameter values ​​for each of the two or more cameras based on the photometric calibration.

[0097] Example 25 may include a multi-camera calibration apparatus comprising: a module for capturing two or more concurrent images of a scene using two or more cameras; a module for detecting features in a first image from a first camera among the two or more cameras; a module for matching features in a second image from a second camera among the two or more cameras; and a module for performing photometric calibration between the first camera and the second camera based on portions of the first image corresponding to the detected features and portions of the second image corresponding to the matched features.

[0098] Example 26 may include the apparatus of Example 25, and further includes: a module for defining a first region of interest for the first camera based on detected features in the first image; a module for defining a second region of interest for the second camera based on matched features in the second image; and a module for performing photometric calibration between the first camera and the second camera based on the first region of interest and the second region of interest.

[0099] Example 27 may include the apparatus of Example 26, and further include: a module for defining the first region of interest and the second region of interest based on the corresponding positions and sizes of features in the first image and the second image.

[0100] Example 28 may include the apparatus of Example 25, further comprising: a module for defining a first region of interest of the first camera; a module for detecting features in the first image based on the first region of interest; a module for defining a second region of interest of the second camera based on matched features in the second image; and a module for performing photometric calibration between the first camera and the second camera based on the first region of interest and the second region of interest.

[0101] Example 29 may include the apparatus of Example 28, and further includes a module for defining the second region of interest based on the position and size of the matched features in the second image.

[0102] Example 30 may include the apparatus of any of Examples 25 to 29, and further includes a module for determining one or more parameter values ​​for each of the two or more cameras based on the photometric calibration.

[0103] The embodiments are applicable to the use of all types of semiconductor integrated circuit (“IC”) chips. Examples of such IC chips include, but are not limited to, processors, controllers, chipset assemblies, programmable logic arrays (PLAs), memory chips, network chips, system-on-a-chip (SoC), SSD / NAND controller ASICs, etc. Furthermore, in some figures, signal lines are represented by straight lines. Some may differ to indicate more constituting signal paths, have numerical labels to indicate multiple constituting signal paths, and / or have arrows at one or more ends to indicate the direction of the main information flow. However, this should not be construed as limiting. Rather, these added details can be used in conjunction with one or more exemplary embodiments to facilitate easier understanding of the circuit. Any represented signal line, whether or not it has additional information, can substantially include one or more signals that can travel in multiple directions and can be implemented by any suitable type of signaling scheme (e.g., digital or analog lines implemented via differential pairs, fiber optic lines, and / or single-ended lines).

[0104] Example sizes / models / values / ranges may have been given, but the embodiments are not limited thereto. As manufacturing technologies (e.g., photolithography) mature over time, it is expected that smaller devices can be manufactured. Furthermore, to simplify the description and discussion and thus avoid obscuring specific aspects of the embodiments, well-known power / ground connections for IC chips and other components may be shown or omitted in the figures. Additionally, arrangements may be shown in block diagram form to avoid obscuring the embodiments, and also given that details regarding the implementation of these block diagram arrangements are highly dependent on the platform on which the embodiments are implemented; i.e., such details should be well within the view of those skilled in the art. In the context of setting forth specific details (e.g., circuitry) to describe exemplary embodiments, it will be apparent to those skilled in the art that the embodiments can be practiced without these specific details or with variations thereof. Therefore, this description is to be considered illustrative rather than limiting.

[0105] The term “coupling” may be used herein to refer to any type of direct or indirect relationship between the components under discussion and may be applied to electrical, mechanical, fluid, optical, electromagnetic, electromechanical, or other connections. Furthermore, the terms “first,” “second,” etc., may be used herein solely for the purpose of facilitating discussion and, unless otherwise indicated, do not carry any specific temporal or chronological meaning.

[0106] As used in this application and claims, a list of entries linked by the term "one or more of" may represent any combination of the listed terms. For example, the phrase "one or more of A, B, and C" and the phrase "one or more of A, B, or C" may both represent A; B; C; A and B; A and C; B and C; or A, B, and C.

[0107] Those skilled in the art will understand from the foregoing description that a wide range of techniques can be implemented in various forms. Therefore, while embodiments have been described in conjunction with specific examples, the true scope of the embodiments should not be limited thereto, as other modifications will become apparent to those skilled in the art upon studying the drawings, specification, and appended claims.

Claims

1. An electronic processing system, comprising: processor; Two or more cameras, communicatively coupled to the processor, are used to capture two or more concurrent images of a scene, wherein one of the two or more cameras acts as a master camera and the other cameras act as slave cameras; and The logic coupled to the processor in a communicative manner is used for: Each of the two or more cameras performs its own image signal processing (ISP) to provide a corresponding complete image after the respective ISP is completed, wherein an initial color correction matrix (CCM) and initial gamma correction parameters are applied in the respective ISP; Detect features in a first complete image from the corresponding complete images of the master camera among the two or more cameras; The detected features are matched with features in a second complete image from the corresponding complete image of each of the two or more slave cameras; Optimization is performed on the RGB values ​​of the feature matching locations in the corresponding complete images provided by each camera to update the initial CCM for the slave camera and the initial gamma correction parameters for the master camera and the slave camera, wherein the optimized CCM for the slave camera and the optimized gamma correction parameters for the master camera and the slave camera are obtained by jointly solving the optimization function for each of the master camera and the slave camera with respect to the gamma correction parameters and the relative CCM adjustment matrix; as well as The optimized CCM for the slave camera and the optimized gamma correction parameters for the master camera and the slave camera are applied to each of the master camera and the slave camera, respectively, to obtain a corrected image.

2. The system as claimed in claim 1, wherein, The logic is also used for: Based on the detected features in the first complete image, a first region of interest for the main camera is defined. Based on the matched features in the second complete image, a second region of interest is defined for each of the slave cameras; as well as Based on the first region of interest and the second region of interest, perform photometric calibration between each of the master camera and the slave camera.

3. The system as described in claim 2, wherein, The logic is also used for: Based on the corresponding positions and sizes of features in the first and second complete images, the first region of interest and the second region of interest are defined.

4. The system as claimed in claim 1, wherein, The logic is also used for: Define the first region of interest of the main camera; Based on the first region of interest, detect features in the first complete image; Based on the matched features in the second complete image, a second region of interest is defined for each of the slave cameras; as well as Based on the first region of interest and the second region of interest, perform photometric calibration between each of the master camera and the slave camera.

5. The system as described in claim 4, wherein, The logic is also used for: The second region of interest is defined based on the location and size of the matched features in the second complete image.

6. The system as claimed in claim 2, wherein, The logic is also used for: Based on the photometric calibration, one or more parameter values ​​are determined for each of the two or more cameras.

7. A semiconductor packaging apparatus, comprising: One or more substrates; and Logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more of configurable logic and fixed-function hardware logic, the logic coupled to the one or more substrates being used for: Two or more concurrent images of a scene are captured by two or more cameras, wherein one of the two or more cameras acts as the master camera and the other cameras act as slave cameras; Each of the two or more cameras performs its own image signal processing (ISP) to provide a corresponding complete image after the respective ISP is completed, wherein an initial color correction matrix (CCM) and initial gamma correction parameters are applied in the respective ISP; Detect features in a first complete image from the corresponding complete images of the master camera among the two or more cameras; The detected features are matched with features in a second complete image from the corresponding complete image of each of the two or more slave cameras; Optimization is performed on the RGB values ​​of the feature-matching locations in the corresponding complete images provided by each camera to update the initial CCM for the slave camera and the initial gamma correction parameters for the master camera and the slave camera, wherein the optimized CCM for the slave camera and the optimized gamma correction parameters for the master camera and the slave camera are obtained by jointly solving the optimization function for each of the master camera and the slave camera with respect to the gamma correction parameters and the relative CCM adjustment matrix; and The optimized CCM for the slave camera and the optimized gamma correction parameters for the master camera and the slave camera are applied to each of the master camera and the slave camera, respectively, to obtain a corrected image.

8. The apparatus of claim 7, wherein, The logic is also used for: Based on the detected features in the first complete image, a first region of interest for the main camera is defined. Based on the matched features in the second complete image, a second region of interest is defined for each of the slave cameras; as well as Based on the first region of interest and the second region of interest, perform photometric calibration between each of the master camera and the slave camera.

9. The apparatus of claim 8, wherein, The logic is also used for: Based on the corresponding positions and sizes of features in the first and second complete images, the first region of interest and the second region of interest are defined.

10. The apparatus of claim 7, wherein, The logic is also used for: Define the first region of interest of the main camera; Based on the first region of interest, detect features in the first complete image; Based on the matched features in the second complete image, a second region of interest is defined for each of the slave cameras; as well as Based on the first region of interest and the second region of interest, perform photometric calibration between each of the master camera and the slave camera.

11. The apparatus of claim 10, wherein, The logic is also used for: The second region of interest is defined based on the location and size of the matched features in the second complete image.

12. The apparatus of claim 8, wherein, The logic is also used for: Based on the photometric calibration, one or more parameter values ​​are determined for each of the two or more cameras.

13. A method for calibrating multiple cameras, comprising: Two or more concurrent images of a scene are captured by two or more cameras, wherein one of the two or more cameras acts as the master camera and the other cameras act as slave cameras; Each of the two or more cameras performs its own image signal processing (ISP) to provide a corresponding complete image after the respective ISP is completed, wherein an initial color correction matrix (CCM) and initial gamma correction parameters are applied in the respective ISP; Detect features in a first complete image from the corresponding complete images of the master camera among the two or more cameras; The detected features are matched with features in a second complete image from the corresponding complete image of each of the two or more slave cameras; Optimization is performed on the RGB values ​​of the feature matching locations in the corresponding complete images provided by each camera to update the initial CCM for the slave camera and the initial gamma correction parameters for the master camera and the slave camera, wherein the optimized CCM for the slave camera and the optimized gamma correction parameters for the master camera and the slave camera are obtained by jointly solving the optimization function for each of the master camera and the slave camera with respect to the gamma correction parameters and the relative CCM adjustment matrix; as well as The optimized CCM for the slave camera and the optimized gamma correction parameters for the master camera and the slave camera are applied to each of the master camera and the slave camera, respectively, to obtain a corrected image.

14. The method of claim 13, further comprising: Based on the detected features in the first complete image, a first region of interest for the main camera is defined. Based on the matched features in the second complete image, a second region of interest is defined for each of the slave cameras; as well as Based on the first region of interest and the second region of interest, perform photometric calibration between each of the master camera and the slave camera.

15. The method of claim 14, further comprising: Based on the corresponding positions and sizes of features in the first and second complete images, the first region of interest and the second region of interest are defined.

16. The method of claim 13, further comprising: Define the first region of interest of the main camera; Based on the first region of interest, detect features in the first complete image; Based on the matched features in the second complete image, a second region of interest is defined for each of the slave cameras; as well as Based on the first region of interest and the second region of interest, perform photometric calibration between each of the master camera and the slave camera.

17. The method of claim 16, further comprising: The second region of interest is defined based on the location and size of the matched features in the second complete image.

18. The method of claim 14, further comprising: Based on the photometric calibration, one or more parameter values ​​are determined for each of the two or more cameras.

19. A multi-camera calibration device, comprising: A module for capturing two or more concurrent images of a scene using two or more cameras, wherein one of the two or more cameras is a master camera and the other cameras are slave cameras; A module for performing a respective image signal processing (ISP) on each of the two or more cameras to provide a corresponding complete image after the respective ISP is completed, wherein an initial color correction matrix (CCM) and initial gamma correction parameters are applied in the respective ISP; A module for detecting features in a first complete image from the corresponding complete images of the master camera among the two or more cameras; A module for matching detected features with features in a second complete image from each of the corresponding complete images of the slave cameras from the two or more cameras; A module for performing optimization on the RGB values ​​of the feature matching locations in the corresponding complete images provided by each camera to update the initial CCM for the slave camera and the initial gamma correction parameters for the master camera and the slave camera, wherein the optimized CCM for the slave camera and the optimized gamma correction parameters for the master camera and the slave camera are obtained by jointly solving the optimization function for each of the master camera and the slave camera with respect to the gamma correction parameters and the relative CCM adjustment matrix; as well as A module for applying the optimized CCM for the slave camera and the optimized gamma correction parameters for the master camera and the slave camera to each of the master camera and the slave camera, respectively, to obtain a corrected image.

20. The apparatus of claim 19, further comprising: A module for defining a first region of interest for the master camera based on the features detected in the first complete image; A module for defining a second region of interest for each of the slave cameras based on the matched features in the second complete image; and A module for performing photometric calibration between each of the master camera and the slave camera based on the first region of interest and the second region of interest.

21. The apparatus of claim 20, further comprising: A module for defining the first region of interest and the second region of interest based on the corresponding positions and sizes of features in the first complete image and the second complete image.

22. The apparatus of claim 19, further comprising: Module used to define the first region of interest of the main camera; A module for detecting features in the first complete image based on the first region of interest; A module for defining a second region of interest for each of the slave cameras based on the matched features in the second complete image; and A module for performing photometric calibration between each of the master camera and the slave camera based on the first region of interest and the second region of interest.

23. The apparatus of claim 22, further comprising: A module for defining the second region of interest based on the location and size of the matched features in the second complete image.

24. The apparatus of claim 20, further comprising: A module for determining one or more parameter values ​​for each of the two or more cameras based on the photometric calibration.

Images