Apparatus and method for multi-input image enhancement pipeline

The multi-input image enhancement pipeline addresses alignment and resolution issues in existing systems by using feature extraction and matching techniques, resulting in improved image alignment and reduced artifacts, enhancing user experience through accurate processing of diverse camera inputs.

WO2024232886A1PCT designated stage expired Publication Date: 2024-11-14ZEKU INC

Patent Information

Application Number
PCT/US2023/021781
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-10
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Existing multi-input image enhancement systems struggle to accurately align and process image data from different cameras with varying fields of view and resolutions, resulting in poor performance and user experience due to image artifacts like object blur.

Method used

A multi-input image enhancement pipeline that includes feature extraction, feature matching, and image enhancement components, using residual channel attention blocks, cost volume-based image matching, and stacked hourglass architectures to generate enhanced images by correlating and aggregating contextual information from multiple image sensors.

Benefits of technology

The solution effectively generates enhanced images with improved alignment and resolution, reducing image artifacts and enhancing user experience by accurately processing diverse image inputs from different cameras.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2023021781_14112024_PF_FP_ABST
    Figure US2023021781_14112024_PF_FP_ABST
Patent Text Reader

Abstract

According to one aspect of the present disclosure, an image-enhancement system is provided. The image-enhancement system may include a feature extraction component configured to receive first image data associated with a scene at a first field-of-view (FOV) and a first resolution and second image data associated with the scene at a second FOV and a second resolution. The feature extraction component may be configured to generate a set of feature maps associated with the first image data and the second image data. The image-enhancement system may include a feature matching component configured to generate correlation information based on the set of feature maps associated with the first image data and the second image data. The image-enhancement system may include an image enhancement component configured to generate an enhanced image of the scene based on the correlation information.
Need to check novelty before this filing date? Find Prior Art

Description

APPARATUS AND METHOD FOR MULTI-INPUT IMAGE ENHANCEMENT PIPELINEBACKGROUND

[0001] Embodiments of the present disclosure relate to apparatuses and methods for operating an image signal processor (ISP).

[0002] An image / video capturing device, such as a camera or camera array, can be used to capture an image / video or a picture of a scene. Cameras or camera arrays have been included in many handheld devices, especially since the advent of social media that allows users to upload pictures and videos of themselves, friends, family, pets, or landscapes on the Internet with ease and in real-time. Examples of camera components that operate together to capture an image / video include lens(es), image sensor(s), ISP(s), and / or encoders, just to name a few components thereof. The lens, for example, may receive and focus light onto one or more image sensors that are configured to detect photons. When photons impinge on the image sensor, an image signal corresponding to the scene is generated and sent to the ISP. The ISP performs various operations associated with the image signal to generate one or more processed images of the scene that can then be output to a user, stored in memory, or output to the cloud.SUMMARY

[0003] According to one aspect of the present disclosure, an image-enhancement system is provided. The image-enhancement system may include a feature extraction component. The feature extraction component may be configured to receive first image data associated with a scene at a first field-of-view (FOV) and a first resolution and second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution. The feature extraction component may be configured to generate a set of feature maps associated with the first image data and the second image data. The image-enhancement system may include a feature matching component. The feature matching component may be configured to receive the set of feature maps from the feature extraction component. The feature matching component may be configured to generate correlation information based on the set of feature maps associated with the first image data and the second image data. The image-enhancement system may include an image enhancement component. The image enhancement component may be configured to receive the correlation information associated with the first image data and the second image data fromthe feature matching component. The image enhancement component may be configured to generate an enhanced image of the scene based on the correlation information.

[0004] According to another aspect of the present disclosure, an ISP is provided. The ISP may include a first image sensor configured to capture first image data associated with a scene at a first FOV and a first resolution. The ISP may include a second image sensor configured to capture second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution. The ISP may include a feature extraction component. The feature extraction component may be configured to receive first image data associated with a scene at a first FOV and a first resolution and second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution. The feature extraction component may be configured to generate a set of feature maps associated with the first image data and the second image data. The ISP may include a feature matching component. The feature matching component may be configured to receive the set of feature maps from the feature extraction component. The feature matching component may be configured to generate correlation information based on the set of feature maps associated with the first image data and the second image data. The ISP may include an image enhancement component. The image enhancement component may be configured to receive the correlation information associated with the first image data and the second image data from the feature matching component. The image enhancement component may be configured to generate an enhanced image of the scene based on the correlation information.

[0005] According to a further aspect of the present disclosure, a method of image enhancement is provided. The method may include receiving, by a feature extraction component, first image data associated with a scene at a first FOV and a first resolution and second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution. The method may include generating, by the feature extraction component, a set of feature maps associated with the first image data and the second image data. The method may include receiving, by a feature matching component, the set of feature maps from the feature extraction component. The method may include generating, by the feature matching component, correlation information based on the set of feature maps associated with the first image data and the second image data. The method may include receiving, by an image enhancement component, the correlation information associated with the first image data and the second image data from the feature matching component. The method may include generating, by the image enhancement component, an enhanced image ofthe scene based on the correlation information.

[0006] These illustrative embodiments are mentioned not to limit or define the present disclosure, but to provide examples to aid understanding thereof. Additional embodiments are discussed in the Detailed Description, and further description is provided there.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments of the present disclosure and, together with the description, further serve to explain the principles of the present disclosure and to enable a person skilled in the pertinent art to make and use the present disclosure.

[0008] FIG. 1 illustrates a block diagram of an ISP device that includes a plurality of different image sensors, according to some embodiments of the present disclosure.

[0009] FIG. 2 illustrates a block diagram of an exemplary image-enhancement pipeline, according to some embodiments of the present disclosure.

[0010] FIG. 3 illustrates a detailed block diagram of an exemplary feature extraction component of the multi-input the image-enhancement pipeline of FIG. 2, according to some embodiments of the present disclosure.

[0011] FIG. 4 illustrates a block diagram of an exemplary image-enhancement component of the multi-input the image-enhancement pipeline of FIG. 2, according to some embodiments of the present disclosure.

[0012] FIG. 5 illustrates a block diagram of an exemplary apparatus that generates enhanced images, according to some embodiments of the present disclosure.

[0013] FIG. 6 illustrates a flowchart of an exemplary method of image-enhancement, according to some embodiments of the present disclosure.

[0014] FIG. 7 is a block diagram illustrating an example of a computer system useful for implementing various embodiments set forth in the disclosure.

[0015] Embodiments of the present disclosure will be described with reference to the accompanying drawings.DETAILED DESCRIPTION

[0016] Although specific configurations and arrangements are discussed, it should be understood that this is done for illustrative purposes only. A person skilled in the pertinent art will recognize that other configurations and arrangements can be used without departing fromthe spirit and scope of the present disclosure. It will be apparent to a person skilled in the pertinent art that the present disclosure can also be employed in a variety of other applications.

[0017] It is noted that references in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” “some embodiments,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it would be within the knowledge of a person skilled in the pertinent art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0018] In general, terminology may be understood at least in part from usage in context. For example, the term “one or more” as used herein, depending at least in part upon context, may be used to describe any feature, structure, or characteristic in a singular sense or may be used to describe combinations of features, structures or characteristics in a plural sense. Similarly, terms, such as “a,” “an,” or “the,” again, may be understood to convey a singular usage or to convey a plural usage, depending at least in part upon context. In addition, the term “based on” may be understood as not necessarily intended to convey an exclusive set of factors and may, instead, allow for existence of additional factors not necessarily expressly described, again, depending at least in part on context.

[0019] Various aspects of method and apparatus will now be described. These apparatus and methods will be described in the following detailed description and illustrated in the accompanying drawings by various blocks, modules, units, components, circuits, steps, operations, processes, algorithms, etc. (collectively referred to as “elements”). These elements may be implemented using electronic hardware, firmware, computer software, or any combination thereof. Whether such elements are implemented as hardware, firmware, or software depends upon the particular application and design constraints imposed on the overall system.

[0020] For ease of nomenclature, the term “camera” is used herein to refer to an image capture device or other data acquisition device. Such a data acquisition device can be any device or system for acquiring, recording, measuring, estimating, determining and / or computing data representative of a scene, including but not limited to two-dimensional image data, three-dimensional image data, and / or light field data. Such a data acquisition device may include optics, sensors, and image processing electronics for acquiring data representative of ascene, using techniques that are well-known in the art. One skilled in the art will recognize that many types of data acquisition devices can be used in connection with the present disclosure, and that the present disclosure is not limited to cameras. Thus, the use of the term “camera” herein is intended to be illustrative and exemplary, but should not be considered to limit the scope of the present disclosure. Specifically, any use of such term herein should be considered to refer to any suitable data acquisition device.

[0021] Image enhancement is one of the most important computer vision applications. Image enhancement units are generally deployed in all cameras, such as mobile phones or digital cameras. Image enhancement is a challenging problem since it consists of multiple units that perform various operations. These units may include, e.g., a super resolution unit, a denoising unit, and / or a high dynamic range (HDR) unit. Recently, deep learning (DL) neural networks (DNNs) have been deployed in image enhancement because they demonstrate significant accuracy improvements. For instance, multiple image data inputs of the same scene captured at slightly different times may be input into the DNN to generate an output-enhanced image. Furthermore, along with the usage of multi -camera devices, computer vision and image processing tasks with multi-camera have become even more important. An example of a multicamera device 102 is depicted in block diagram 100 of FIG. 1.

[0022] For instance, multi-camera device 102 may include a camera system 104 that includes a wide FOV image sensor 106a, a standard FOV image sensor 106b, a telescopic FOV image sensor 106c, and a microscopic FOV image sensor 106d. Each of the image sensors in camera system 104 may capture image data of different FOVs and / or resolutions.

[0023] Existing multi-input image enhancement systems rely on image data captured from each of the different cameras and process the image data respectively. For example, two or more images of a first scene captured by wide FOV image sensor 106a may be used to generate an enhanced image of the first scene, while two or more images of a second scene captured by the microscopic FOV image sensor 106d may be used to generate an enhanced image of the second scene. These multi-input image enhancement systems suffer from poor performance in handling the diversities between the image data inputs. The input images captured in the real-world may not be well aligned due to various factors, e.g., the motion of objects, illumination, etc. Existing multi-input image enhancement systems focus on motion only or apply pre-defined image processing technologies to pre-process the images. However, motion-only compensation and / or pre-defined image processing techniques are unable to align the image data inputs with a high-degree of accuracy. Consequently, enhanced images outputted by the existing multi-input image enhancement systems contain image artifacts, e.g.,such as object blur, among other issues. This results in poor performance and an inadequate user experience.

[0024] To overcome these and other challenges, the present disclosure provides an exemplary multi-input image enhancement system with a pipeline that generates enhanced images based on input images captured by different image sensors using different FOVs and resolutions. For example, first image data may be captured by a first image sensor at first FOV and a first resolution, and second image data may be captured by a second image sensor at a second FOV that is smaller than the first FOV and a second resolution that is greater than the first resolution. Thus, the first image data may include a larger FOV but a smaller object size, while the second image data may include a smaller FOV but a larger object size. The first image data and the second image data may be of a scene or an object. The multi-input image enhancement pipeline described herein applies feature extraction, feature matching, and aggregation / enhancement to the input image data to generate an enhanced image. In some embodiments, the enhanced image may include an image with the first FOV and the second resolution (or a third resolution greater than the first resolution but less than the second resolution. In some other embodiments, when the second image data contains blur or motion, the enhanced image may include the second FOV and is sharpened or motion-corrected using the first image data. Additional details of the exemplary multi-input image enhancement pipeline and its exemplary operations are provided below in connection with FIGs. 2-7.

[0025] FIG. 2 illustrates a block diagram of an exemplary image-enhancement pipeline 200, according to some embodiments of the present disclosure. FIG. 3 illustrates a detailed block diagram 300 of an exemplary feature extraction component 204 of the multi-input the image-enhancement pipeline 200 of FIG. 2, according to some embodiments of the present disclosure. FIG. 4 illustrates a block diagram 400 of an exemplary image-enhancement component 208 of the multi-input the image-enhancement pipeline 200 of FIG. 2, according to some embodiments of the present disclosure. FIGs. 2-4 will be described together.

[0026] Referring to FIG. 1, multi-input image enhancement pipeline 200 may generate an enhanced image 210 based on first image data 202a captured by a first image sensor (e.g., a wide FOV image sensor) using a first FOV and a first resolution and second image data 202b captured by a second image sensor (e.g., a telescopic FOV image sensor) using a second FOV smaller than the first FOV and a second resolution greater than the first resolution. To that end, multi-input image enhancement pipeline 200 may include, e.g., a feature extraction component 204, a feature matching component 206, and an image enhancement component 208. First image data 202a and second image data 202b may be input into feature extraction component204, and image enhancement component 208 may output enhanced image 210. Additional details of feature extraction component 204, feature matching component 206, and image enhancement component 208 now follow.

[0027] For example, feature extraction component 204 may extract contextual features from first image data 202a and second image data 202b, respectively. These contextual features may be extracted as feature maps. Referring to FIG. 3, feature extraction component 204 may be a residual channel attention block (RCAB) that includes a residual block 302, a channel attention block 304, and an element-wise adder 316. First image data 202a and second image data 202b may be input into residual block 302 and into element-wise adder 316. The residual block 302 may include a neural network with a convolutional layer 306, an activation layer 308 (e.g., Leaky Rectified Linear Unit (ReLU)), and another convolutional layer 306. Channel attention block 304 may include a neural network with a global pooling layer 310, convolutional layer 306, activation layer 308, and convolutional layer 306, as well as a sigmoid function unit 312 and an element-wise multiplier 314.

[0028] The output from residual block 302 may be input into the neural network of channel attention block 304 and into its element-wise multiplier 314. The output from the neural network at channel attention block 304 may be input into sigmoid function unit 312. A dot product may be applied to the output of sigmoid function unit 312 and the output of residual block 302. Element-wise adder 316 may combine first image data 202a with its corresponding first feature maps, while second image data 202b may be combined with its corresponding second feature maps. The first image data 202a, first feature maps associated with first image data 202a, second image data 202b, and second feature maps associated with second image data 202b may be output to feature matching component 206.

[0029] Referring again to FIG. 2, feature matching component 206 may generate correlation information that correlates first image data 202a and second image data 202b based on the first feature maps and the second feature maps received from feature extraction component 204. Feature matching component 206 may be implemented using cost volumebased image matching technology, holography, or feature point matching, just to name a few.

[0030] To generate the correlation information, feature matching component 206 may compare the first feature maps of first image data 202a and the second feature maps of second image data 202b. For example, feature matching component 206 may move the first feature maps (associated with first image data 202a) based on a pre-estimated motion. Then, the correlation information may be generated by calculating the correlation between the moved first feature maps and the unmoved second feature maps (associated with second image data202b). The ‘moving’ operation may include homography operations, e.g., such as shifting several pixels, scaling, or rotation, by way of example. If the pre-estimated motion is correct, the correlation will be maximum. The correlation information includes a set of matching volume feature map(s), which may be input into image enhancement component 208.

[0031] Referring to FIGs. 2 and 4, to generate enhanced image 210, image enhancement component 208 may aggregate multi-scale contextual information from the set of matching volume feature map(s) received from feature matching component 206. To that end, image enhancement component 208 may include a plurality of stacked hourglasses. The hourglasses of image enhancement component 208 are stacked in a sequential order, as shown in FIG. 4. Since image enhancement requires more accurate pixel-to-pixel mapping than semantic segmentation, image enhancement component 208 may use dilated convolution (e.g., three-dimensional convolution 402, three-dimensional convolution, stride 2 404, and three- dimensional deconvolution 406) rather than standard convolution to increase the receptive field. The dilation factors may increase as the hourglass goes deeper to increase the receptive field. The increasing dilation factors may enable the capture of denser multi-scale contextual information without losing the spatial resolution. By way of example and not limitation, the stacked hourglass architecture may include three main hourglass networks, where each hourglass may output an enhanced image. The first two hourglasses’ output intermediate results, which may be used for training purposes, while the final hourglass outputs enhanced image 210.

[0032] In some embodiments, the techniques described above in connection with FIGs. 2-4 may generate an enhanced image 210 with the first FOV and a resolution that is greater than the first resolution of first image data 202a. In other words, the first resolution of first image data 202a may be increased using second image data as reference data. Here, when first image data 202a is captured using a wide FOV image sensor and second image data 202b is captured using a telescopic FOV image sensor, enhanced image 210 may be referred to as a “tele-wide super resolution image.”

[0033] In some other embodiments, the techniques described above in connection with FIGs. 2-4 may generate an enhanced image 210 with the second FOV of second image data 202b using first image data 202a as reference data. This embodiment may be applicable to scenarios in which second image data 202b was captured with unexpected motion (e.g., blur) or an incorrect focus, thereby missing image information. Multi-input image enhancement pipeline 200 may retrieve the missing information using first image data 202a. Here, when first image data 202a is captured using a wide FOV image sensor and second image data 202bis captured using a telescopic FOV image sensor, enhanced image 210 may be generated using“tele-wide image retrieval.”

[0034] FIG. 5 illustrates an exemplary block diagram of an apparatus 500 having an image signal processor (ISP), according to some embodiments. Apparatus 500 may include an application processor (AP) 540, an ISP 520, a memory 518, and input-output devices 508. ISP 520 may include an image enhancement unit 502 and a camera 506. Input-output device 508 may include user input devices 510, inertial measurement unit (IMU) 512 (e.g., translational motion sensor, rotational motion sensor, accelerometer, gyroscope, etc.), display and audio devices 514, and wireless communication devices 516. In some embodiments, apparatus 500 may be an image-capturing device, such as a smartphone or digital camera.

[0035] AP 540 may be the main application process of apparatus 500, and may host the operating system (OS) of apparatus 500 and all the applications. AP 540 may be any kind of general-purpose processors such as a microprocessor, a microcontroller, a digital signal processor, or a central processing unit, and other needed integrated circuits such as glue logic. The term “processor” may refer to a device having one or more processing units or elements, e.g., a central processing unit (CPU) with multiple processing cores. AP 540 may be used to control the operations of apparatus 500 by executing instructions stored in memory 518, which can be in the same chip as AP 540 or in a separate chip from AP 540. AP 540 may also generate control signals and transmit them to various parts of apparatus 500 to control and monitor the operations of these parts. In some embodiments, AP 540 can run the OS of apparatus 500, control the communications between the user and apparatus 500, and control the operations of various applications. For example, AP 540 may be coupled to a communications circuitry and execute software to control the wireless communications functionality of apparatus 500. In another example, AP 540 may be coupled to ISP 520 and input-output devices 508 to control the processing and display of sensor data, e.g., image data, one or more frames, HDR images, low dynamic range (LDR) images, etc.

[0036] ISP 520 may include software and / or hardware operatively coupled to AP 540 and input-output devices 508. In some embodiments, components, e.g., circuitry, of ISP 520 may be integrated on a single chip. In some embodiments, ISP 520 includes an image processing hardware coupled to (e.g., placed between) AP 540 and image enhancement unit 502 / camera 506. ISP 520 may include a suitable circuitry that, when controlled by AP 540, performs functions not supported by AP 540, e.g., generating an enhanced image using the exemplary image enhancement model of multi-input image enhancement pipeline 200 described above in connection with FIGs. 2-4. In various embodiments, ISP 520 may includea field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a microprocessor, a microcontroller, a digital signal processor, and other needed integrated circuits for its purposes.

[0037] Image enhancement unit 502 may include multi-input image enhancement pipeline 200 of FIG. 2. For example, image enhancement unit 502 may receive first image data 202a captured by a first image sensor (e.g., a wide FOV image sensor of camera 506) using a first FOV and a first resolution and second image data 202b captured by a second image sensor (e.g., a telescopic FOV image sensor of camera 506) using a second FOV smaller than the first FOV and a second resolution greater than the first resolution. Image enhancement unit 502 may input first image data 202a and second image data 202b into its exemplary multi-input image enhancement pipeline 200 to generate enhanced image 210 using the techniques described above in connection with FIGs. 2-4.

[0038] FIG. 6 illustrates a flowchart of an exemplary method 600 of image enhancement, according to some embodiments of the present disclosure. Exemplary method 600 may be performed by a system, e.g., such as multi-input image enhancement pipeline 200, feature extraction component 204, feature matching component 206, image enhancement component 208, image enhancement unit 502, camera 506 (e.g., a wide FOV image sensor and a telescopic FOV image sensor), ISP 520, and / or computer system 700. Method 600 may include steps 602-612 as described below. It is to be appreciated that some of the steps may be optional, and some of the steps may be performed simultaneously, or in a different order than shown in FIG. 6.

[0039] Referring to FIG. 6, at 602, the system may receive, by a feature extraction component, first image data associated with a scene at a first FOV and a first resolution and second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution. For example, referring to FIGs. 2 and 5, feature extraction component 204 may receive first image data 202a captured by a first image sensor (e.g., a wide FOV image sensor of camera 506) using a first FOV and a first resolution and second image data 202b captured by a second image sensor (e.g., a telescopic FOV image sensor of camera 506) using a second FOV smaller than the first FOV and a second resolution greater than the first resolution.

[0040] At 604, the system may generate, by the feature extraction component, a set of feature maps associated with the first image data and the second image data. For example, referring to FIGs. 2 and 3, feature extraction component 204 may extract contextual features from first image data 202a and second image data 202b, respectively. These contextual featuresmay be extracted as feature maps. Referring to FIG. 3, feature extraction component 204 may be an RCAB that includes a residual block 302, a channel attention block 304, and an element- wise adder 316. First image data 202a and second image data 202b may be input into residual block 302 and into element-wise adder 316. The residual block 302 may include a neural network with a convolutional layer 306, an activation layer 308 (e.g., Leaky ReLU), and another convolutional layer 306. Channel attention block 304 may include a neural network with a global pooling layer 310, convolutional layer 306, activation layer 308, and convolutional layer 306, as well as a sigmoid function unit 312 and an element-wise multiplier 314. The output from residual block 302 may be input into the neural network of channel attention block 304 and into its element-wise multiplier 314. The output from the neural network at channel attention block 304 may be input into sigmoid function unit 312. A dot product may be applied to the output of sigmoid function unit 312 and the output of residual block 302. Element-wise adder 316 may combine first image data 202a with its corresponding first feature maps, while second image data 202b may be combined with its corresponding second feature maps.

[0041] At 606, the system may receive, by a feature matching component, the set of feature maps from the feature extraction component. For example, referring to FIG. 2, feature matching component 206 may receive the first image data 202a, first feature maps associated with first image data 202a, second image data 202b, and second feature maps associated with second image data 202b from feature extraction component 204.

[0042] At 608, the system may generate, by the feature matching component, correlation information based on the set of feature maps associated with the first image data and the second image data. For example, referring to FIG. 2, feature matching component 206 may generate correlation information that correlates first image data 202a and second image data 202b based on the first feature maps and the second feature maps received from feature extraction component 204. Feature matching component 206 may be implemented using cost volume-based image matching technology, holography, or feature point matching, just to name a few. To generate the correlation information, feature matching component 206 may compare the first feature maps of first image data 202a and the second feature maps of second image data 202b. For example, feature matching component 206 may move the first feature maps (associated with first image data 202a) based on a pre-estimated motion. Then, the correlation information may be generated by calculating the correlation between the moved first feature maps and the unmoved second feature maps (associated with second image data 202b). The ‘moving’ operation may include homography operations, e.g., such as shifting several pixels,scaling, or rotation, by way of example. If the pre-estimated motion is correct, the correlation will be maximum. The correlation information includes a set of matching volume feature map(s).

[0043] At 610, the system may receive, by an image enhancement component, the correlation information associated with the first image data and the second image data from the feature matching component. For example, referring to FIGs. 2 and 4, a set of matching volume feature map(s) generated by feature matching component 206 may be received by image enhancement component 208.

[0044] At 612, the system may generate, by the image enhancement component, an enhanced image of the scene based on the correlation information. For example, referring to FIG. 2, to generate enhanced image 210, image enhancement component 208 may aggregate multi-scale contextual information from the set of matching volume feature map(s) received from feature matching component 206. To that end, image enhancement component 208 may include a plurality of stacked hourglasses. The hourglasses of image enhancement component 208 are stacked in a sequential order, as shown in FIG. 4. Since image enhancement requires more accurate pixel-to-pixel mapping than semantic segmentation, image enhancement component 208 may use dilated convolution (e.g., three-dimensional convolution 402, three- dimensional convolution, stride 2 404, and three-dimensional deconvolution 406) rather than standard convolution to increase the receptive field. The dilation factors may increase as the hourglass goes deeper to increase the receptive field. The increasing dilation factors may enable the capture of denser multi-scale contextual information without losing the spatial resolution. By way of example and not limitation, the stacked hourglass architecture may include three main hourglass networks, where each hourglass may output an enhanced image. The first two hourglasses’ output intermediate results, which may be used for training purposes, while the final hourglass outputs enhanced image 210.

[0045] Various embodiments can be implemented, for example, using one or more computer systems, such as computer system 700 shown in FIG. 7. One or more computer system 700 can be used, for example, to implement method 600 of FIG. 6. For example, computer system 700 can generate an enhanced image based on first image data captured by a first image sensor using a first FOV and first resolution and second image data captured by a second image sensor using a second FOV and second resolution, according to various embodiments. Computer system 700 can be any computer capable of performing the functions described herein.

[0046] Computer system 700 can be any well-known computer capable of performingthe functions described herein. Computer system 700 includes one or more processors (also called central processing units, or CPUs), such as a processor 704. Processor 704 is connected to a communication infrastructure 706 (e.g., a bus). One or more processors 704 may each be a graphics processing unit (GPU). In an embodiment, a GPU is a processor that is a specialized electronic circuit designed to process mathematically intensive applications. The GPU may have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, videos, etc.

[0047] Computer system 700 also includes user input / output device(s) 703, such as monitors, keyboards, pointing devices, etc., that communicate with communication infrastructure 706 through user input / output interface(s) 702.

[0048] Computer system 700 also includes a main (or primary) memory 708, such as random-access memory (RAM). Main memory 708 may include one or more levels of cache. Main memory 708 has stored therein control logic (i.e., computer software) and / or data. Computer system 700 may also include one or more secondary storage devices or memory 710. Secondary memory 710 may include, for example, a hard disk drive 712 and / or a removable storage device or drive 714. Removable storage drive 714 may be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, tape backup device, and / or any other storage device / drive. Removable storage drive 714 may interact with a removable storage unit 716. Removable storage unit 716 includes a computer usable or readable storage device having stored thereon computer software (control logic) and / or data. Removable storage unit 716 may be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and / any other computer data storage device. Removable storage drive 714 reads from and / or writes to removable storage unit 716 in a well-known manner.

[0049] According to an exemplary embodiment, secondary memory 710 may include other means, instrumentalities or other approaches for allowing computer programs and / or other instructions and / or data to be accessed by computer system 700. Such means, instrumentalities or other approaches may include, for example, a removable storage unit 722 and an interface 720. Examples of the removable storage unit 722 and the interface 720 may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and universal serial bus (USB) port, a memory card and associated memory card slot, and / or any other removable storage unit and associated interface.

[0050] Computer system 700 may further include a communication (or network) interface 724. Communication interface 724 enables computer system 700 to communicate andinteract with any combination of remote devices, remote networks, remote entities, etc. (individually and collectively referenced as 726). For example, communication interface 724 may allow computer system 700 to communicate with remote devices 726 over communication path 728, which may be wired and / or wireless, and which may include any combination of LANs, WANs, the Internet, etc. Control logic and / or data may be transmitted to and from computer system 700 via communication path 728.

[0051] In an embodiment, a tangible apparatus or article of manufacture comprising a tangible computer useable or readable medium having control logic (software) stored thereon is also referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system 700, main memory 708, secondary memory 710, and removable storage units 716 and 722, as well as tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system 700), causes such data processing devices to operate as described herein.

[0052] Based on the teachings contained in this disclosure, it will be apparent to persons skilled in the relevant art(s) how to make and use embodiments of the present disclosure using data processing devices, computer systems and / or computer architectures other than that shown in FIG. 7. For example, embodiments may operate with software, hardware, and / or operating system implementations other than those described herein.

[0053] In various aspects of the present disclosure, the functions described herein may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or encoded as instructions or code on a non- transitory computer-readable medium. Computer-readable media includes computer storage media. Storage media may be any available media that can be accessed by a computing device, such as multi-input image enhancement pipeline 200 in FIG. 2. By way of example, and not limitation, such computer-readable media can include random-access memory (RAM), readonly memory (ROM), EEPROM, compact disc read-only memory (CD-ROM) or other optical disk storage, hard disk drive (HDD), such as magnetic disk storage or other magnetic storage devices, Flash drive, solid state drive (SSD), or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a processing system, such as a mobile device or a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and floppy disk where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope ofcomputer-readable media.

[0054] According to one aspect of the present disclosure, an image-enhancement system is provided. The image-enhancement system may include a feature extraction component. The feature extraction component may be configured to receive first image data associated with a scene at a first FOV and a first resolution and second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution. The feature extraction component may be configured to generate a set of feature maps associated with the first image data and the second image data. The imageenhancement system may include a feature matching component. The feature matching component may be configured to receive the set of feature maps from the feature extraction component. The feature matching component may be configured to generate correlation information based on the set of feature maps associated with the first image data and the second image data. The image-enhancement system may include an image enhancement component. The image enhancement component may be configured to receive the correlation information associated with the first image data and the second image data from the feature matching component. The image enhancement component may be configured to generate an enhanced image of the scene based on the correlation information.

[0055] In some embodiments, the feature extraction component may include a residual block comprising at least one first convolutional layer and a first activation layer. In some embodiments, the feature extraction component may include a channel attention block comprising a global pooling layer, a second activation layer, and at least one second convolutional layer. In some embodiments, the feature extraction component may include an element-wise summation component following the channel attention block.

[0056] In some embodiments, the channel attention block may further include a sigmoid function component and an element-wise dot product component.

[0057] In some embodiments, to generate the correlation information based on the set of feature maps associated with the first image data and the second image data, the feature matching component may be configured to apply homograph operations to first feature maps associated with the first image data based on a pre-estimated motion. In some embodiments, to generate the correlation information based on the set of feature maps associated with the first image data and the second image data, the feature matching component may be configured to generate the correlation information based on a comparison of the first feature maps associated with the first image data and second feature maps associated with the second image data after the homography operations are applied to the first feature maps.

[0058] In some embodiments, to generate the enhanced image of the scene based on the correlation information, the image enhancement component may be configured to input the correlation information into a plurality of hourglass convolutional networks. In some embodiments, to generate the enhanced image of the scene based on the correlation information, the image enhancement component may be configured to perform, by each of the plurality of hourglass convolutional networks, a dilated convolution with dilation factors that increase with each layer. In some embodiments, to generate the enhanced image of the scene based on the correlation information, the image enhancement component may be configured to output, by a final hourglass convolutional network, the enhanced image of the scene.

[0059] In some embodiments, the enhanced image of the scene may be generated at the first FOV and with a third resolution greater than the first resolution.

[0060] In some embodiments, the enhanced image of the scene may be generated at the second FOV.

[0061] According to another aspect of the present disclosure, an ISP is provided. The ISP may include a first image sensor configured to capture first image data associated with a scene at a first FOV and a first resolution. The ISP may include a second image sensor configured to capture second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution. The ISP may include a feature extraction component. The feature extraction component may be configured to receive first image data associated with a scene at a first FOV and a first resolution and second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution. The feature extraction component may be configured to generate a set of feature maps associated with the first image data and the second image data. The ISP may include a feature matching component. The feature matching component may be configured to receive the set of feature maps from the feature extraction component. The feature matching component may be configured to generate correlation information based on the set of feature maps associated with the first image data and the second image data. The ISP may include an image enhancement component. The image enhancement component may be configured to receive the correlation information associated with the first image data and the second image data from the feature matching component. The image enhancement component may be configured to generate an enhanced image of the scene based on the correlation information.

[0062] In some embodiments, the feature extraction component may include a residual block comprising at least one first convolutional layer and a first activation layer. In someembodiments, the feature extraction component may include a channel attention block comprising a global pooling layer, a second activation layer, and at least one second convolutional layer. In some embodiments, the feature extraction component may include an element-wise summation component following the channel attention block.

[0063] In some embodiments, the channel attention block may further include a sigmoid function component and an element-wise dot product component.

[0064] In some embodiments, to generate the correlation information based on the set of feature maps associated with the first image data and the second image data, the feature matching component may be configured to apply homograph operations to first feature maps associated with the first image data based on a pre-estimated motion. In some embodiments, to generate the correlation information based on the set of feature maps associated with the first image data and the second image data, the feature matching component may be configured to generate the correlation information based on a comparison of the first feature maps associated with the first image data and second feature maps associated with the second image data after the homography operations are applied to the first feature maps.

[0065] In some embodiments, to generate the enhanced image of the scene based on the correlation information, the image enhancement component may be configured to input the correlation information into a plurality of hourglass convolutional networks. In some embodiments, to generate the enhanced image of the scene based on the correlation information, the image enhancement component may be configured to perform, by each of the plurality of hourglass convolutional networks, a dilated convolution with dilation factors that increase with each layer. In some embodiments, to generate the enhanced image of the scene based on the correlation information, the image enhancement component may be configured to output, by a final hourglass convolutional network, the enhanced image of the scene.

[0066] In some embodiments, the enhanced image of the scene may be generated at the first FOV and with a third resolution greater than the first resolution.

[0067] In some embodiments, the enhanced image of the scene may be generated at the second FOV.

[0068] According to a further aspect of the present disclosure, a method of image enhancement is provided. The method may include receiving, by a feature extraction component, first image data associated with a scene at a first FOV and a first resolution and second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution. The method may include generating, by the feature extraction component, a set of feature maps associated with the first image data and thesecond image data. The method may include receiving, by a feature matching component, the set of feature maps from the feature extraction component. The method may include generating, by the feature matching component, correlation information based on the set of feature maps associated with the first image data and the second image data. The method may include receiving, by an image enhancement component, the correlation information associated with the first image data and the second image data from the feature matching component. The method may include generating, by the image enhancement component, an enhanced image of the scene based on the correlation information.

[0069] In some embodiments, the generating, by the feature extraction component, the set of feature maps associated with the first image data and the second image data may include inputting the first image data and the second image data into a residual block comprising at least one first convolutional layer and a first activation layer, a channel attention block comprising a global pooling layer, a second activation layer, and at least one second convolutional layer, and an element-wise summation component following the channel attention block. In some embodiments, the generating, by the feature extraction component, the set of feature maps associated with the first image data and the second image data may include obtaining the set of feature maps as an output of the element-wise summation component.

[0070] In some embodiments, the generating, by the feature matching component, the correlation information based on the set of feature maps associated with the first image data and the second image data may include applying homograph operations to first feature maps associated with the first image data based on a pre-estimated motion. In some embodiments, the generating, by the feature matching component, the correlation information based on the set of feature maps associated with the first image data and the second image data may include generating the correlation information based on a comparison of the first feature maps associated with the first image data and second feature maps associated with the second image data after the homography operations are applied to the first feature maps.

[0071] In some embodiments, the generating, by the image enhancement component, the enhanced image of the scene based on the correlation information may include inputting the correlation information into a plurality of hourglass convolutional networks. In some embodiments, the generating, by the image enhancement component, the enhanced image of the scene based on the correlation information may include performing, by each of the plurality of hourglass convolutional networks, a dilated convolution with dilation factors that increase with each layer. In some embodiments, the generating, by the image enhancement component,the enhanced image of the scene based on the correlation information may include outputting, by a final hourglass convolutional network, the enhanced image of the scene.

[0072] In some embodiments, the enhanced image of the scene may be generated at the first FOV and with a third resolution greater than the first resolution.

[0073] In some embodiments, the enhanced image of the scene may be generated at the second FOV.

[0074] Embodiments of the present disclosure have been described above with the aid of functional building blocks illustrating the implementation of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.

[0075] The Summary and Abstract sections may set forth one or more but not all exemplary embodiments of the present disclosure as contemplated by the inventor(s), and thus, are not intended to limit the present disclosure and the appended claims in any way.

[0076] Various functional blocks, modules, and steps are disclosed above. The particular arrangements provided are illustrative and without limitation. Accordingly, the functional blocks, modules, and steps may be re-ordered or combined in different ways than in the examples provided above. Likewise, some embodiments include only a subset of the functional blocks, modules, and steps, and any such subset is permitted.

[0077] The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

WHAT IS CLAIMED IS1. An image-enhancement system, comprising: a feature extraction component configured to: receive first image data associated with a scene at a first field-of-view (FOV) and a first resolution and second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution; and generate a set of feature maps associated with the first image data and the second image data; a feature matching component configured to: receive the set of feature maps from the feature extraction component; and generate correlation information based on the set of feature maps associated with the first image data and the second image data; an image enhancement component configured to: receive the correlation information associated with the first image data and the second image data from the feature matching component; and generate an enhanced image of the scene based on the correlation information.

2. The image-enhancement system of claim 1, wherein the feature extraction component comprises: a residual block comprising at least one first convolutional layer and a first activation layer; a channel attention block comprising a global pooling layer, a second activation layer, and at least one second convolutional layer; and an element-wise summation component following the channel attention block.

3. The image-enhancement system of claim 2, wherein the channel attention block further comprises: a sigmoid function component and an element-wise dot product component.

4. The image-enhancement system of claim 1, wherein, to generate the correlation information based on the set of feature maps associated with the first image data and the second image data, the feature matching component is configured to: apply homograph operations to first feature maps associated with the first image data based on a pre-estimated motion; andgenerate the correlation information based on a comparison of the first feature maps associated with the first image data and second feature maps associated with the second image data after the homography operations are applied to the first feature maps.

5. The image-enhancement system of claim 1, wherein, to generate the enhanced image of the scene based on the correlation information, the image enhancement component is configured to: input the correlation information into a plurality of hourglass convolutional networks; perform, by each of the plurality of hourglass convolutional networks, a dilated convolution with dilation factors that increase with each layer; and output, by a final hourglass convolutional network, the enhanced image of the scene.

6. The image-enhancement system of claim 1, wherein the enhanced image of the scene is generated at the first FOV and with a third resolution greater than the first resolution.

7. The image-enhancement system of claim 1, wherein the enhanced image of the scene is generated at the second FOV.

8. An image signal processor (ISP), comprising: a first image sensor configured to: capture first image data associated with a scene at a first field-of-view (FOV) and a first resolution; a second image sensor configured to: capture second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution; a feature extraction component configured to: receive the first image data from the first image sensor and the second image data from the second image sensor; and generate a set of feature maps associated with the first image data and the second image data; a feature matching component configured to: receive the set of feature maps from the feature extraction component; and generate correlation information based on the set of feature maps associated with the first image data and the second image data;an image enhancement component configured to: receive the correlation information associated with the first image data and the second image data from the feature matching component; and generate an enhanced image of the scene based on the correlation information.

9. The ISP of claim 8, wherein the feature extraction component comprises: a residual block comprising at least one first convolutional layer and a first activation layer; a channel attention block comprising a global pooling layer, a second activation layer, and at least one second convolutional layer; and an element-wise summation component following the channel attention block.

10. The ISP of claim 9, wherein the channel attention block further comprises: a sigmoid function component and an element-wise dot product component.

11. The ISP of claim 8, wherein, to generate the correlation information based on the set of feature maps associated with the first image data and the second image data, the feature matching component is configured to: apply homograph operations to first feature maps associated with the first image data based on a pre-estimated motion; and generate the correlation information based on a comparison of the first feature maps associated with the first image data and second feature maps associated with the second image data after the homography operations are applied to the first feature maps.

12. The ISP of claim 8, wherein, to generate the enhanced image of the scene based on the correlation information, the image enhancement component is configured to: input the correlation information into a plurality of hourglass convolutional networks; perform, by each of the plurality of hourglass convolutional networks, a dilated convolution with dilation factors that increase with each layer; and output, by a final hourglass convolutional network, the enhanced image of the scene.

13. The ISP of claim 8, wherein the enhanced image of the scene is generated at the first FOV and with a third resolution greater than the first resolution.

14. The ISP of claim 8, wherein the enhanced image of the scene is generated at the second FOV.

15. A method of image enhancement, comprising: receiving, by a feature extraction component, first image data associated with a scene at a first field-of-view (FOV) and a first resolution and second image data associated with the scene at a second FOV smaller than the first FOV and a second resolution greater than the first resolution; generating, by the feature extraction component, a set of feature maps associated with the first image data and the second image data; receiving, by a feature matching component, the set of feature maps from the feature extraction component; generating, by the feature matching component, correlation information based on the set of feature maps associated with the first image data and the second image data; receiving, by an image enhancement component, the correlation information associated with the first image data and the second image data from the feature matching component; and generating, by the image enhancement component, an enhanced image of the scene based on the correlation information.

16. The method of claim 15, wherein the generating, by the feature extraction component, the set of feature maps associated with the first image data and the second image data comprises: inputting the first image data and the second image data into a residual block comprising at least one first convolutional layer and a first activation layer, a channel attention block comprising a global pooling layer, a second activation layer, and at least one second convolutional layer, and an element-wise summation component following the channel attention block; and obtaining the set of feature maps as an output of the element-wise summation component.

17. The method of claim 15, wherein the generating, by the feature matching component, the correlation information based on the set of feature maps associated with the first image data and the second image data comprises: applying homograph operations to first feature maps associated with the first imagedata based on a pre-estimated motion; and generating the correlation information based on a comparison of the first feature maps associated with the first image data and second feature maps associated with the second image data after the homography operations are applied to the first feature maps.

18. The method of claim 15, wherein the generating, by the image enhancement component, the enhanced image of the scene based on the correlation information comprises: inputting the correlation information into a plurality of hourglass convolutional networks; performing, by each of the plurality of hourglass convolutional networks, a dilated convolution with dilation factors that increase with each layer; and outputting, by a final hourglass convolutional network, the enhanced image of the scene.

19. The method of claim 15, wherein the enhanced image of the scene is generated at the first FOV and with a third resolution greater than the first resolution.

20. The method of claim 15, wherein the enhanced image of the scene is generated at the second FOV.

Citation Information

Patent Citations

  • Cascaded architecture for disparity and motion prediction with block matching and convolutional neural network (CNN)

    US20210192752A1

  • Method for image segmentation using cnn

    US20210248761A1

  • Estimating camera motion through visual tracking in low contrast high motion single camera systems

    US20220282978A1

  • Reference-Based Super-Resolution for Image and Video Enhancement

    US20230098437A1

Cited By

  • Image data enhancement method and system applied to maneuvering command screen

    CN121481855A