Image processing methods, apparatus and electronic equipment

Abstract

This disclosure provides an image processing method, apparatus, and electronic device. The method includes: acquiring an input image in a first color space; converting the input image in the first color space to an input image in a second color space; extracting a first channel image from the input image in the first color space and a second channel image from the input image in the second color space; training a branch lookup table for channels in the first color space and channels in the second color space based on an image enhancement task; inputting the first channel image and the second channel image into the branch lookup table to obtain an enhanced first channel image and an enhanced second channel image; and performing channel fusion on the enhanced first channel image and the enhanced second channel image to obtain an enhanced image of the input image. Therefore, by using a lookup table, the computational power requirements of the edge device can be reduced, and the output image can be flexibly fused from multiple branch lookup tables for multiple channels according to task requirements.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, particularly to image enhancement and image generation, and especially to an image processing method, apparatus, electronic circuit, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] In edge devices (such as mobile phones, drones, vehicle cameras, embedded security cameras, and smart doorbells), computing power is often limited, storage resources are restricted, and power consumption requirements are stringent. Meanwhile, with the diversification of imaging sensor and camera module resolutions and usage scenarios, common image enhancement tasks (such as super-resolution enhancement, noise reduction, rain removal, fog removal, and low-light scene enhancement) are beginning to become practical needs on the edge. Summary of the Invention

[0003] This disclosure provides an image processing method, apparatus, electronic circuit, electronic device, computer-readable storage medium, and computer program product.

[0004] According to one aspect of this disclosure, an image processing method is provided, the method comprising: acquiring an input image in a first color space; converting the input image in the first color space into an input image in a second color space; extracting a plurality of first channel images of the input image in the first color space and a plurality of second channel images of the input image in the second color space; training a plurality of branch lookup tables for one or more channels of the plurality of channels in the first color space and the plurality of channels in the second color space based on one or more image enhancement tasks; inputting the plurality of first channel images and the plurality of second channel images into the plurality of branch lookup tables respectively to obtain a plurality of enhanced first channel images and a plurality of enhanced second channel images; and performing channel fusion on at least two of the plurality of enhanced first channel images and the plurality of enhanced second channel images to obtain one or more enhanced images of the input image corresponding to one or more image enhancement tasks.

[0005] In some embodiments, the receptive field size of the multiple branch lookup tables is the same, and the receptive field size is the size of the image region perceived by a single query of the multiple branch lookup tables.

[0006] In some embodiments, the input image in the first color space is segmented into multiple input sub-images in the first color space and stored in an input buffer. The input buffer is used for query operations on all branch lookup tables, wherein the resolution of the multiple input sub-images is the same as the receptive field size of the multiple branch lookup tables.

[0007] In some embodiments, one or more enhanced images undergo color space conversion for display or subsequent image processing tasks.

[0008] In some embodiments, the first color space is the RGB color space and the second color space is the YCbCr color space.

[0009] In some embodiments, one or more image enhancement tasks include at least one of super-resolution enhancement, denoising, deraining, and dehazing of the input image.

[0010] In some embodiments, the pixel values ​​of a plurality of first channel images, a plurality of second channel images, and a plurality of branch lookup tables are quantized to reduce the size of the plurality of branch lookup tables.

[0011] In some embodiments, multiple branch lookup tables can be obtained through individual training or joint training.

[0012] In some embodiments, the multiple enhanced first channel images and the multiple enhanced second channel images have the same resolution.

[0013] In some embodiments, channel fusion of at least two images from a plurality of enhanced first channel images and a plurality of enhanced second channel images to obtain one or more enhanced images of the input image corresponding to one or more image enhancement tasks includes: channel stitching of at least two images from a plurality of enhanced first channel images and a plurality of enhanced second channel images to obtain one or more channel stitched images; and convolving the stitched channel components of the one or more channel stitched images to reduce the number of channels to obtain one or more enhanced images of the input image corresponding to one or more image enhancement tasks.

[0014] According to one aspect of this disclosure, an image processing apparatus is provided, comprising: a first module configured to acquire an input image in a first color space; a second module configured to convert the input image in the first color space into an input image in a second color space; a third module configured to extract a plurality of first channel images of the input image in the first color space and a plurality of second channel images of the input image in the second color space; a fourth module configured to train a plurality of branch lookup tables for one or more channels of the plurality of channels in the first color space and the plurality of channels in the second color space based on one or more image enhancement tasks; a fifth module configured to input the plurality of first channel images and the plurality of second channel images into the plurality of branch lookup tables respectively to obtain a plurality of enhanced first channel images and a plurality of enhanced second channel images; and a sixth module configured to perform channel fusion on at least two of the plurality of enhanced first channel images and the plurality of enhanced second channel images to obtain one or more enhanced images of the input image corresponding to one or more image enhancement tasks.

[0015] According to another aspect of this disclosure, an electronic circuit is provided, including circuitry configured to perform the steps of the methods provided above.

[0016] According to another aspect of this disclosure, an electronic device is provided, including a processor; and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the steps of the method provided above in this disclosure.

[0017] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0018] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0019] According to one or more embodiments of this disclosure, replacing the image augmentation neural network model with a lookup table can reduce the computing power requirements of the edge device and enable flexible fusion of the output image of multiple branches of the lookup table for multiple channels according to task requirements.

[0020] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0021] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0022] Figure 1 This is a flowchart illustrating an image processing method 100 according to an exemplary embodiment.

[0023] Figure 2 This is a detailed flowchart illustrating an exemplary process 200 of an image processing method 100 according to an exemplary embodiment.

[0024] Figure 3 This is a block diagram illustrating an image processing apparatus 300 according to an exemplary embodiment.

[0025] Figure 4 This is a block diagram illustrating an exemplary electronic device 400 that can be applied to an exemplary embodiment. Detailed Implementation

[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0028] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0029] In edge devices (such as mobile phones, drones, vehicle cameras, embedded security cameras, smart doorbells, etc.), computing power is often limited, storage resources are restricted, and power consumption requirements are strict. At the same time, with the diversification of imaging sensor and camera module resolutions and usage scenarios, common image enhancement tasks (such as super-resolution enhancement, noise reduction, rain removal, fog removal, and enhancement of low-light scenes) are beginning to become practical needs on the edge.

[0030] In edge devices, storage (e.g., internal RAM, cache, NVM), computing power, and power consumption are key bottlenecks. Even though lookup table (LUT) lookup costs less than CNN operations, if the LUT is too large (e.g., tens or even hundreds of MB), it may not fit into the device cache or high-speed access area, and may also affect latency and power consumption. Furthermore, image enhancement often involves multiple tasks (super-resolution, denoising, deraining, dehazing) sharing the same input source (camera capture). If each task requires a separate set of huge LUTs for storage, it will further increase the resource burden. Therefore, there is a need for an image processing method that can reuse inputs or indexes, reduce storage, meet the needs of multi-task enhancement, and is suitable for devices with low computing power.

[0031] The embodiments of this disclosure provide an image processing method that reduces the computing power requirements of edge devices by replacing the image augmentation neural network model with a lookup table, and enables flexible fusion of the output image of multiple branches of the lookup table for multiple channels according to task requirements.

[0032] Figure 1 This is a flowchart illustrating an image processing method 100 according to an exemplary embodiment.

[0033] like Figure 1 As shown, this disclosure proposes an image processing method 100, including the following steps: S102, acquiring an input image in a first color space; S104, converting the input image in the first color space into an input image in a second color space; S106, extracting multiple first channel images of the input image in the first color space and multiple second channel images of the input image in the second color space; S108, training multiple branch lookup tables for one or more channels of the multiple channels in the first color space and the multiple channels in the second color space based on one or more image enhancement tasks; S110, inputting the multiple first channel images and the multiple second channel images into the multiple branch lookup tables respectively to obtain multiple enhanced first channel images and multiple enhanced second channel images; and S112, performing channel fusion on at least two images of the multiple enhanced first channel images and the multiple enhanced second channel images to obtain one or more enhanced images of the input image corresponding to one or more image enhancement tasks.

[0034] In step S102, the input image in the first color space is obtained.

[0035] In the example, the input image can be obtained from an edge device, which may include, for example, a mobile phone, a drone, a vehicle camera, an embedded security camera, a smart doorbell, etc. These devices can capture images or videos, but their storage space, computing power, latency, and power consumption are all limited. Therefore, when such devices process multiple image enhancement tasks that have high requirements for storage space, computing power, latency, and power consumption, they often run into bottlenecks.

[0036] In the example, the first color space can be any color space such as RGB, YUV, LAB, HSV, or HSV, which can be determined according to the attributes of the end device or user settings, and is not restricted here.

[0037] In step S104, the input image in the first color space is converted into an input image in the second color space.

[0038] In the example, the second color space can be a different color space from the first color space, which can be determined based on the image enhancement task and is not restricted here.

[0039] In some embodiments, the first color space is the RGB color space, and the second color space is the YCbCr color space. Images or videos acquired by the edge device can be in the RGB color space, where R represents the red channel, G represents the green channel, and B represents the blue channel. These RGB images or videos can then be converted to other color spaces more conducive to image enhancement, such as YCbCr, where Y represents the luminance channel and CbCr represents the chroma / saturation channel. Thus, the luminance channel and chroma / saturation channel are decoupled through color space conversion, facilitating feature extraction for different task requirements.

[0040] In step S106, multiple first channel images of the input image in the first color space and multiple second channel images of the input image in the second color space are extracted.

[0041] Continuing with the example of using RGB as the first color space and YCbCr as the second color space, we can extract the R, G, and B channels of the input image in the RGB color space, and the Y and CbCr channels of the input image in the YCbCr color space. It's important to note that a channel image can correspond to one or more channels; for example, a CbCr channel image corresponds to both the Cb and Cr channels.

[0042] In step S108, a multi-branch lookup table is trained based on one or more image enhancement tasks for one or more channels in a first color space and a second color space.

[0043] In some embodiments, one or more image enhancement tasks may include at least one of super-resolution enhancement, denoising, deraining, and dehazing of the input image. It should be noted that the above image enhancement tasks are for illustrative purposes only and not for limiting purposes.

[0044] In some embodiments, multiple branch lookup tables can be obtained through individual training or joint training.

[0045] In the example, a joint training strategy can be used when a multi-branch lookup table fusion output is required. For a single task, the result of the lookup table network is directly output. After training, the correspondence between the input image and the output image (i.e., the augmented image) is obtained based on the frozen network weights, forming a lookup table for inference (search).

[0046] Therefore, training methods can be flexibly set for various image enhancement tasks.

[0047] In some examples, taking RGB as the first color space and YCbCr as the second color space as an example, a branch lookup table can be trained for the Y channel and CbCr channel of the second color space YCbCr, and a branch lookup table can be trained for the R channel, G channel and B channel of the first color space RGB.

[0048] In some examples, lookup tables tailored to the characteristics of different channels can be trained for specific image enhancement tasks. For instance, the human eye is highly sensitive to brightness information, so the Y channel can contain a large amount of high-frequency structural information in an image. Therefore, the Y channel can be used to extract features such as texture, edges, and blocks, and a branch LUT for detail enhancement and denoising can be trained for the Y channel. Similarly, a branch LUT for color restoration and denoising can be trained for the CbCr channel to adjust the color consistency of the image, remove color cast, and correct illumination. In the RGB domain, a branch LUT is trained for other enhancement processing.

[0049] It's important to note that branch lookup tables don't necessarily have to correspond one-to-one with each channel. One branch lookup table can correspond to multiple channels (for example, it can correspond to both Cb and Cr channels simultaneously). Different branch lookup tables used for different image enhancement tasks can also correspond to the same channels; there are no restrictions here. For example, for the Y channel, two LUTs can be trained for denoising and super-resolution enhancement, respectively.

[0050] In step S110, multiple first-channel images and multiple second-channel images are input into multiple branch lookup tables to obtain multiple enhanced first-channel images and multiple enhanced second-channel images.

[0051] In the example, by performing parallel query operations on multiple first-channel images and multiple second-channel images in multiple branch lookup tables, multiple enhanced first-channel images and multiple enhanced second-channel images can be obtained.

[0052] Therefore, by directly querying the branch lookup table to obtain the corresponding enhanced image, rather than through real-time computation of the enhanced image algorithm, the computing power requirements of the edge device can be reduced, thus enabling large-scale independent real-time image enhancement processing on various edge devices without the need for backend server involvement.

[0053] It should be noted that the end-side device can directly output one or more enhanced channel images according to task requirements, or it can output them after performing step S112.

[0054] In some embodiments, the receptive field size of the multiple branch lookup tables is the same, and the receptive field size is the size of the image region perceived by a single query of the multiple branch lookup tables.

[0055] In the example, the size of the receptive field can be determined by the network structure. For instance, a lookup table requires 3x3 pixels to infer a result; 3x3 is the receptive field, representing the size of a portion of the input image perceived during a single lookup operation. The output result after a lookup operation can vary depending on the image enhancement task. If the image enhancement task is super-resolution enhancement (e.g., a magnification of 2x2), the output will be 4 values ​​(or pixels); if the image enhancement task is denoising, the output will be 1 value (or pixel).

[0056] Generally speaking, the size of the receptive field of a lookup table directly affects the size of the lookup table, and the resolution of the input image of the lookup table needs to be greater than or equal to the size of the receptive field of the lookup table to ensure the integrity of the lookup table.

[0057] In the example, having the same receptive field size for multiple branch lookup tables allows the use of the same resolution input image during training and inference (i.e., querying), avoiding problems such as resolution inconsistency, increased fusion complexity, and training instability caused by receptive field differences.

[0058] In some embodiments, the input image in the first color space is segmented into multiple input sub-images in the first color space and stored in an input buffer. The input buffer is used for query operations on all branch lookup tables, wherein the resolution of the multiple input sub-images is the same as the receptive field size of the multiple branch lookup tables.

[0059] In the example, the input buffer can store all combinations of multiple input sub-image patches and index values. The index values ​​can be used for direct table lookup, avoiding complex calculations and thus reducing real-time computing overhead.

[0060] Taking RGB as the primary color space as an example, the RGB input image is segmented into multiple RGB sub-images of the same resolution as the receptive field size of the multiple branch lookup tables and cached. Thus, all branch lookup tables can share or reuse this RGB input buffer without occupying additional storage space, avoiding a linear increase in storage space due to implementing multiple enhancement tasks. It should be understood that the input buffer can also be in any other color space; this is not restricted here.

[0061] In the example, the input image can be used to obtain sub-images using a sliding window method, and the size of the sliding window can be the same as the receptive field resolution.

[0062] In some embodiments, method 100 further includes: quantizing the pixel values ​​of a plurality of first channel images, a plurality of second channel images, and a plurality of branch lookup tables to reduce the size of the plurality of branch lookup tables.

[0063] The image pixel value range is 0-255. Quantizing the pixel values ​​in the branch lookup table (correspondingly, multiple first channel images and multiple second channel images also need to be quantized) can reduce the size of the branch lookup table. This will reduce the output image quality, but it can improve the lookup speed, save storage space, and is beneficial for large-scale application on edge devices. Therefore, the specific quantization accuracy can be selected according to the actual task requirements.

[0064] In step S112, at least two images from multiple enhanced first channel images and multiple enhanced second channel images are fused to obtain one or more enhanced images of the input image corresponding to one or more image enhancement tasks.

[0065] In the example, multiple enhanced first-channel images and multiple enhanced second-channel images can be sampled or interpolated to make them have the same resolution before channel fusion.

[0066] In some examples, image enhancement tasks may only need to be processed in one color gamut, rather than both. For instance, in the YCbCr color space, when super-resolution enhancement is required, it is sufficient to interpolate and enlarge the Y and CbCr channel images, and then merge the enlarged Y and CbCr channel images.

[0067] In the example, taking the first color space as RGB and the second color space as YCbCr, multi-branch feature fusion can be performed on the output image of the Y channel branch lookup table, the CbCr channel branch lookup table and multiple branch lookup tables of the RGB domain, so as to achieve complementarity of brightness, color and original RGB features, improve the overall enhancement effect and multi-task adaptability.

[0068] In some embodiments, the multiple enhanced first channel images and the multiple enhanced second channel images have the same resolution, that is, the output images of the multiple branch lookup tables have the same resolution, which facilitates direct channel fusion without the need for sampling or interpolation.

[0069] In some embodiments, step S112 includes channel stitching of at least two images from a plurality of enhanced first channel images and a plurality of enhanced second channel images to obtain one or more channel stitched images; convolving the stitched channel components of the one or more channel stitched images to reduce the number of channels to obtain one or more enhanced images of the input image corresponding to one or more image enhancement tasks.

[0070] In this example, to address the insufficient computing power on the edge, the aforementioned low-computing-power fusion method is employed. The outputs of multiple branch lookup tables are concatenated along the channel dimension. For instance, the channel images output by multiple branch lookup tables LUT1, LUT2, and LUT3 are [H, W, C1], [H, W, C2], and [H, W, C3], respectively, where H is the image height, W is the image width, and C1, C2, and C3 are the channel components. After channel concatenation, the resulting image is [H, W, C1+C2+C3]. Then, a 1x1 convolution is performed on the [H, W, C1+C2+C3] image, resulting in a compressed channel count of 1 and an output of [H, W, C], thus achieving cross-channel linear mapping. Therefore, channel fusion is achieved through a low-computing-power algorithm to adapt to the low computing power limitations of edge devices.

[0071] In the example, one or more enhanced images output can correspond to multiple image enhancement tasks simultaneously and be output in parallel. For example, if one of the image enhancement tasks is denoising, then one of the enhanced images output can be the denoised input image; if one of the image enhancement tasks is dehazing, then one of the enhanced images output can be the dehazed input image; one of the image enhancement tasks can also include denoising, dehazing, and super-resolution simultaneously, then one of the enhanced images output can be the denoised, dehazed, and super-resolution input image, without any restrictions.

[0072] In some embodiments, method 100 further includes: performing color space conversion on one or more enhanced images for display or subsequent image processing tasks.

[0073] In the example, one or more enhanced images can be converted to any color space according to task requirements for display or subsequent image processing tasks, without any restrictions.

[0074] It should be noted that this disclosure does not limit the order of steps S102-S112, nor does it limit the order of obtaining the various consideration factors. For example, in step S106, the order in which multiple first channel images of the input image in the first color space and multiple second channel images of the input image in the second color space are extracted can be arbitrary. As another example, the training of the multi-branch lookup table in step S108 can be performed before step S102, and this is not limited here.

[0075] Figure 2 This is a detailed flowchart illustrating an exemplary process 200 of an image processing method 100 according to an exemplary embodiment.

[0076] refer to Figure 2An exemplary process 200 of the image processing method 100 includes the following steps:

[0077] The input image 201 is stored in a shared buffer 202 in the RGB color space;

[0078] A color gamut conversion 203 is performed on the input image 201 in the shared buffer 202 under the RGB color space, that is, a color space conversion, for example, converting the input image 201 from RGB to YCbCr;

[0079] Input image 201 in RGB color space into RGB domain LUT1 206 and RGB domain LUT2 207;

[0080] The input image 201 after color gamut conversion 203 is input into the Y channel branch LUT 204 and the CbCr channel branch LUT 205;

[0081] The output images of the Y channel branch LUT 204, CbCr channel branch LUT 205, RGB domain LUT1 206 and RGB domain LUT2 207 are subjected to multi-branch feature fusion 208;

[0082] Perform an inverse color gamut conversion 209 on the fused image, for example, from YCbCr to RGB; and

[0083] Perform multi-task output based on task requirements 210.

[0084] It is important to note that Figure 2 The exemplary process 200 is for illustrative purposes only and not for limiting purposes. For example, the color space of the image can be any color space such as RGB, YCbCr, LAB, HSV, or HSV.

[0085] Embodiments of this disclosure also provide an image processing apparatus.

[0086] Figure 3 This is a schematic block diagram illustrating an image processing apparatus 300 according to an exemplary embodiment.

[0087] like Figure 3As shown, in some embodiments, the apparatus 300 includes: a first module 310 configured to acquire an input image in a first color space; a second module 320 configured to convert the input image in the first color space into an input image in a second color space; a third module 330 configured to extract multiple first channel images of the input image in the first color space and multiple second channel images of the input image in the second color space; a fourth module 340 configured to train multiple branch lookup tables for one or more channels of the multiple channels in the first color space and the multiple channels in the second color space based on one or more image enhancement tasks; a fifth module 350 configured to input the multiple first channel images and the multiple second channel images into the multiple branch lookup tables respectively to obtain multiple enhanced first channel images and multiple enhanced second channel images; and a sixth module 360 ​​configured to perform channel fusion on at least two images of the multiple enhanced first channel images and the multiple enhanced second channel images to obtain one or more enhanced images of the input image corresponding to one or more image enhancement tasks.

[0088] The operations of the first module 310, the second module 320, the third module 330, the fourth module 340, the fifth module 350, and the sixth module 360 ​​described above can be combined. Figure 1 The operations of steps S102, S104, S106, S108, S110 and S112 are the same, so the details of each aspect will not be repeated here.

[0089] While specific functions have been discussed above with reference to specific modules, it should be noted that the functions of the modules discussed herein can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module. The specific actions performed by the modules discussed herein include the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, a specific module performing an action can include the specific module performing the action itself and / or another module that performs the action, called or otherwise accessed by the specific module.

[0090] It should also be understood that the above regarding Figure 3The described modules can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit. For example, these modules can be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (which includes one or more components in a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0091] According to one aspect of this disclosure, an electronic circuit is also provided, which includes circuitry configured to perform the steps of any of the method embodiments described above.

[0092] According to one aspect of this disclosure, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0093] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0094] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0095] In the following text, combined with Figure 4 Illustrative examples describing such electronic devices, non-transitory computer-readable storage media, and computer program products.

[0096] Figure 4 An example configuration of an electronic device 400 that can be used to implement the methods described herein is shown.

[0097] Electronic device 400 can be a variety of different types of devices. Examples of electronic device 400 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices, etc.

[0098] Electronic device 400 may include at least one processor 402, memory 404, multiple communication interfaces 406, display device 408, other input / output (I / O) devices 410, and one or more mass storage devices 412 capable of communicating with each other, such as via system bus 414 or other suitable connections.

[0099] Processor 402 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 402 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 402 may be configured to acquire and execute computer-readable instructions stored in memory 404, mass storage device 412, or other computer-readable media, such as program code of operating system 416, program code of application program 418, program code of other program 420, etc.

[0100] Memory 404 and mass storage device 412 are examples of computer-readable storage media for storing instructions that are executed by processor 402 to perform the various functions described above. For example, memory 404 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 412 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 404 and mass storage device 412 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 402 as a specific machine configured to perform the operations and functions described in the examples herein.

[0101] Multiple programs may be stored on mass storage device 412. These programs include operating system 416, one or more application programs 418, other programs 420, and program data 422, and they may be loaded into memory 404 for execution. Examples of such application programs or program modules may include computer program logic (e.g., computer program code or instructions) of the methods described herein and / or other embodiments described herein.

[0102] Although Figure 4 The modules 416, 418, 420, and 422, or portions thereof, are illustrated as being stored in memory 404 of electronic device 400; however, modules 416, 418, 420, and 422 may be implemented using any form of computer-readable medium accessible by electronic device 400. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0103] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by electronic devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0104] One or more communication interfaces 406 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces, near field communication (NFC) interfaces, etc. Communication interface 406 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 406 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0105] In some examples, a display device 408, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 410 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0106] The technologies described herein can be supported by these various configurations of electronic device 400, and are not limited to specific examples of the technologies described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on a server remote from electronic device 400. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect electronic device 400 to other electronic devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality may be implemented partly on electronic device 400 and partly through a platform that abstracts the functionality of the cloud.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the input image in the first color space; The input image in the first color space will be converted to the input image in the second color space; Extract multiple first channel images of the input image in the first color space and multiple second channel images of the input image in the second color space; Train multiple branch lookup tables for multiple channels in the first color space and multiple channels in the second color space based on one or more image enhancement tasks. The plurality of first-channel images and the plurality of second-channel images are respectively input into the plurality of branch lookup tables to obtain a plurality of enhanced first-channel images and a plurality of enhanced second-channel images; and At least two of the multiple enhanced first channel images and the multiple enhanced second channel images are fused to obtain one or more enhanced images of the input image corresponding to one or more image enhancement tasks.

2. The method according to claim 1, characterized in that, The receptive field size of the multiple branch lookup tables is the same, and the receptive field size is the size of the image region perceived by a single query of the multiple branch lookup tables.

3. The method according to claim 2, characterized in that, The input image in the first color space is segmented into multiple input sub-images in the first color space and stored in an input buffer. The input buffer is used for querying all branch lookup tables, wherein the resolution of the multiple input sub-images is the same as the receptive field size of the multiple branch lookup tables.

4. The method according to claim 1, characterized in that, The method further includes: The color space of the one or more enhanced images is converted for display or subsequent image processing tasks.

5. The method according to claim 1, characterized in that, The first color space is the RGB color space, and the second color space is the YCbCr color space.

6. The method according to claim 1, characterized in that, The one or more image enhancement tasks include at least one of super-resolution enhancement, noise reduction, rain removal, and haze removal of the input image.

7. The method according to claim 1, characterized in that, The method further includes: The pixel values ​​of the plurality of first channel images, the plurality of second channel images, and the plurality of branch lookup tables are quantized to reduce the size of the plurality of branch lookup tables.

8. The method according to claim 1, characterized in that, The multiple branch lookup tables can be obtained through individual training or joint training.

9. The method according to claim 1, characterized in that, The multiple enhanced first channel images and the multiple enhanced second channel images have the same resolution.

10. The method according to claim 9, characterized in that, Channel fusion is performed on at least two images from the plurality of enhanced first channel images and the plurality of enhanced second channel images to obtain one or more enhanced images of the input image corresponding to one or more image enhancement tasks, including: At least two images from the plurality of enhanced first channel images and the plurality of enhanced second channel images are stitched together to obtain one or more channel stitched images; Convolution is performed on the stitched channel components of the one or more channel stitched images to reduce the number of channels, resulting in one or more enhanced images of the input image corresponding to one or more image enhancement tasks.

11. An image processing apparatus, characterized in that, The device includes: The first module is configured to acquire the input image in a first color space; The second module is configured to convert the input image in the first color space into the input image in the second color space; The third module is configured to extract multiple first channel images of the input image in the first color space and multiple second channel images of the input image in the second color space; The fourth module is configured to train multiple branch lookup tables for one or more channels of the first color space and the second color space based on one or more image enhancement tasks. The fifth module is configured to input the plurality of first channel images and the plurality of second channel images into the plurality of branch lookup tables respectively, to obtain a plurality of enhanced first channel images and a plurality of enhanced second channel images; and The sixth module is configured to perform channel fusion on at least two of the plurality of enhanced first channel images and the plurality of enhanced second channel images to obtain one or more enhanced images of the input image corresponding to one or more image enhancement tasks.

12. An electronic circuit, characterized in that, The electronic circuit includes: A circuit configured to perform the steps of the method according to any one of claims 1 to 10.

13. An electronic device, characterized in that, The electronic device includes: Processor; and A memory storing a program, the program comprising instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 10.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 10.

Images