Image enhancement method and system

Abstract

The invention discloses a computer-implemented image enhancement method and an image enhancement system. The method comprises the following steps: receiving an input image which is a low-illumination image; processing the input image through a pre-trained image intensifier to generate an initial enhanced image; accessing, from the image memory, a response value corresponding to a sample-specific attribute of the normal image; generating an adjustment factor based on the response value from the image memory; a final enhanced image is generated by applying the adjustment factor to the initial enhanced image.

Description

Technical Field

[0001] The present disclosure relates to an image enhancement method and system, and in particular to a low-light image enhancement method and system. Background Art

[0002] Low-light image enhancement, i.e., enhancing images captured in low light, is a representative problem in image processing. So far, several methods can be applied to try to improve the resolution, i.e., improve the quality of low-light images. Common methods include conventional image enhancement methods, deep learning image enhancement methods, and visual transformation methods.

[0003] Conventional methods can generally be divided into two categories: 1) Histogram Equalization (HE) based methods; 2) Retinex model based methods. HE based methods adjust the histogram of low-light images to a specific mathematical distribution through global or local operations. HE based methods usually lead to unnecessary local illumination and may also have underexposure or overexposure problems. This is because these methods are usually not flexible enough for visual content. Retinex model based methods enhance images according to the Retinex theory. Retinex models can be computationally expensive and usually do not work well.

[0004] Some deep learning methods have been used for low-light image enhancement. LLNet proposed a stacked autoencoder network that can achieve enhancement and denoising simultaneously. Other methods have also used a similar structure to LLNet, with an end-to-end network for low-light image enhancement, such as a multi-scale framework residual learning model, a progressive recurrent network. Other deep learning methods have introduced the Retinex model into a deep learning neural network, which usually generates an enhanced image by adopting a dedicated sub-network for the illumination and reflection components of the image respectively. Since the correlation between the low-light image and the reference image is not a one-to-one relationship, in order to associate the input low-light image with the reference image, the deep learning network usually has to apply hyperparameters. However, most of the settings of the hyperparameters of the existing deep learning methods must be selected externally (i.e., the ratio must be pre-calculated and determined by the user before applying the learning model). Since considerable additional calculations are required, the practical application of deep learning methods is limited.

[0005] Visual Transformers (ViT) are often applied to Natural Language Processing (NLP), where multi-head self-attention excels in handling long-range dependencies and contexts of sequential data. However, for low-level vision tasks, the output from the transformer model always outputs images, rather than labels or boxes in high-level vision tasks. This greatly increases the computational cost and time. Transformer models with self-attention are useful for low-light image enhancement, but current methods are complex and expensive, making their application challenging. Summary of the invention

[0006] Image enhancement, especially low-light image enhancement, has become a current challenge. Several methods have been tried, but still have unsatisfactory results or are computationally intensive. Memory networks are a way to generate enhanced images with a reasonable computational load. The present disclosure relates to an image enhancement system and method, in particular, low-light image enhancement using a memory network. The system and method utilize an external memory network.

[0007] In one embodiment, the present disclosure relates to an image enhancement system, in particular, low-light image enhancement. The system may include an image processing network, which includes an image memory. The image memory is configured to store sample-specific attributes of a training data set, that is, the image memory is configured to cache sample-specific attributes of a training data set of images. The stored sample-specific attributes can be called during testing and / or training of the network to facilitate adaptive adjustment of test samples. Adaptive adjustment can be applied to the initial image enhancement output to improve the output and generate a higher quality (i.e., higher resolution) image. The system and method apply an external memory enhancement network to low-light image enhancement.

[0008] The image enhancement method and system of the present disclosure enhances a low-light image to align with a normal-light image. The enhanced image output by the method and system is an enhanced low-light image aligned with a normal-light image.

[0009] According to a first aspect of the present disclosure, a computer-implemented image enhancement method is provided. The method comprises the following steps:

[0010] Receiving an input image, wherein the input image is a low-light image;

[0011] Processing the input image through a pre-trained image enhancer to generate an initial enhanced image;

[0012] accessing, from the image memory, response values ​​corresponding to sample-specific attributes of the normal image;

[0013] generating an adjustment factor based on the response value from the image memory;

[0014] A final enhanced image is generated by applying the adjustment factor to the initial enhanced image.

[0015] In one embodiment, the adjustment factor is generated based on combining the response value with global average pooling data associated with the initial enhanced image.

[0016] In one embodiment, the adjustment factor is generated by applying an adaptive fusion function.

[0017] In one embodiment, the step of applying the adaptive fusion function comprises:

[0018] Calculate ratios of sample-specific attributes;

[0019] Calculate the global average pooling data through element-by-element division;

[0020] cascading the sample-specific attribute with the global average pooled data to generate a cascade value;

[0021] Determining one or more weight vectors by applying a softmax function to the concatenated values;

[0022] The adaptive adjustment factor is obtained based on the one or more weight vectors and a ratio of the sample-specific attribute to the global average pooling data.

[0023] In one embodiment, the step of obtaining the adaptive adjustment factor includes: multiplying a first weight vector by the ratio of the sample-specific attribute to the global average pooling data, and summing the result with a second weight vector; wherein the first weight vector corresponds to the sample-specific attribute, and the second weight vector corresponds to the global average pooling data.

[0024] In one embodiment, the image enhancement method comprises the following additional steps:

[0025] Processing the input image by a feature generator to generate query features;

[0026] The response value is accessed from the image memory based on the query feature.

[0027] In one embodiment, the query features are generated by feeding the input image to a pre-trained feature generator.

[0028] In one embodiment, the pre-trained feature generator is a ResNet-18 network.

[0029] In one embodiment, the image enhancement method comprises the following steps:

[0030] identifying within the image memory a memory key corresponding to the query feature;

[0031] identifying a response value corresponding to the identified memory key;

[0032] A response value is accessed from a memory address of the image memory corresponding to the memory key.

[0033] In one embodiment, the memory key is identified from the plurality of memory keys by calculating a cosine similarity between the query and the plurality of memory keys and identifying a memory key having a closest cosine similarity to the query.

[0034] In one embodiment, the image enhancer is a pre-trained transformer based image enhancer comprising a symmetric encoder-decoder comprising a four-level pyramid feature map with skip connections.

[0035] In one embodiment, the step of processing the input image by the image enhancer comprises the following steps:

[0036] Feeding the input image into a first convolutional layer;

[0037] extracting one or more low-level features by processing the input image through the first convolutional layer, wherein each of the low-level features is defined by a dimension including a spatial dimension and a plurality of channels;

[0038] said one or more said low-level features passing through a four-stage symmetric encoder-decoder;

[0039] An initial enhanced image is generated from the four-stage symmetric encoder-decoder.

[0040] In one embodiment, the image enhancement method includes a memory writing process, which includes the following steps:

[0041] receiving the response value corresponding to the query feature;

[0042] By processing the reference image, a desired memory value is obtained;

[0043] updating one or more memory keys in a dictionary using the response value;

[0044] If the difference between the response value and the expected memory value is within a threshold, the one or more memory keys are updated.

[0045] In an embodiment, the method is applied to a low-light image to enhance the low-light image, and the image memory includes a plug-and-play mechanism for integration into the image enhancement method.

[0046] According to a second aspect of the present disclosure, an image enhancement system is provided. The system comprises:

[0047] Image enhancement processor;

[0048] An image memory for communicating with the image enhancement processor, the image memory defining a memory dictionary, the memory dictionary comprising one or more memory keys and one or more response values, each of the memory keys corresponding to a response value, wherein the response value corresponds to a value from a normal image;

[0049] Wherein, the image enhancement processor is configured as follows:

[0050] receiving an input image, wherein the input image is a low-light image,

[0051] Processing the input image through a pre-trained image enhancer to generate an initial enhanced image,

[0052] Access the response values ​​corresponding to the sample-specific attributes of the normal image from the image memory,

[0053] generating an adjustment factor based on the response value from the image memory,

[0054] By applying the adjustment factor to the initial enhanced image, a final enhanced image is output.

[0055] In one embodiment, the image enhancement processor comprises a learning network, wherein the learning network comprises:

[0056] a pre-trained image enhancer configured to receive one or more input images and generate the initial enhanced image;

[0057] a pre-trained feature generator configured to receive the one or more input images and generate query features based on feeding the one or more input images to the feature generator;

[0058] a memory reading module configured to receive the query feature and access the response value from the image memory based on the received query feature;

[0059] an adaptive fusion module, configured to receive the response value and generate the adjustment factor based on the response value;

[0060] The output module is configured to apply the adjustment factor to the initial enhanced image and output the final enhanced image.

[0061] In one embodiment, the image enhancement processor is configured as follows:

[0062] Generate global average pooling data from the initial enhanced image;

[0063] The adaptive fusion module is configured as follows:

[0064] Receive global average pooling data and response values ​​from the memory reading module,

[0065] An adjustment factor is generated based on combining the response value and global average pooling data associated with the initial enhanced image; wherein the adaptive fusion module is configured to apply an adaptive fusion function to generate the adjustment factor.

[0066] In one embodiment, the adaptive fusion module is further configured as follows:

[0067] Calculate the ratio of the sample-specific attribute to the global average pooled data by element-wise division;

[0068] cascading the sample-specific attribute with the global average pooled data to generate a cascade value;

[0069] Determining one or more weight vectors by applying a softmax function to the concatenated values;

[0070] The adaptive adjustment factor is obtained based on the one or more weight vectors and a ratio of the sample-specific attribute to the global average pooling data.

[0071] In one embodiment, the adaptive fusion module, as a part of generating the adaptive adjustment factor, is configured as follows:

[0072] multiplying the first weight vector by the ratio of the sample-specific attribute to the global average pooled data and summing with the second weight vector;

[0073] The first weight vector corresponds to the sample-specific attribute, and the second weight vector corresponds to the global average pooling data.

[0074] In one embodiment, the image enhancer is a pre-trained transformer based image enhancer comprising a symmetric encoder-decoder comprising a four-level pyramid feature map with skip connections.

[0075] In one embodiment, the memory reading module is further configured as follows:

[0076] identifying within the image memory a memory key corresponding to the query feature;

[0077] identifying a response value corresponding to the identified memory key;

[0078] A response value is accessed from a memory address of the image memory corresponding to the memory key.

[0079] In one embodiment, the memory reading module is configured as follows:

[0080] calculating a cosine similarity between the query and the plurality of memory keys;

[0081] The memory key is identified from a plurality of memory keys based on an output of the cosine similarity calculation, wherein the memory key is identified as a memory key having a closest cosine similarity to the query.

[0082] In one embodiment, the image intensifier is further configured as:

[0083] Feeding the received input image into the first convolutional layer;

[0084] Processing the input image by applying the first convolutional layer;

[0085] Based on the processing in the first convolutional layer, extracting one or more low-level features, wherein each of the low-level features is defined by a dimension including a spatial dimension and a plurality of channels;

[0086] passing said one or more said low-level features through a four-stage symmetric encoder-decoder;

[0087] By applying the four-stage symmetric encoder-decoder, an initial enhanced image is generated.

[0088] In one embodiment, the system includes a display in electrical communication with the image enhancement processor and configured to receive the output of the enhanced image and display the enhanced image.

[0089] According to a third aspect of the present disclosure, a low-light image enhancement system is provided. The system comprises:

[0090] a computing device including an image enhancement processor;

[0091] An image memory, comprising sample-specific attributes of an image training set; wherein the sample-specific attributes relate to enhancement information of low-light images;

[0092] The image enhancement processor is configured to implement a memory enhancement network, and the memory enhancement network is configured to perform the memory enhancement method as described above. The method may include any one or more steps of the method described in the first aspect above.

[0093] In one embodiment, the image memory includes a memory unit configured to be plug-and-play into the computing device such that the processor can access data from the external memory.

[0094] According to a fourth aspect of the present disclosure, a machine learning network for image enhancement is provided, in particular a machine learning network for executing the method as described above and any steps thereof. The machine learning network comprises:

[0095] A pre-trained image enhancer comprising a symmetric encoder-decoder comprising a four-level pyramid feature map with skip connections, each level of the encoder-decoder comprising a plurality of transformer blocks comprising a layer normalization layer, a multi-head channel self-attention module, and a feed-forward network;

[0096] Res-net18 feature generator;

[0097] a pre-trained image memory defining a memory dictionary, the memory dictionary comprising one or more memory keys and one or more response values, each of the memory keys corresponding to a response value, wherein the response value corresponds to a value from a normal image;

[0098] an adaptive fusion module in communication with the stored image and the pre-trained image enhancer;

[0099] wherein the one or more input images are processed in parallel, wherein a first parallel path of the network comprises the pre-trained image enhancer, a second parallel path comprises the Res-net 18 features and the image memory, and outputs of the first and second paths are fed to the adaptive fusion module.

[0100] As described above, the machine learning network can be configured to implement or perform any of the method steps described with reference to the first aspect.

[0101] According to another aspect of the present disclosure, an image enhancement processor for low-light image enhancement to generate an enhanced image is provided, wherein the image enhancement processor is configured to communicate with an image memory, wherein the image enhancement processor is configured to perform the steps of the image enhancement method described above. In one example, the image enhancement processor can be configured to perform the method steps according to the first aspect described above.

[0102] According to another aspect of the present disclosure, a computer-readable medium including instructions for low-light image enhancement is provided, wherein when the instructions are executed by a computing device, the computing device performs the image enhancement method described above. The image enhancement method may be the method described in the first aspect above.

[0103] According to another aspect of the present disclosure, a method for training a machine learning network for image enhancement is provided. The method comprises:

[0104] receiving the response value corresponding to the query feature;

[0105] By processing the reference image, a desired memory value is obtained;

[0106] updating one or more memory keys in a dictionary using the response value;

[0107] If the difference between the response value and the expected memory value is within a threshold, the one or more memory keys are updated.

[0108] In one embodiment, the training method is applied to a learning network as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Embodiments of the present disclosure will be described below by way of example with reference to the accompanying drawings, in which:

[0110] Figure 1 A block diagram of an embodiment of an image enhancement system is shown;

[0111] Figure 2 shows a block diagram of a learning network used in an image enhancement processor;

[0112] Figure 3 A block diagram showing an image memory and its contents, and a memory read process to extract appropriate response values ​​from the memory;

[0113] Figure 4 An example of a pre-trained image enhancer as part of an image enhancement system is shown;

[0114] Figure 5 Shows details of the transformer layers used in a pre-trained image enhancer;

[0115] Figure 6 shows an example of a Multi-Head Channel Self-Attention module (MCSA) as part of a Transformer layer;

[0116] Figure 7 The function of the adaptive fusion module for performing the adaptive fusion method is shown in FIG.

[0117] Figure 8 is a block diagram illustrating an example method of a memory write process. DETAILED DESCRIPTION

[0118] The present disclosure relates to an image enhancement method and system. In particular, the present disclosure relates to a method and system for image processing to achieve image enhancement. Embodiments of the present disclosure may also relate to a method and system for low-light image enhancement (i.e., enhancing an image with low or weak light) to generate an enhanced image of higher quality (i.e., higher resolution).

[0119] In an example embodiment, an image enhancement system may include an image enhancement processor and an image memory. The image memory may include a separate (i.e., external) memory unit configured to store sample-specific attributes. The sample-specific attributes relate to expected values ​​from a normal light image. The expected values ​​relate to enhanced values ​​of the normal light image. The image memory may define a memory dictionary. The memory dictionary includes one or more memory keys and one or more response values, each memory key corresponding to a response value, wherein the response value corresponds to a value from a normal image. The image memory may be an external memory. The image memory may be implemented as a plug-and-play mechanism.

[0120] The computing device can be configured to implement a learning network, i.e., an AI model or a machine learning network for image enhancement. The image processor can be configured to apply or execute the network. The machine learning network is programmed to enhance low-light images. The machine learning network can be configured to receive low-light images and perform image enhancement to output enhanced images. The low-light images can be low-resolution images. These low-light images can correspond to images captured under low-light conditions. The learning network is configured to apply stored values ​​from an image memory as part of the processing to enhance the low-light images. The learning network can be configured to apply sample-specific attributes stored in the image memory to facilitate adaptive adjustment of the test sample. The enhanced image can be an image with a higher resolution than the low-light image. The enhanced image can be a normal-light image, i.e., an image that can correspond to an image captured in normal light.

[0121] The learned network can be configured to apply or utilize the values ​​stored in the image memory during testing to re-augment the already enhanced images to further improve the quality.

[0122] refer to Figure 1 , shows a block diagram of an embodiment of an image enhancement system 100. The system 100 is particularly suitable for low-light image enhancement to output an enhanced, higher quality (ie, higher resolution) image from a received low-light image.

[0123] The system 100 includes an image enhancement processor 102, an input interface 104, an output interface 106, and an image memory 120. The system 100 may be configured to implement an image enhancement method, particularly a low-light image enhancement method.

[0124] The system 100 and method are configured to: receive an input low-light image, process the input image through a pre-trained image enhancer to generate an initial enhanced image; access response values ​​corresponding to sample-specific attributes of a normal image from an image memory; generate adjustment factors based on the response values ​​from the image memory; and generate a final enhanced image by applying the adjustment factors to the initial enhanced image. The input low-light image may be received through an input interface. The enhanced image may be output through an output interface.

[0125] In an example embodiment, the image enhancement system and method may be implemented on a computing device in software, hardware, firmware, or a combination of both.

[0126] The computing device (i.e., computing equipment or computing system) can be implemented by any computing architecture, including: a portable computer, a tablet computer, a stand-alone personal computer (PC), a smart device, an Internet of Things (IoT) device, an edge computing device, a client / server architecture, a "dumb" terminal / host architecture, a cloud computing-based architecture, a camera, a smartphone, a wearable device with a camera, a road monitoring device, a drone, or any other appropriate architecture. The computing device can be appropriately programmed to implement the low-light image enhancement system 100 and method.

[0127] In one example, the computing device includes appropriate components necessary to receive, store and execute appropriate computer instructions. These components may include: a processing unit, such as a central processing unit (CPU), a math co-processing unit (Math Co-processing Unit, Math Processor), a graphics processing unit (Graphics Processing Unit, GPU) or a tensor processing unit (TPU) for tensor or multi-dimensional array calculation or manipulation operations; read-only memory (ROM); random access memory (RAM); input / output (I / O) devices (e.g., disk drives); input devices (e.g., Ethernet ports, USB ports, etc.); displays, such as liquid crystal displays, light-emitting displays or any other suitable displays; and communication links. The computing device may include instructions that can be stored in ROM, RAM or disk drives and can be executed by the processing unit. A plurality of communication links may be provided that may be variously connected to one or more other computing devices, such as servers, personal computers, terminals, wireless or handheld computing devices, IoT devices, smart devices, and edge computing devices. At least one of the plurality of communication links may be connected to an external computing network via a telephone line or other type of communication link.

[0128] The computing device or computer may include storage devices such as disk drives, which may include solid-state drives, hard disk drives, optical drives, tape drives, or remote or cloud-based storage devices. The computing device may use a single disk drive or multiple disk drives, or a remote storage service. The computing device may also have a suitable operating system resident in its disk drive or ROM.

[0129] A computing device (i.e., a computer or computing equipment) may also provide the necessary computing power to operate or interface with a machine learning network, such as a neural network, to provide various functions and outputs. The neural network may be implemented locally, or may also be accessed or partially accessed through a server or cloud-based service. The machine learning network may also be untrained, partially trained, or fully trained, and / or may also be retrained, modified, or updated over time. In an example, the computing device described herein may be a smart phone or a digital camera, which includes the components described herein, and may implement the image enhancement system 100 and the image enhancement system method 300.

[0130] In this embodiment, the image enhancement system 100 can be implemented to include: an image enhancement processor; and an image memory, which is used to communicate with the image enhancement processor. The image memory defines a memory dictionary. The memory dictionary includes one or more memory keys and one or more response values. Each memory key corresponds to a response value, wherein the response value corresponds to a value from a normal image. The image enhancement processor is configured to:

[0131] receiving an input image, wherein the input image is a low-light image,

[0132] The input image is processed by a pre-trained image enhancer to generate an initial enhanced image.

[0133] accessing, from the image memory, response values ​​corresponding to sample-specific attributes of the normal image;

[0134] generating an adjustment factor based on the response value from the image memory;

[0135] By applying the adjustment factor to the initial enhanced image, a final enhanced image is output.

[0136] like Figure 1 As shown, the image enhancement system, particularly the low light image enhancement system, includes an image enhancement processor 102 for receiving one or more input images 104 from an input interface 106. The input interface 106 is in electrical communication with the image enhancement processor 102. The image enhancement processor 102 is a processing unit having the structure as described above.

[0137] The input interface 106 is configured to be electrically connected to an image capture device 108, such as a camera or a smart phone. The input interface 106 is used to communicate with the image capture device 108. Alternatively, the input interface 106 may receive one or more images from an image source such as a server, a file system, a cloud system, an external disk drive such as a portable hard drive, or a data bearing medium such as a CD or DVD. Optionally, the system 100 may include the image capture device 108. The input interface 106 is configured to receive the image 104 and transmit it to the image enhancement processor 102 for enhancement. The input image may be a still image or a video stream or a moving image.

[0138] In a typical operation scenario, the input image 104 may be a low-light image, i.e., an image captured in low light. The image may be anything, such as a night photo, a backlit photo, a road surveillance photo in low light, a land survey image in low light, etc.

[0139] The system 100 also includes an output interface 110 (i.e., output module). The output interface 110 is configured to receive and output the enhanced image from the image enhancement processor 102. The output interface 110 can send the enhanced image to another remote device, for example, via one or more communication links. Optionally, the output interface 110 can transmit the enhanced image to a display 112 (i.e., user interface) for presentation to a user.

[0140] The image enhancement processor 102 is configured to communicate with the image memory 120 to receive enhancement data, ie, response values ​​corresponding to sample-specific properties of a normal image.

[0141] The image memory 120 may be a computer readable medium or a machine readable medium configured to store data. The image memory 120 is configured to communicate with the image enhancement processor 102 and any one or more learning networks to provide the stored data.

[0142] Image memory 120 may be a plug-and-play mechanism that may be used with image enhancement processor 102 for low light image enhancement. Image memory 120 may be an external unit and part of image enhancement system 100. Image memory 120 may be stored in a plug-and-play mechanism (i.e., a plug-and-play device), such as a USB or portable hard drive or other disk drive. Image memory 120 may be an input device configured to be removably connected to a computing device implementing image enhancement system 100.

[0143] Alternatively, the image storage 120 may be stored in a remote source and may be electronically downloaded, for example from an electronic storage such as a server or cloud system or remote storage. The image storage may be accessed directly from the remote source using a suitable protocol, for example, File Transfer Protocol (FTP).

[0144] The image enhancement processor 102 is configured to process the received input image and generate an initial enhanced image. The initial enhanced image is an enhanced low-light image. The image enhancement processor 102 is configured to access the response value and generate an adjustment factor based on the response value extracted from the image memory 120. The adjustment factor is used to adjust the initial enhanced image, that is, to re-enhance the initial enhanced image. The image enhancement processor 102 is configured to output a final enhanced image. The final enhanced image is the initial enhanced image further enhanced by applying the adjustment factor.

[0145] The image enhancement system 100 is configured to apply an image enhancement learning network. The image enhancement processor 102 can be implemented by a suitable learning network or machine learning device, or can include a suitable learning network or machine learning device, such as an external memory enhancement network trained using a suitable training data set. Optionally, the learning network can include multiple sub-learning networks or sub-machine learning devices, which can operate together to define a learning network. The processor 102 can implement a sub-learning network or a sub-machine learning device. The training process can include providing training data. The training data is data in the form of image pairs. The image pair can include a low-light image and a corresponding reference image. The reference image can be a normal light image. The learning network can be trained to enhance the image based on the image pair data.

[0146] The image memory 120 can capture sample-specific properties of the training data set to further guide the enhancement. The system 100 can be configured to perform a memory write process during a training phase to define a memory dictionary. The memory dictionary can include one or more memory keys and one or more response values. Each memory key corresponds to a response value. The response value corresponds to a value from a normal image, i.e., a sample-specific property of a normal image. Alternatively, the image memory 120 can be pre-trained using a training data set through a separate training process, independent of training the learning network.

[0147] The system 100 generates an improved enhanced image. The improvement of the enhanced image is derived from the learned response values ​​(i.e., normal image values), which serve as adjustment factors for image re-enhancement. The external image memory 120 allows a more complex distribution of reference images in the entire data set to be remembered to facilitate more adaptive adjustment of test samples or input images. The image memory can be implemented as a plug-and-play mechanism so that it can be integrated into any existing image enhancement method or system to further improve the enhancement quality.

[0148] refer to Figure 2 , shows a block diagram of a learning network 200 used in the image enhancement processor 102. The learning network 200 may include an external memory enhancement network, such as Figure 2As shown. The processor 102 can be implemented with an external memory enhancement network 200 to perform various functions of the processor. The learning network can implement the image enhancement method described herein.

[0149] The learning network 200 includes a pre-trained image enhancer 202, a pre-trained feature generator 204, a memory reading module 206, an adaptive fusion module 208, and an output module 210. Optionally, the learning network 200 may further include a memory writing module 212. The memory reading module 206 and the memory writing module 212 are used to communicate with the image memory.

[0150] The image enhancement processor 102 may be configured to implement a pre-trained image enhancer 202. The pre-trained image enhancer 202 may be configured to receive one or more input images (low-light images) and generate an initial enhanced image. The pre-trained image enhancer may be a pre-trained low-light image enhancement model. In an example, the pre-trained image enhancer 202 may be an image enhancer that is a pre-trained transformer including a symmetric encoder-decoder of a four-level pyramid feature map with skip connections.

[0151] Reference Figure 2 , the pre-trained image enhancer 202 processes the input image I to generate an enhanced image The processor 102 is configured to calculate the global average pooling data o. Calculate the global average pooling data and obtain the global average pooling data for further adaptive fusion.

[0152] The pre-trained feature generator 204 is configured to receive the one or more input images (low-light images) I and generate query features based on feeding the one or more input images to the feature generator 204. The feature generator can be a pre-trained ResNet-18 model. The ResNet-18 model obtains the low-light image I and outputs the query feature q.

[0153] The memory reading module 206 is configured to receive a query feature q and access the most relevant response value v from the memory dictionary based on the received query feature q. r For reference Figure 3 As part of the memory read process, the response value v is read from the image memory 120. r During the training phase, the response value v r is fed to the memory write module 212 to update the memory dictionary.

[0154] The adaptive fusion module 208 is configured to receive the response value v r , and based on the response value v rThe adaptive fusion module 208 is configured to fuse the response value v r And global average pooling data o The output module 210 is configured to apply the adjustment factor to the initial enhanced image (or image), and output the final enhanced image I a The output module 210 can be configured based on the expression Adaptively tune the initial enhanced image To align the initial enhanced image with the normal light image. The output module 210 may be a summing block.

[0155] Figure 3 A block diagram of the image memory 206 and its contents is shown. Figure 3 The memory read process 300 is further described to extract the appropriate response value v r . Reference Figure 3 , a memory dictionary 124 is defined and stored in the image memory 120. The memory dictionary 124 includes one or more memory keys. The memory key is defined as and define the memory value (i.e., the response value) as Additionally, the memory age is defined as where c k Indicates k i The dimension, c v Indicates v i , the dimension is 1≤i≤s, and s Represents the memory size. The entire memory dictionary 124 contains three terms and can be expressed as: M = (K, V, A) (1) where K = {k1, k2, ..., ks}, V = {v1, v2, ..., vs}, and A = {a1, a2, ..., as}. Optionally, the memory dictionary 124 can be defined as a matrix.

[0156] like Figure 3 As shown, the memory dictionary 124 includes a memory key 126 (K), a response value 128 (V), and an age value 130 (A). The key value, i.e., the key 126, can define a specific memory address. The response value 128 corresponds to a value from a normal image, i.e., a sample-specific attribute of a normal image. The age value 130 defines the age (age) of the specific data stored in association with the specific key.

[0157] The memory K stores information about the high-level features of the input image (i.e., input data) and also serves as addresses corresponding to the V and A components. The memory dictionary M stores the expected values ​​from the normal light image m, which are subsequently used to generate the adaptive adjustment factors (i.e., adjustment factors). The memory K and V components are extracted from the training data. The data points A in the memory dictionary are used for memory updates to track the age of the data.

[0158] The memory read module 206 is also configured to identify a specific memory key k corresponding to the input query q i ,like Figure 3 As shown in step 302 in the above example, the memory read module 206 is configured to calculate the query q and the multiple memory keys k. i Based on the output of the cosine similarity calculation, an appropriate memory key is identified from the plurality of memory keys. The memory key is identified as the memory key having the closest cosine similarity to the query.

[0159] The query feature q is obtained by feeding the input low-light image I into the pre-trained feature generator 204. In one example, the pre-trained feature generator 204 is a ResNet-18 model. The dimension of the query q is the same as K. Given a query q and K, identify the query q and the i-th memory key k i to find the most relevant memory key k r The following equation can be used to determine cosine similarity:

[0160]

[0161] The memory read module 206 is configured to identify the memory key k corresponding to the identified memory key k at step 304. i The appropriate response value (i.e., memory value) v r More specifically, the memory reading module 206 is configured to retrieve the memory dictionary M corresponding to the identified memory key k i The most relevant memory elements.

[0162] At step 306, the memory read module 206 is configured to access the response value v from the memory address of the image memory (ie, the memory dictionary M) corresponding to the memory key. r .like Figure 3 As shown, the appropriate response value is v2. r =v2, such as Figure 3 As shown. The response value is accessed from the memory dictionary M stored in the image memory.

[0163] The image enhancer 202 may be a pre-trained model. A pre-trained model may be an optimized model that does not require any further training. Figure 4 An example of a pre-trained image enhancer 202 is shown. Figure 4 As shown, the image intensifier 202 is a transformer image intensifier. Figure 4 In the form shown, the image enhancer adopts a symmetric encoder-decoder structure with a four-level pyramid feature map with skip connections.

[0164] like Figure 4 As shown, the image enhancer includes an initial convolution block 402 used as an input block. The image enhancer also includes a first down-sampling transformer block 404, a second down-sampling transformer block 406, a central transformer block 408, a first up-sampling transformer block 410, a second up-sampling transformer block 412, and an output block 414.

[0165] Convolution block 402 may include convolution layer 420. Convolution block 402 may also include downsampling layer 422. Each downsampling transformer block includes transformer layer 424 and downsampling layer 422. The central transformer block includes only transformer layer 424. The upsampling transformer block includes transformer layer 424 and upsampling layer 426. Output block 412 may include upsampling layer 426, transformer layer 424, and convolution layer 420. Figure 4 As shown, these blocks define a symmetric decoder-encoder architecture with a four-level pyramid feature map, with skip connections between the blocks (i.e., levels).

[0166] In one example, the pre-trained image enhancer 202 can be configured to feed the received input image into a first convolutional layer. The image enhancer 202 is configured to process the input image by applying the first convolutional layer and extract one or more low-level features based on the processing in the first convolutional layer. Each low-level feature can be defined by a dimension including a spatial dimension and a plurality of channels. The image enhancer 202 is further configured to pass the one or more low-level features through a four-level symmetric encoder-decoder. The image enhancer 202 is configured to generate an initial enhanced image by applying the four-level symmetric encoder-decoder.

[0167] Figure 4 It also shows the processing of the input image when it is processed by a pre-trained image enhancer. Given the definition of I∈R H×W×3 The convolution block 402 is configured to receive and process the input image 104. First, a 3×3 convolution layer is applied to the convolution block 402 to process the input image 104 to extract a size R H×W×C The low-level features of . H×W represents the spatial dimension, and C represents the number of channels.

[0168] The extracted features are passed through a four-stage symmetric encoder-decoder to obtain the enhanced image The four-stage symmetric encoder-decoder structure includes blocks 402 to 414 as described above. Between each two-stage encoder / decoder structure, there may be a 2× downsampling / upsampling rate. Feature downsampling / upsampling processing is achieved by pixel deshuffling and pixel shuffling operations. Of course, other alternative processing methods may also be used.

[0169] Image enhancers can present a hierarchical structure to handle multi-scale features. The encoder gradually reduces the resolution while expanding the channel capacity, starting from a high-resolution input. The decoder takes the low-resolution features at the bottleneck as input, gradually generates high-resolution features, and passes them through convolutional layers to generate a residual image.

[0170] To help the recovery process, skip connections can be utilized by concatenating the encoder and decoder features at each level and adding the residual image to the input image to obtain an enhanced output In each stage of the encoder-decoder (i.e., each block 402-414), multiple transformer blocks may be included, including a layer normalization layer, a multi-head channel self-attention module, and a feed-forward network (FFN).

[0171] Figure 5 Details of the transformer block 424 (i.e., the transformer layer) are shown. Figure 5 As shown, each layer includes a self-attention module 426, a multi-layer normalization 428, and a feed-forward network 430. Figure 5 In one example form shown, the encoder-decoder or transformer block has two layers of normalization 428. Any suitable transformer block structure may be used. Figure 6 An example of a multi-head channel self-attention module 426 (MCSA) as part of the transformer layer 424 is shown. Figure 6 An example function of the self-attention module 426 and how to parse the input into the output is shown. Any suitable self-attention module structure can be used as part of the transformer block (ie, encoder-decoder).

[0172] In one example, the input features may include a size R H×W×C The input features are projected through convolutions to generate the query, key, and value of the transformer. Instead of following the O(H 2 W 2 ) complexity, the transformer layer 424 is configured to perform self-attention on the spatial dimension with O(C 2 ) The self-attention mechanism is applied on the channel dimension of . The multi-head channel self-attention process is defined as:

[0173] in represents the self-attention operation, Q t , K t 、Vt ∈R H×W×C denotes the channel-wise reconstruction query, key, and value of the transformer. σ() denotes the softmax function and ε denotes a learnable scaling parameter.

[0174] After traditional multi-head self-attention, the number of channels can be divided into "heads", where attention is performed in parallel. The results are cascaded for multi-head self-attention.

[0175] The objective function is only used for the image enhancer 202. The image enhancer 202 is first trained using the objective function. After training, the pre-trained image enhancer 202 can be embedded in a memory mechanism. For example, the memory mechanism can be an image memory. The weight of the memory mechanism can be fixed during memory reads and memory writes. The loss function of the image enhancer 202 is described below.

[0176] The structural similarity loss evaluates the closeness of the reconstructed structural similarity and can be expressed as follows:

[0177]

[0178] Where SSIM() is the structural similarity [2], I^ represents the enhanced image, and I gt represents the reference ground truth image.

[0179] The distance between deep features can be used as a constraint for better visual quality, which can be formulated as:

[0180]

[0181] in Represents the process of extracting deep features from a pre-trained network (e.g., a known low-light image enhancement network).

[0182] The entire loss function is performed by jointly considering the above losses and using the following equation:

[0183]

[0184] refer to Figure 7 , showing the functionality of the adaptive fusion module 208. The adaptive fusion module 208 is configured to calculate the ratio of the sample-specific attribute to the global average pooled data by element-by-element division. The adaptive fusion module 208 is further configured to concatenate the sample-specific attribute with the global average pooled data to generate a concatenated value. The adaptive fusion module 208 is further configured to: determine one or more weight vectors by applying a softmax function to the concatenated value; and obtain an adaptive adjustment factor based on the one or more weight vectors and the ratio of the sample-specific attribute to the global average pooled data.

[0185] To generate the adaptive adjustment factor, the adaptive fusion module 208 is configured to multiply a first weight vector by a ratio of a sample-specific attribute to global average pooling data and sum it with a second weight vector. The first weight vector corresponds to the sample-specific attribute and the second weight vector corresponds to the global average pooling data.

[0186] Enhanced images The exposure level may not be satisfactory, i.e., the resolution may not be high enough due to the output of the image enhancer 202. The image enhancement processor 102 is configured to further adjust the enhancement result by using the response value accessed from the image memory. The learning network 200 is configured to provide adaptive adjustment of the enhancement result without additional training. The image memory 120 stores sample-specific attributes of the entire training data that can be called up during testing or operation. This avoids or reduces the chance of "forgetting" sample-specific mappings.

[0187] Obtain enhanced image from image enhancer 202 After that, the information o can be expressed by the following equation generate:

[0188]

[0189] Figure 7 The function of the adaptive fusion module for executing the adaptive fusion method 700 is shown. Figure 7 , the adaptive fusion module 208 is configured to calculate the relationship between the memory value vr and the pooling information o. In step 702, the response value vr and the pooling information o are initially used to generate a ratio by element-by-element division. In step 704, the response value vr and the pooling information o can be cascaded by applying a cascade. In step 706, a softmax function is used to generate a weight from the cascade of vr and o. The adaptive fusion module 208 is configured to calculate the weight in step 708 by using the following equation:

[0190] where ω1 and ω2 represent the weight vectors from vr and o, respectively.

[0191] In step 710, an adaptive adjustment factor α is derived by an addition function in the adaptive adjustment module 208. The adaptive adjustment factor α is derived as follows:

[0192]

[0193] The adaptive adjustment factor α can be used as the default value without manual intervention. Adjust the indicator of the enhanced image, where Represents the final adjusted image.

[0194] The memory write process is used to update the image memory 120. The image memory 120 is updated by the memory write module 212. The memory write process begins with the step of receiving a response value corresponding to the query feature. The next step includes: obtaining an expected memory value by processing a reference image; the method also includes: using the response value, updating one or more memory keys in the dictionary; if the difference between the response value and the expected memory value is within a threshold, updating the one or more memory keys.

[0195] Figure 8 An example method of a memory write process 800 is shown. The memory write process is used to update the image memory 120 with new relationships between low-light images and normal-light images. The memory write process 800 can be used to create a memory dictionary (M) 124 from a training data set including low-light images and corresponding normal-light images. The memory write process 800 can also be performed to update the image memory 120 with new data.

[0196] During the training phase, when the response value vr and the expected memory m are obtained, the image memory 120 is updated. The memory m defines data about the normal light image. Step 802 includes: receiving an image pair. The image pair includes a low light image I and a normal light image (ie, a high resolution image) I gt In step 804, the desired memory is m, where m is the memory from the reference image I gt (i.e., the ground truth image) is calculated, in which the average value of each channel is taken as follows:

[0197]

[0198] Where m represents the global average value of region R.

[0199] I gt (p,r,i) represents the pixel value at (p,q) associated with the ith channel in the image region R = H × W, where H and W are the reference ground truth image I, respectively. gt The desired storage value is obtained and defined as m∈R 1×1×cm , where for RGB images, c m =3.

[0200] In step 806, the low-light image may be processed by the feature generator 204. In step 808, the memory read module 206 uses the feature generator output (ie, the query feature q) to identify the response value v r .

[0201] According to the response value v rWhether the distance d between the expected memory value m and the image memory 120 is within the threshold γ, the image memory 120 can be updated in two different cases. In step 810, the distance d is calculated using the following equation:

[0202]

[0203] Among them, c v Yes r and the dimension of m.

[0204] In step 812, the first case is shown. In case 1, if d(m,v r )≤γ, it means that the value of the memory is correct and is only obtained by taking the current key k r The updated key is shown as feature 820 in the memory dictionary 124 and is labeled k'. After the rth item is updated, a r The age is reset to zero:

[0205]

[0206] In step 814, the second case is shown. In case 2, if d(m,v r )>γ, then it means that the current value of the memory does not match the expected memory. In this case, a new location in the dictionary 124 can be randomly selected to write the image pair (q, m). The new location is selected by finding the memory item with the oldest age. Assume that the oldest memory item is (k old ,v old ,a old ), then the update is performed as:

[0207] k old ←q,v old ←m,a old ←0.

[0208] The updated memory dictionary is shown as a new memory item 822m, and the age 824 is updated to 0, as shown by age a3.

[0209] In addition to the updated memory items, the age of the non-updated memory items will be increased by 1 in each round of updates. After training, the updated memory dictionary 124 (M) is configured to store pair information of low-light images in the memory key K and their corresponding normal-light images in the memory value V. The memory dictionary 124 (M) is updated with the new response value, which can be used to re-enhance the low-light image output from the image enhancer 202 in the learning network 200.

[0210] The image enhancement system 100 and the image enhancement method 300, in particular the system and method for low-light image enhancement, can be used in scenes requiring low-light enhancement, such as night photography, backlight photography, road monitoring or terrain monitoring, such as capturing images by a target or a vehicle. The system 100 can be implemented in a smart phone or camera or other image capture device mounted on a target or a vehicle.

[0211] The advantage of the system 100 including an external image memory 120 for storing sample-specific attributes throughout the training is that the stored data can be called during testing and during image enhancement to re-enhance low-light images. The image memory 120 can provide adaptive adjustments to test samples and / or other low-light images. The image memory 120 as a plug-and-play mechanism is further advantageous because it can be integrated with existing image enhancement equipment or systems to further improve the quality of enhancement. The image memory 120 stores response values, i.e., the relationship between low-light images and normal-light images. The stored response values ​​can be used for further re-enhancement. The learning network 20 described herein is advantageous in that an improved image enhancement network is provided when the stored relationship from the image memory 120 is applied.

[0212] The proposed external image memory 120 is designed as a plug-and-play mechanism that can be integrated with any existing enhancement system during testing to further improve the enhancement quality. This is very useful because the stored relationship between normal images and low-light images from the training dataset can be used in any image enhancement system. Since the image memory 120 is used to further enhance the enhanced image from the pre-trained image enhancer, the learning network has further advantages.

[0213] Although not required, the embodiments described with reference to the accompanying drawings may be implemented as an application programming interface (API) or a series of libraries used by developers, or may be included in another software application, such as a terminal or personal computer operating system or a portable computing device operating system. Generally, since program modules include routines, programs, objects, components, and data files that help perform specific functions, those skilled in the art will understand that the functionality of a software application may be distributed among multiple routines, objects, or components to achieve the same functionality required herein.

[0214] Unless otherwise indicated, any reference to prior art contained herein should not be taken as an admission that the information is common general knowledge.

[0215] Without departing from the scope of the present disclosure, one or more components and functions shown in the drawings may be rearranged and / or combined into a single component or embodied as several components. Additional elements or components may also be added without departing from the scope of the present disclosure.

[0216] It will be appreciated by those skilled in the art that many changes and / or modifications may be made to the present disclosure as shown in the specific embodiments without departing from the spirit or scope of the present disclosure as broadly described. Therefore, the present embodiments are to be considered in all aspects as illustrative and not restrictive.

Claims

1. A computer-implemented image enhancement method, characterized in that: The following steps are involved: Receiving an input image; wherein the input image is a low-light image; Processing the input image through a pre-trained image enhancer to generate an initial enhanced image; accessing, from the image memory, response values ​​corresponding to sample-specific attributes of the normal image; generating an adjustment factor based on the response value from the image memory; A final enhanced image is generated by applying the adjustment factor to the initial enhanced image.

2. The computer-implemented image enhancement method according to claim 1, characterized in that in, The adjustment factor is generated based on combining the response value and global average pooling data associated with the initial enhanced image.

3. The computer-implemented image enhancement method of claim 1, wherein: in, The adjustment factor is generated by applying an adaptive fusion function.

4. The computer-implemented image enhancement method of claim 3, wherein: in, Application of adaptive fusion functions includes: Calculate ratios of sample-specific attributes; Calculate the global average pooling data through element-by-element division; cascading the sample-specific attribute with the global average pooled data to generate a cascade value; Determining one or more weight vectors by applying a softmax function to the concatenated values; The adaptive adjustment factor is obtained based on the one or more weight vectors and a ratio of the sample-specific attribute to the global average pooling data.

5. The computer-implemented image enhancement method of claim 4, wherein: in, The step of obtaining the adaptive adjustment factor comprises: multiplying the first weight vector by the ratio of the sample-specific attribute to the global average pooling data, and summing the result with the second weight vector; The first weight vector corresponds to the sample-specific attribute, and the second weight vector corresponds to the global average pooling data.

6. The computer-implemented image enhancement method of claim 1, wherein: Include the following additional steps: Processing the input image by a feature generator to generate query features; The response value is accessed from the image memory based on the query feature.

7. The computer-implemented image enhancement method of claim 6, wherein: The following steps are involved: identifying within the image memory a memory key corresponding to the query feature; identifying a response value corresponding to the identified memory key; A response value is accessed from a memory address of the image memory corresponding to the memory key.

8. The computer-implemented image enhancement method of claim 7, wherein: in, The memory key is identified from a plurality of memory keys by calculating a cosine similarity between the query and a plurality of memory keys and identifying a memory key having a closest cosine similarity to the query.

9. The computer-implemented image enhancement method of claim 1, wherein: in, The step of processing the input image by the image enhancer comprises the following steps: Feeding the input image into a first convolutional layer; Extracting one or more low-level features by processing the input image through the first convolutional layer; wherein each of the low-level features is defined by a dimension including a spatial dimension and a plurality of channels; said one or more said low-level features passing through a four-stage symmetric encoder-decoder; An initial enhanced image is generated from the four-stage symmetric encoder-decoder.

10. The computer-implemented image enhancement method of claim 1, wherein: in, The method includes a memory writing process, and the memory writing process includes the following steps: receiving the response value corresponding to the query feature; By processing the reference image, a desired memory value is obtained; updating one or more memory keys in a dictionary using the response value; If the difference between the response value and the expected memory value is within a threshold, the one or more memory keys are updated.

11. The computer-implemented image enhancement method of claim 1, wherein: in, The method is applied to a low-light image to enhance the low-light image, and the image memory includes a plug-and-play mechanism for integration into the image enhancement method.

12. An image enhancement system, characterized in that: include: Image enhancement processor; An image memory for communicating with the image enhancement processor, the image memory defining a memory dictionary, the memory dictionary comprising one or more memory keys and one or more response values, each of the memory keys corresponding to a response value, wherein the response value corresponds to a value from a normal image; Wherein, the image enhancement processor is configured as follows: receiving an input image, wherein the input image is a low-light image, Processing the input image through a pre-trained image enhancer to generate an initial enhanced image, Access the response values ​​corresponding to the sample-specific attributes of the normal image from the image memory, generating an adjustment factor based on the response value from the image memory, By applying the adjustment factor to the initial enhanced image, a final enhanced image is output.

13. The image enhancement system according to claim 12, characterized in that: in, The image enhancement processor includes a learning network, and the learning network includes: a pre-trained image enhancer configured to receive one or more input images and generate the initial enhanced image; a pre-trained feature generator configured to receive the one or more input images and generate query features based on feeding the one or more input images to the feature generator; a memory reading module configured to receive the query feature and access the response value from the image memory based on the received query feature; an adaptive fusion module, configured to receive the response value and generate the adjustment factor based on the response value; The output module is configured to apply the adjustment factor to the initial enhanced image and output the final enhanced image.

14. The image enhancement system according to claim 13, characterized in that: in, The image enhancement processor is configured to generate global average pooling data from the initial enhanced image; The adaptive fusion module is configured as follows: Receive global average pooling data and response values ​​from the memory reading module, An adjustment factor is generated based on combining the response value and global average pooling data associated with the initial enhanced image; wherein the adaptive fusion module is configured to apply an adaptive fusion function to generate the adjustment factor.

15. The image enhancement system according to claim 14, characterized in that: in, The adaptive fusion module is further configured as: Calculate the ratio of the sample-specific attribute to the global average pooled data by element-wise division; cascading the sample-specific attribute with the global average pooled data to generate a cascade value; Determining one or more weight vectors by applying a softmax function to the concatenated values; The adaptive adjustment factor is obtained based on the one or more weight vectors and a ratio of the sample-specific attribute to the global average pooling data.

16. The image enhancement system according to claim 15, characterized in that: in, As part of generating the adaptive adjustment factor, the adaptive fusion module is configured to: multiplying the first weight vector by the ratio of the sample-specific attribute to the global average pooled data and summing with the second weight vector; The first weight vector corresponds to the sample-specific attribute, and the second weight vector corresponds to the global average pooling data.

17. The image enhancement system according to claim 16, characterized in that: in, The image enhancer is a pre-trained transformer based image enhancer comprising a symmetric encoder-decoder comprising a four-level pyramid feature map with skip connections.

18. The image enhancement system according to claim 13, characterized in that: in, The memory reading module is further configured as follows: identifying within the image memory a memory key corresponding to the query feature; identifying a response value corresponding to the identified memory key; A response value is accessed from a memory address of the image memory corresponding to the memory key.

19. The image enhancement system according to claim 18, characterized in that: in, The memory reading module is configured as follows: calculating a cosine similarity between the query and the plurality of memory keys; The memory key is identified from a plurality of memory keys based on an output of the cosine similarity calculation; wherein the memory key is identified as a memory key having a closest cosine similarity to the query.

20. The image enhancement system according to claim 19, characterized in that in, The image intensifier is further configured as: Feeding the received input image into the first convolutional layer; Processing the input image by applying the first convolutional layer; Based on the processing in the first convolutional layer, one or more low-level features are extracted; wherein each of the low-level features is defined by a dimension including a spatial dimension and a plurality of channels; passing said one or more said low-level features through a four-stage symmetric encoder-decoder; By applying the four-stage symmetric encoder-decoder, an initial enhanced image is generated.

Images