4K night scene image illumination compensation method and device for open scene

Abstract

The invention discloses a 4K night scene image illumination compensation method and device for an open scene, and the method comprises the steps: carrying out the illumination compensation of a night scene image through a first-stage network, and generating a preliminary illumination compensation image; and inputting the original night scene image and the initial illumination compensation image into the second-stage network at the same time, and finally generating a high-quality illumination compensation image by using detail texture information provided by the night scene image and information with full-image brightness perception provided by the initial illumination compensation image. According to the method, the brightness consistency of the illumination compensation image is ensured while the image quality is improved, a two-stage network is established to process the 4K-level image, and the problem that the brightness of the illumination compensation image is split due to the adoption of a block cutting strategy in the conventional method is solved.

Description

Technical Field

[0001] This invention belongs to the field of night scene image illumination compensation based on deep learning technology, and particularly relates to a method and apparatus for 4K night scene image illumination compensation in open scenes. Background Technology

[0002] With the continuous development of digital imaging technology, illumination compensation for night scene images has become an important research direction in computer vision. Early research mainly focused on illumination compensation for sRGB domain images. However, raw data can retain unprocessed sensor information, providing a wider dynamic range and richer details, and has received increasing attention in recent years, which is particularly crucial for illumination compensation of night scene images.

[0003] With the development of imaging sensors, ultra-high-definition imaging has become the foundation of modern vision systems. High-pixel-density devices can capture rich details, significantly improving the user's visual experience. However, applying existing illumination compensation methods to ultra-high-definition raw domain images often faces severe computational challenges. A common approach to address this challenge is to employ a block-based strategy. Specifically, it divides the 4K image into multiple image blocks, processes each block independently, and then reassembles them into a complete image. Because each image block is processed independently, the model lacks illumination context information from neighboring regions, leading to inconsistent brightness in the compensation results between adjacent image blocks. This results in visible block artifacts, severely impacting the subjective visual experience. Summary of the Invention

[0004] To address the issue of significant brightness fragmentation at the boundaries of clipping blocks caused by the aforementioned night scene image illumination compensation algorithms, this invention provides a 4K night scene image illumination compensation method and apparatus for open scenes. This method improves image quality while ensuring consistent brightness in the illumination compensation image. It establishes a two-stage network to process 4K images, solving the problem of brightness fragmentation in the illumination compensation image caused by the clipping strategy used in previous algorithms.

[0005] In a first aspect, embodiments of this application provide a method for illumination compensation of 4K night scene images in open scenes, the basic steps of which are as follows: First, the night scene image undergoes illumination compensation through a first-stage network to generate a preliminary illumination-compensated image with global brightness awareness.

[0006] The original night scene image and the preliminary illumination compensation image are preprocessed by cropping, and then input into the second-stage network in blocks.

[0007] The corresponding night scene image patch and the preliminary illumination compensation image patch are simultaneously input into the second-stage network. Based on the designed fine-tuning block, the network utilizes the detailed texture information provided by the night scene image and the information with full-image brightness perception provided by the preliminary illumination compensation image to finally generate a high-quality illumination compensation image patch.

[0008] Finally, all the illumination-compensated image blocks are stitched together according to their original spatial positions to restore the complete final illumination-compensated image.

[0009] In one possible implementation, the first-stage network has the following structure: The input image first passes through a 3×3 convolutional layer to obtain shallow features. Then, the shallow features are processed through a four-layer aligned encoder-decoder framework to extract deep features. In the encoding stage, each encoder layer contains a luminance encoding module. The decoding stage is symmetrical to the encoder structure, with each decoder layer containing a luminance decoding module. Finally, the obtained deep features pass through a 3×3 convolutional layer and a PixelShuffle module to obtain a preliminary illumination-compensated image.

[0010] In one possible implementation, the luma encoding module has the following structure: It consists of two 3×3 channel-wise convolutions, one 1×1 pointwise convolution, and a GELU activation function. The input features are first subjected to a channel-wise convolution to extract local spatial information, followed by a pointwise convolution for inter-channel feature fusion. Then, the GELU activation function introduces non-linear representation capabilities, and a second channel-wise convolution is performed to obtain intermediate features. Finally, the intermediate features are added to the input features to obtain the output features.

[0011] In one possible implementation, the luma decoding module has the following structure: First, the input features are sequentially passed through a Layer Normalization layer, a 1×1 convolutional layer, and a 3×3 channel-wise convolutional layer to extract initial spatial information. Then, the convolutional features are flattened, projecting them to generate a query vector, a key vector, and a value vector. Next, the query vector and key vector are multiplied by a matrix to obtain a transition matrix. Then, the transition matrix is ​​multiplied by the value vector to obtain the encoded global features, and the spatial dimension is restored through reconstruction. Finally, this global feature is sequentially subjected to a 1×1 pointwise convolution and a 3×3 channel-wise convolution to obtain the output features.

[0012] In one possible implementation, before inputting the image into the second-stage network, the initial illumination-compensated image is bilinearly downsampled to obtain a downsampled compensated image with the same size as the night scene image, thus achieving spatial scale alignment. The downsampled compensated image and the night scene image undergo block preprocessing, dividing them into several corresponding image block pairs, which are then input into the second-stage network in blocks.

[0013] In one possible implementation, the second-stage network has the following structure: Night scene image patches and downsampled compensation image patches are each processed by a 3×3 convolution to extract shallow texture features and shallow brightness features. Then, the extracted features are input into a layer-aligned encoder-decoder framework to extract global brightness-aware features; the first three layers of the encoder-decoder framework maintain a symmetrical structure, with the bottom layer containing only one encoder. Each encoder and decoder layer in this framework consists of a fine-tuning block. To accommodate the processing requirements of multi-scale fine-tuning blocks, the shallow brightness features are sequentially processed by three 3×3 convolution operations, each generating a brightness feature at one scale, resulting in three scale brightness features for subsequent multi-scale modeling. During decoding, a skip connection mechanism is introduced, concatenating the corresponding scale features from the encoder and the features from the decoder along the channel dimension, and then using a 1×1 convolution operation to reduce the number of channels to half, promoting information fusion. Furthermore, the downsampling and upsampling operations in the entire network are implemented through the PixelUnShuffle and PixelShuffle modules, respectively. Finally, the global brightness-aware features output by the encoder-decoder framework are first further refined into local and global features through a fine-tuning block, and then input into a 3×3 convolutional layer and a PixelShuffle module to obtain the final illumination-compensated image block.

[0014] In one possible implementation, the fine-tuning block consists of a multi-head self-attention module and a feedforward network. The two inputs to the fine-tuning block are texture features and brightness features at the corresponding scale. First, the texture features pass through a Layer Normalization layer in the multi-head self-attention module. Then, 1×1 and 3×3 convolutions are applied sequentially to fuse local and global feature information, resulting in joint features. These joint features are then divided into three sub-features along the channel dimension, serving as the query vector, key vector, and value vector, respectively. Simultaneously, the brightness features at the corresponding scale are divided into two parts along the channel dimension. The first quarter of the channel part is concatenated with the value vector, and the last three quarters of the channel part is concatenated with the query vector. These are then passed through a 1×1 convolution and a sigmoid activation function to obtain a new value vector and a new query vector with weighted modulation. Next, the new query vector and key vector are reshaped and multiplied to obtain a transition matrix. The new value vector is multiplied by the original value vector and then multiplied by the transition matrix to complete the context enhancement operation. Finally, the context-enhanced feature is convolved with 1×1 and added to the input night scene image feature to obtain the final output context feature.

[0015] The obtained context features are then fed into a feedforward network. Specifically, the context features first pass through a Layer Normalization layer; then, the features are simultaneously fed into two parallel branches: the upper branch passes through a 1×1 convolution and a 3×3 convolution sequentially, mainly extracting local contextual information; the lower branch also passes through 1×1 and 3×3 convolutions, and then through the GELU activation function to enhance non-linear expressiveness and improve feature discriminability. The output features of the two branches are multiplied element-wise. Subsequently, the fused features pass through a 1×1 convolution, and finally are added to the context features to obtain updated texture features, which serve as the input for the next layer's texture features.

[0016] In one possible implementation, both stages of the network incorporate independent loss supervision mechanisms. During training, both stages use the L1 loss function and employ backpropagation for learning. During training, both stages generate several image patches by randomly cropping the complete image, and these patches are then fed into the network for training.

[0017] Secondly, embodiments of this application provide a 4K night scene image illumination compensation device for open scenes, comprising the following modules: Illumination compensation module: The night scene image undergoes illumination compensation through the first-stage network to generate a preliminary illumination-compensated image with global brightness awareness.

[0018] Preprocessing module: Performs block preprocessing on the original night scene image and the preliminary illumination compensation image, and inputs them into the second-stage network in blocks.

[0019] Image enhancement module: The corresponding night scene image block and the preliminary illumination compensation image block are simultaneously input into the second-stage network. Based on the designed fine-tuning block, the detailed texture information provided by the night scene image and the information with full-image brightness perception provided by the preliminary illumination compensation image are used to finally generate a high-quality illumination compensation image block.

[0020] Image synthesis module: stitches together all the illumination-compensated image blocks generated by the image enhancement module according to their original spatial positions to restore the complete final illumination-compensated image.

[0021] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory; The memory is used to store computer programs.

[0022] When the processor executes the program stored in the memory, it implements any of the 4K night scene image illumination compensation methods described in this application.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the 4K night scene image illumination compensation methods described in this application.

[0024] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the 4K night scene image illumination compensation methods described in this application.

[0025] The beneficial effects of this invention are as follows: This invention proposes a two-stage network-based illumination compensation method for night scene images, effectively mitigating the block artifact problem in existing technologies. In the first stage, the network learns to generate a preliminary illumination-compensated image with global brightness awareness, which serves as auxiliary information input to the second-stage network. In the second stage, a fine-tuning block is designed, which can dynamically sense and adaptively adjust illumination changes in different regions, effectively coordinating the illumination distribution between adjacent image blocks during inference, thereby ensuring the consistency of brightness across the entire image. Compared to traditional single-stage processing methods, the two-stage network of this invention not only significantly improves the quality of the illumination-compensated image and eliminates the block artifacts caused by traditional block processing, but more importantly, ensures brightness consistency and visual coherence across the entire image. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0027] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention.

[0028] Figure 2 This is a network structure diagram for the first stage of an embodiment of the present invention.

[0029] Figure 3 This is a diagram of the second-stage network structure in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0031] This invention first defines and explains the following: ： Low-resolution night scene images ： Preliminary illumination compensation image ： Final illumination-compensated image ： Ultra-high definition, realistic normal lighting images To address the problems existing in the prior art, this invention proposes a method for illumination compensation of 4K night scene images in open environments. The method comprises two stages: the first stage utilizes a lightweight network to globally enhance the brightness of the entire image, ensuring overall brightness balance; the second stage uses the illumination compensation result from the first stage as a brightness guide to further enhance image details while suppressing block artifacts caused by image cropping, thereby improving image quality while ensuring brightness consistency and visual coherence of the illumination compensation results.

[0032] Implementation steps: This application proposes a method for illumination compensation of 4K night scene images in open scenes, such as... Figure 2 As shown, the basic steps are as follows: Step 1: Obtain a high-resolution night scene image dataset for training the network model.

[0033] Specifically, the publicly available See-in-the-Dark (SID) dataset is downloaded and used. The SID dataset is a dataset of real-world raw domain night scene images, where each normal lighting image corresponds to multiple night scene images taken under different exposure conditions. This dataset contains two subsets: the Sony subset and the Fuji subset, where the Sony subset images have a resolution of 4240×2832 and the Fuji subset images have a resolution of 6030×4032. This method uses the night scene images as input images for training the network model, while the normal lighting images serve as reference images, used for comparison with the illumination-compensated images predicted by the model during training. This embodiment uses the Sony subset as the dataset for training the network model.

[0034] Step 2: Construct a 4K night scene image illumination compensation network for open scenes.

[0035] Step 2.1: Theoretical Modeling for Illumination Compensation Task of 4K Night Scene Images in Open Scenes. Assume the input is a night scene RAW image, denoted as... Because this patent employs a two-stage network structure, it first... Input the first-stage network to achieve overall illumination compensation, and output the image after preliminary illumination compensation. Then, the second-stage network simultaneously receives... and As input, where As the primary input, it provides texture and structural information of the image; As auxiliary input, it provides brightness guidance to the second-stage network, enhancing the network's ability to perceive local illumination changes. Through the fusion of primary and auxiliary information, the second-stage network ultimately generates high-quality illumination-compensated images. This achieves synergistic optimization of brightness enhancement and detail restoration.

[0036] The theoretical model of a lightweight night scene illumination compensation network for 4K images can be expressed by the following formula: in This indicates the first phase of the network. This indicates the second phase of the network.

[0037] Step 2.2: Construct a training network for illumination compensation of 4K night scene images in open scenes, and train the network to form a network model that produces illumination compensation effects for night scene images.

[0038] The network consists of two main stages: a first stage for global illumination compensation and a second stage for detail restoration. First, the night scene image undergoes illumination compensation in the first stage, generating a preliminary illumination-compensated image with global brightness awareness. Then, the night scene image and the preliminary illumination-compensated image are simultaneously input into the second stage. Utilizing the detailed texture information provided by the night scene image and the global brightness awareness information provided by the preliminary illumination-compensated image, a high-quality illumination-compensated image is finally generated.

[0039] The specific structure of the first-stage network, such as Figure 2 As shown: This stage mainly includes a luminance encoding module and a luminance decoding module. Let the input night scene image be... First, shallow features are obtained through a 3×3 convolutional layer. Subsequently, shallow features Deep features are extracted through a four-layer aligned encoder-decoder framework. In the encoding stage, each encoder layer contains a luminance encoding module. The top three encoder layers are each followed by a 3×3 convolutional layer to downsampling the spatial dimensions while increasing the channel dimension and enhancing feature representation. The decoding stage is symmetrical to the encoder structure; each decoder layer contains a luminance decoding module, and the bottom three decoder layers are each followed by a 3×3 transposed convolutional layer to restore the spatial resolution of the features. The final deep features are obtained as follows. A preliminary illumination-compensated image is obtained after passing through a 3×3 convolutional layer and a PixelShuffle module. .

[0040] The specific structure of the luminance encoding module: This module mainly consists of two 3×3 channel-wise convolutions, one 1×1 pointwise convolution, and a GELU activation function. The input features first undergo a channel-wise convolution to extract local spatial information, followed by a pointwise convolution for inter-channel feature fusion. Then, the GELU activation function introduces non-linear representation capabilities to enhance feature expression. A second channel-wise convolution is then performed to obtain intermediate features. Finally, these intermediate features are added to the input features to obtain the output features.

[0041] The specific structure of the luminance decoding module is as follows: First, the input features are sequentially passed through a Layer Normalization layer, a 1×1 convolutional layer, and a 3×3 channel-wise convolutional layer to extract initial spatial information. Then, the convolutional features are flattened and projected to generate a query vector. Key vector Sum value vector Next, the query vector will be... and key vector Matrix multiplication is performed to obtain the transition matrix. This is used to model the global dependencies between channels. Then, the transition matrix is ​​used... AND value vector Multiply to obtain the encoded global features. And restored to spatial dimensions through reconstruction. Finally, the global feature is output by performing a 1×1 pointwise convolution and a 3×3 channelwise convolution.

[0042] Before inputting the preliminary illumination-compensated image into the second-stage network... Bilinear downsampling was performed to obtain images of the same size as the night scene image. Consistent downsampling compensated image To achieve spatial scale alignment, the downsampled compensated image and the night scene image are preprocessed by cropping, dividing them into several corresponding image block pairs, and then inputting them into the second-stage network in blocks.

[0043] like Figure 3 As shown, the specific structure of the second-stage network is as follows: Night scene image blocks and downsampling compensated image patches Shallow texture and lightness features are extracted using a 3×3 convolution. These two extracted features are then input into a layer-aligned encoder-decoder framework to extract global lightness-aware features. The first three layers of the encoder-decoder framework maintain a symmetrical structure, with the bottom layer containing only one encoder. Each encoder and decoder layer in this framework consists of a fine-tuning block, used to finely adjust the feature distribution at different scales, thereby enhancing the modeling ability for complex lightness distributions and local details. To accommodate the processing requirements of multi-scale fine-tuning blocks, the shallow lightness features are sequentially processed through three 3×3 convolution operations, each generating a lightness feature at one scale, resulting in three scales of lightness features for subsequent multi-scale modeling. During decoding, to preserve the structural details of high-resolution images, a skip connection mechanism is introduced, concatenating the corresponding scale features from the encoder and the decoder along the channel dimension, and then using a 1×1 convolution operation to reduce the number of channels to half, promoting information fusion. Furthermore, downsampling and upsampling operations throughout the network are implemented through the PixelUnShuffle and PixelShuffle modules, respectively, effectively preserving structural information and improving reconstruction quality. Finally, the global luminance-sensing features output by the encoder-decoder framework are first further refined into local and global features through a fine-tuning block, and then input into a 3×3 convolutional layer to expand the number of channels to 12, denoted as . Finally, the PixelShuffle module is used to... Restored to a spatial resolution of The final illumination-compensated image with 3 channels. .

[0044] The fine-tuning block's specific structure: This module aims to achieve deep fusion and contextual modeling of texture and brightness features, thereby improving global brightness consistency and local detail restoration quality. Its structure consists of a multi-head self-attention module and a feedforward network. Let the two inputs to this module be texture features... and brightness characteristics First, input texture features. First, a LayerNormalization layer is applied in the multi-head self-attention module. Then, 1×1 and 3×3 convolution operations are used sequentially to fuse local and global feature information, obtaining joint features. These joint features are then divided into three sub-features along the channel dimension, each serving as a query vector. Key vector Sum value vector All dimensions At the same time, the brightness features at the corresponding scale are... Divided into two parts along the channel dimension: the first 1 / 4 of the channel is The last 3 / 4 channels are Then, and By splicing , and By splicing This stitching mechanism explicitly incorporates information from brightness features to modulate the attention mechanism's focus area on texture features. Subsequently, and After passing through a 1×1 convolution and a sigmoid activation function, the new value vector after weight modulation is obtained. and new query vector Next, and After reshaping, the matrix multiplication yields the transition matrix. This is used to model the relationships between different channels. Similarly, [the following is used to]... and The input features are multiplied and then multiplied again with the transition matrix M to complete the context enhancement operation. Finally, the resulting context-enhanced features are convolved with the input using a 1×1 convolution. Adding them together yields the context features. Subsequently, the obtained context features The input is fed into a feedforward network to further enhance nonlinear expressive power and fuse multi-scale contextual information. Specifically, First, the feature distribution is normalized through a LayerNormalization layer to enhance network stability. Then, the feature is simultaneously fed into two parallel branches: the upper branch passes through a 1×1 convolution and a 3×3 convolution sequentially to extract local contextual information; the lower branch similarly passes through 1×1 and 3×3 convolutions, followed by a GELU activation function to enhance non-linear expressiveness and improve feature discriminativeness. The output features of the two branches are multiplied element-wise to achieve cross-channel information modulation and fusion. Subsequently, the fused feature passes through a 1×1 convolution to integrate information and adjust the channel structure, and finally is combined with the input feature. Adding them together yields updated texture features. This serves as the input for the texture features of the next level.

[0045] The network structure described in this invention consists of two-stage networks, each incorporating an independent loss supervision mechanism. During training, both stages use the L1 loss function, and the output image is reduced through backpropagation in deep learning. , and truth image This effectively improves the overall enhancement performance by eliminating the gap in image size. During training, both stages of the network use random cropping of the complete image to generate image patches of size 4×512×512, which are then fed into the network for training.

[0046] Step 3: The trained network model receives the night scene image to be processed, completes the illumination compensation for the night scene image, and outputs the image.

[0047] First, the weights of the night scene image illumination compensation network model trained in step two are loaded. Then, the complete night scene image is used as input data and fed into the first-stage network to obtain a preliminary illumination-compensated image. Next, the original night scene image and the preliminary illumination-compensated image undergo block preprocessing, i.e., they are divided into several image blocks, and each block is input into the second-stage network. The second-stage network compensates for each pair of image blocks, outputting multiple final illumination-compensated image blocks. Finally, all illumination-compensated image blocks are stitched together according to their original spatial positions to restore the complete final illumination-compensated image.

[0048] Step 4: Experimental Results.

[0049] The effectiveness of our method is validated on the Sony subset of the SID dataset. Compared to the previous state-of-the-art full-graph inference method, our method outperforms it by 0.21 dB and 0.014 in PSNR and LPIPS, respectively. Compared to the segmented inference method, our method outperforms it by 0.76 dB and 1.26 dB in PSNR and PSNR-B, respectively.

[0050] Table 1: Quantitative Results This application also provides a 4K night scene image illumination compensation device for open scenes, including the following modules: Brightness compensation module: The night scene image undergoes illumination compensation through the first-stage network to generate a preliminary illumination-compensated image with global brightness awareness.

[0051] Preprocessing module: Performs block preprocessing on the original night scene image and the preliminary illumination compensation image, and inputs them into the second-stage network in blocks.

[0052] Image enhancement module: The corresponding night scene image block and the preliminary illumination compensation image block are simultaneously input into the second-stage network. Based on the designed fine-tuning block, the detailed texture information provided by the night scene image and the information with full-image brightness perception provided by the preliminary illumination compensation image are used to finally generate a high-quality illumination compensation image block.

[0053] Image synthesis module: stitches together all the illumination-compensated image blocks generated by the image enhancement module according to their original spatial positions to restore the complete final illumination-compensated image.

[0054] In one possible implementation, the specific structure of the first-stage network of the illumination compensation module is as follows: The input image first passes through a 3×3 convolutional layer to obtain shallow features. Subsequently, the shallow features are passed through a four-layer aligned encoder-decoder framework to extract depth features. In the encoding stage, each encoder layer contains a luminance encoding module. The decoding stage is symmetrical to the encoder structure, with each decoder layer containing a luminance decoding module. Finally, the obtained deep features pass through a 3×3 convolutional layer and a PixelShuffle module to obtain the preliminary illumination-compensated image.

[0055] In one possible implementation, the luma encoding module has the following structure: It consists of two 3×3 channel-wise convolutions, one 1×1 pointwise convolution, and a GELU activation function. The input features are first subjected to a channel-wise convolution to extract local spatial information, followed by a pointwise convolution for inter-channel feature fusion. Then, the GELU activation function introduces non-linear representation capabilities, and a second channel-wise convolution is performed to obtain intermediate features. Finally, the intermediate features are added to the input features to obtain the output features.

[0056] In one possible implementation, the luma decoding module has the following structure: First, the input features are sequentially passed through a Layer Normalization layer, a 1×1 convolutional layer, and a 3×3 channel-wise convolutional layer to extract initial spatial information. Then, the convolutional features are flattened, projecting them to generate a query vector, a key vector, and a value vector. Next, the query vector and key vector are multiplied by a matrix to obtain a transition matrix. Then, the transition matrix is ​​multiplied by the value vector to obtain the encoded global features, and the spatial dimension is restored through reconstruction. Finally, this global feature is sequentially subjected to a 1×1 pointwise convolution and a 3×3 channel-wise convolution to obtain the output features.

[0057] In one possible implementation, the preprocessing module performs bilinear downsampling on the initial illumination-compensated image to obtain a downsampled compensated image with the same size as the night scene image, thereby achieving spatial scale alignment. The downsampled compensated image and the night scene image undergo block preprocessing, dividing them into several corresponding image block pairs, which are then input into the second-stage network in block-by-block form.

[0058] In one possible implementation, the second-stage network of the image enhancement module has the following structure: Night scene image blocks and downsampling compensation image blocks are each processed by a 3×3 convolution to extract shallow texture features and shallow brightness features. Then, the extracted features are input into a layer-aligned encoder-decoder framework to extract global brightness-aware features; the first three layers of the encoder-decoder framework maintain a symmetrical structure, with the bottom layer containing only one encoder. Each encoder and decoder layer in this framework consists of a fine-tuning block. To accommodate the processing requirements of multi-scale fine-tuning blocks, the shallow brightness features are sequentially processed by three 3×3 convolution operations, each generating a brightness feature at one scale, resulting in three scale brightness features for subsequent multi-scale modeling. During decoding, a skip connection mechanism is introduced, concatenating the features at the corresponding scale in the encoder with the features in the decoder along the channel dimension, and then reducing the number of channels to half the original number through a 1×1 convolution operation, promoting information fusion. Furthermore, the downsampling and upsampling operations in the entire network are implemented through the PixelUnShuffle and PixelShuffle modules, respectively. Finally, the global brightness-aware features output by the encoder-decoder framework are first further refined into local and global features through a fine-tuning block, and then input into a 3×3 convolutional layer and the final illumination-compensated image block obtained by the PixelShuffle module.

[0059] In one possible implementation, the fine-tuning block consists of a multi-head self-attention module and a feedforward network. The two inputs to the fine-tuning block are texture features and brightness features at the corresponding scale. First, the texture features pass through a Layer Normalization layer in the multi-head self-attention module. Then, 1×1 and 3×3 convolutions are applied sequentially to fuse local and global feature information, resulting in joint features. These joint features are then divided into three sub-features along the channel dimension, serving as the query vector, key vector, and value vector, respectively. Simultaneously, the brightness features at the corresponding scale are divided into two parts along the channel dimension. The first quarter of the channel part is concatenated with the value vector, and the last three quarters of the channel part is concatenated with the query vector. These are then passed through a 1×1 convolution and a sigmoid activation function to obtain a new value vector and a new query vector with weighted modulation. Next, the new query vector and key vector are reshaped and multiplied to obtain a transition matrix. The new value vector is multiplied by the original value vector and then multiplied by the transition matrix to complete the context enhancement operation. Finally, the context-enhanced feature is convolved with 1×1 and added to the input night scene image feature to obtain the final output context feature.

[0060] The obtained context features are then fed into a feedforward network. Specifically, the context features first pass through a Layer Normalization layer; then, the features are simultaneously fed into two parallel branches: the upper branch passes through a 1×1 convolution and a 3×3 convolution sequentially, mainly extracting local contextual information; the lower branch also passes through 1×1 and 3×3 convolutions, and then through the GELU activation function to enhance non-linear expressiveness and improve feature discriminability. The output features of the two branches are multiplied element-wise. Subsequently, the fused features pass through a 1×1 convolution, and finally are added to the context features to obtain updated texture features, which serve as the input for the next layer's texture features.

[0061] In one possible implementation, the device further includes a training module: independent loss supervision mechanisms are introduced in both the brightness compensation module and the image enhancement module's two-stage networks. During training, both-stage networks use the L1 loss function and employ backpropagation learning. During training, both-stage networks generate several image patches by randomly cropping the complete image, and these patches are used as input to the network for training.

[0062] This application also provides an electronic device, which includes a processor and a memory.

[0063] The memory is used to store computer programs.

[0064] When the processor executes a program stored in the memory, it implements any of the methods described in this application.

[0065] In one possible implementation, the electronic device of this application embodiment further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0066] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0067] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0068] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0069] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0070] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the methods described in this application.

[0071] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in this application.

[0072] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0073] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0074] The various embodiments in this specification are described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0075] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for illumination compensation of 4K night scene images in open scenes, characterized in that, The basic steps are as follows: First, the night scene image undergoes illumination compensation through the first-stage network to generate a preliminary illumination-compensated image with global brightness awareness; The original night scene image and the preliminary illumination compensation image are preprocessed by cropping, and then input into the second-stage network in blocks. The corresponding night scene image patch and the preliminary illumination compensation image patch are simultaneously input into the second stage network. Based on the designed fine-tuning block, the detailed texture information provided by the night scene image and the information with full-image brightness perception provided by the preliminary illumination compensation image are used to finally generate a high-quality illumination compensation image patch. Finally, all the illumination-compensated image blocks are stitched together according to their original spatial positions to restore the complete final illumination-compensated image.

2. The illumination compensation method for 4K night scene images in open scenes according to claim 1, characterized in that, The specific structure of the first-stage network is as follows: The input image first passes through a 3×3 convolutional layer to obtain shallow features; then, the shallow features pass through a four-layer aligned encoder-decoder framework to extract deep features. During the encoding phase, each encoder layer contains a luminance encoding module; The decoding stage is symmetrical to the encoder structure, with each decoder layer containing a luminance decoding module; the final deep features are processed through a 3×3 convolutional layer and a PixelShuffle module to obtain a preliminary illumination-compensated image.

3. The illumination compensation method for 4K night scene images in open scenes according to claim 2, characterized in that, The specific structure of the luminance encoding module is as follows: This module consists of two 3×3 channel-wise convolutions, one 1×1 pointwise convolution, and a GELU activation function. The input features first undergo a channel-wise convolution to extract local spatial information, followed by a pointwise convolution for inter-channel feature fusion. Then, the GELU activation function introduces non-linear representation capabilities, and the input features are obtained through a second channel-wise convolution. Finally, the intermediate features are added to the input features to obtain the output features.

4. The illumination compensation method for 4K night scene images in open scenes according to claim 2, characterized in that, The specific structure of the luminance decoding module is as follows: First, the input features are sequentially passed through a Layer Normalization layer, a 1×1 convolutional layer, and a 3×3 channel-wise convolutional layer to extract initial spatial information; Subsequently, the convolutional features are flattened and projected to generate query vectors, key vectors, and value vectors. Then, the query vectors and key vectors are multiplied by matrix to obtain the transition matrix. Next, the transition matrix is ​​multiplied by the value vector to obtain the encoded global features, and the spatial dimension is restored through reconstruction. Finally, the global features are sequentially subjected to a 1×1 pointwise convolution and a 3×3 channelwise convolution to obtain the output features.

5. The illumination compensation method for 4K night scene images in open scenes according to claim 1, characterized in that, Before inputting into the second-stage network, the initial illumination compensation image is bilinearly downsampled to obtain a downsampled compensation image with the same size as the night scene image, so as to achieve spatial scale alignment. The downsampled compensated image and the night scene image are preprocessed by cropping. The downsampled compensated image and the night scene image are divided into several corresponding image block pairs, and each block is input into the second stage network.

6. The illumination compensation method for 4K night scene images in open scenes according to claim 1, characterized in that, The specific structure of the second-stage network is as follows: the night scene image patch and the downsampled compensated image patch are respectively processed by a 3×3 convolution to extract shallow texture features and shallow brightness features; then, the two extracted features are input into the layer-aligned encoder-decoder framework to extract global brightness perception features. The encoder-decoder framework maintains a symmetrical structure in its first three layers, with the bottom layer containing only one encoder. Each encoder and decoder layer in this framework consists of a fine-tuning block. To accommodate the processing requirements of multi-scale fine-tuning blocks, shallow brightness features are sequentially processed through three 3×3 convolution operations, each generating a brightness feature at one scale, resulting in three brightness features for subsequent multi-scale modeling. During decoding, a skip connection mechanism is introduced to concatenate the corresponding scale features from the encoder and the features from the decoder along the channel dimension, and a 1×1 convolution operation is used to reduce the number of channels to half, promoting information fusion. Furthermore, downsampling and upsampling operations in the entire network are implemented through the PixelUnShuffle and PixelShuffle modules, respectively. Finally, the global brightness-aware features output by the encoder-decoder framework are first further refined into local and global features through a fine-tuning block, and then input into a 3×3 convolutional layer and the PixelShuffle module to obtain the final illumination-compensated image block.

7. The illumination compensation method for 4K night scene images in open scenes according to claim 6, characterized in that, The fine-tuning block consists of a multi-head self-attention module and a feedforward network. The two inputs to the fine-tuning block are texture features and brightness features at the corresponding scale. First, the texture features pass through a LayerNormalization layer in the multi-head self-attention module. Then, 1×1 and 3×3 convolutions are applied sequentially to fuse local and global feature information, resulting in a joint feature. This joint feature is then divided into three sub-features along the channel dimension, serving as the query vector, key vector, and value vector, respectively. Simultaneously, the brightness features at the corresponding scale are divided into two parts along the channel dimension. The first quarter of the channel part is concatenated with the value vector, and the last three quarters of the channel part is concatenated with the query vector. These parts are then passed through a 1×1 convolution and a Sigmoid activation function to obtain a new value vector and a new query vector with weighted modulation. Next, the new query vector and key vector are reshaped and multiplied to obtain a transition matrix. The new value vector is multiplied by the original value vector and then multiplied by the transition matrix to complete the context enhancement operation. Finally, this context-enhanced feature is convolved with 1×1 and added to the input night scene image features to obtain the final output context feature. Subsequently, the obtained contextual features are input into the feedforward network; specifically, the contextual features first pass through a LayerNormalization layer; Then, the feature is fed into two parallel branches simultaneously: the upper branch passes through a 1×1 convolution and a 3×3 convolution in sequence to extract local context information; the lower branch also passes through 1×1 and 3×3 convolutions first, and then through the GELU activation function to enhance non-linear expressive power and improve the discriminativeness of the feature. The output features of the two branches are multiplied element-wise; then, the fused features are passed through a 1×1 convolution, and finally added to the context features to obtain the updated texture features, which are used as the input for the texture features of the next level.

8. The illumination compensation method for 4K night scene images in open scenes according to claim 1, characterized in that, Both stages of the network introduce independent loss supervision mechanisms; during training, both stages use the L1 loss function and employ backpropagation learning; during training, both stages generate several image patches by randomly cropping the complete image and use them as input to train the network.

9. A 4K night scene image illumination compensation device for open scenes, characterized in that, Includes the following modules: Illumination compensation module: The night scene image undergoes illumination compensation through the first-stage network to generate a preliminary illumination compensation image with global brightness awareness; Preprocessing module: Performs block preprocessing on the original night scene image and the preliminary illumination compensation image, and inputs them into the second-stage network in blocks; Image enhancement module: The corresponding night scene image block and the preliminary illumination compensation image block are simultaneously input into the second stage network. Based on the designed fine-tuning block, the detailed texture information provided by the night scene image and the information with full-image brightness perception provided by the preliminary illumination compensation image are used to finally generate a high-quality illumination compensation image block. Image synthesis module: stitches together all the illumination-compensated image blocks generated by the image enhancement module according to their original spatial positions to restore the complete final illumination-compensated image.

10. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the 4K night scene image illumination compensation method according to any one of claims 1-8.

Images