Method, system and computer readable medium for improving image color quality

By using convolutional blocks, global pooling layers and jump-connected encoders in the compiled network, the global information and local information are integrated, and the color inconsistency in low-light or underwater images is solved, and the signal-to-noise ratio of the image is improved. Denoising effect.

CN113939845BActive Publication Date: 2024-11-08GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980096730.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-05-31
Filing Date
2019-12-02
Publication Date
2024-11-08
Estimated Expiration
2039-12-02

AI Technical Summary

Technical Problem

Images collected under low light or underwater conditions are difficult to recognize due to low signal-to-noise ratio, low contrast and narrow dynamic range. In the prior art, the coding network SID-net has halo and color inconsistency during color repair. The problem.

Method used

Using an encoder containing convolutional blocks, global pooling layer and jump connections, the fused feature map is generated to ensure the color consistency of different parts of the image, and the compilation network of U-net architecture is used to perform images. Denoising and enhancing.

Benefits of technology

The signal-to-noise ratio (PSNR) performance of the image is improved, the color inconsistency is solved, and more efficient image denoising and enhancement effects are achieved without adding learnable parameters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113939845B_ABST
    Figure CN113939845B_ABST
Patent Text Reader

Abstract

In one embodiment, a method includes: receiving and processing a first color image by an encoder. The first color image includes a first portion and a second portion located at different positions of the first color image. The encoder is used to output at least one first feature map including fused global information and local information, so that whether a color consistency relationship between the first portion and the second portion of the first color image exists is encoded as the fused global information and the local information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, a system and a computer-readable medium for amplifying the color quality of an image. Background Art

[0002] When images are captured in low light or underwater conditions, the content of the image may be difficult to discern due to low signal-to-noise ratio (SNR), low contrast, and / or narrow dynamic range. Image denoising techniques can remove image noise. Image enhancement techniques can optimize perceived quality, such as image contrast. Image denoising techniques and / or image enhancement techniques aim to provide images with rich colors and details in low light, underwater, and other conditions.

[0003] In the prior art described in “Learning to see in the dark,” C. Chen, Q. Chen, J. Xu, V. Koltun, In CVPR, 2018, a codec network SID-net applies image denoising and / or image enhancement to an input color image to generate an output color image. The codec network SID-net includes an encoder and a decoder. The encoder obtains an abstract representation of color information by extracting features of the input color image. In order to obtain the output color image, the color information of the abstract representation needs to be repaired by the decoder. Figure 1 Schematically shows a coding network SID-net in the related art inputting a color image 102 and outputting a color image 122. Figure 1 , the input color image 102 contains a chair that is difficult to identify due to low SNR, low contrast and / or narrow dynamic range. The first portion 104 of the input color image 102 is the edge of a chair back, and the second portion 106 of the input color image 102 is the middle portion of the chair back. Although the edge of the chair back and the middle portion of the chair back have the same color, when the color information of the abstract representation is restored by the decoder, the first portion 124 of the output color image 122 corresponding to the first portion 104 of the input color image 102 may have inconsistent colors. The color inconsistency between the edge and the middle portion of the chair back in the output color image 122 is shown by the dotted line of the edge of the chair back. This color inconsistency is called halo. Other color inconsistencies, such as spots recorded in the related art, are similar to halos. Summary of the invention

[0004] The present disclosure provides a method, system, and computer-readable medium for optimizing image color quality.

[0005] The first solution provided by the present disclosure is a computer-implemented method, which includes: receiving and processing a first color image through an encoder. The first color image includes a first part and a second part of the first color image located at different positions of the first color image. The encoder is used to output at least one first feature map, and the at least one first feature map includes fused global information and local information, then the existence or non-existence of the color consistency relationship between the first part and the second part of the first color image is encoded as fused global information and local information. The encoder includes a first block, a second block and a first jump connection. The first block includes a convolution block, which is used to output at least one second feature map including local information and includes a first receptive field. The second block includes a global pooling layer for globally pooling at least one second feature map and outputting at least one third feature map containing global information. The second block also includes a second receptive field, and the second receptive field is wider than the first receptive field. The second block also includes an enlargement layer, which is used to enlarge at least one third feature map and output at least one fourth feature map, wherein the at least one fourth feature map has the same size as the at least one second feature map, and the at least one fourth feature map includes a global information feature map; the second block also includes a first jump connection, which is used to fuse the at least one second feature map and the at least one fourth feature map, and generate at least one first feature map, wherein the at least one first feature map has the same number of channels as the at least one second feature map, wherein the fused global information and local information are obtained from the global information and the local information.

[0006] The second solution provided by the present disclosure is a system, which includes: at least one memory for storing program instructions; at least one processor for executing program instructions, so that at least one processor performs the steps: the encoder receives and processes a first color image. Wherein, the first color image includes a first part and a second part of the first color image located at different positions of the first color image. The encoder is used to output at least one first feature map, and the at least one first feature map includes fused global information and local information, then the existence or non-existence of the color consistency relationship between the first part and the second part of the first color image is encoded as fused global information and local information. The encoder includes a first block, a second block and a first jump connection. The first block includes a convolution block, which is used to output at least one second feature map including local information and includes a first receptive field. The second block includes a global pooling layer for globally pooling at least one second feature map and outputting at least one third feature map containing global information. The second block also includes a second receptive field, and the second receptive field is wider than the first receptive field. The second block also includes an enlargement layer, which is used to enlarge at least one third feature map and output at least one fourth feature map, wherein the at least one fourth feature map has the same size as the at least one second feature map, and the at least one fourth feature map includes a global information feature map; the second block also includes a first jump connection, which is used to fuse the at least one second feature map and the at least one fourth feature map, and generate at least one first feature map, wherein the at least one first feature map has the same number of channels as the at least one second feature map, wherein the fused global information and local information are obtained from the global information and the local information.

[0007] The third solution provided by the present disclosure is a non-transitory computer-readable medium storing program instructions, including: at least one memory for storing program instructions; at least one processor for executing program instructions, so that at least one processor performs the following steps: the encoder receives and processes a first color image, wherein the first color image includes a first part and a second part of the first color image located at different positions of the first color image. The encoder is used to output at least one first feature map, the at least one first feature map includes fused global information and local information, and whether the color consistency relationship between the first part and the second part of the first color image exists is encoded as fused global information and local information. The encoder includes a first block, a second block and a first jump connection. The first block includes a convolution block, which is used to output at least one second feature map including local information and includes a first receptive field. The second block includes a global pooling layer for globally pooling at least one second feature map and outputting at least one third feature map containing global information. The second block also includes a second receptive field, and the second receptive field is wider than the first receptive field. The second block also includes an enlargement layer, which is used to enlarge at least one third feature map and output at least one fourth feature map, wherein the at least one fourth feature map has the same size as the at least one second feature map, and the at least one fourth feature map includes a global information feature map; the second block also includes a first jump connection, which is used to fuse the at least one second feature map and the at least one fourth feature map, and generate at least one first feature map, wherein the at least one first feature map has the same number of channels as the at least one second feature map, wherein the fused global information and local information are obtained from the global information and the local information. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present invention or related technologies, the following drawings will be described when briefly introducing the embodiments. Obviously, the drawings are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0009] Figure 1 A schematic diagram showing an input color image and an output color image of a compilation network in the related art.

[0010] Figure 2 A block diagram of input, processing, and output hardware modules in a terminal in an embodiment of the present disclosure is shown.

[0011] Figure 3 A schematic diagram illustrating a compilation network for image denoising and / or enhancement in an embodiment of the present disclosure is shown.

[0012] Figure 4An input color image, an output color image, and intermediate details for color restoration performed by a compilation network in an embodiment of the present disclosure are shown.

[0013] Figure 5 Another input color image, another output color image, and another intermediate detail of color restoration performed by the compiled network in an embodiment of the present disclosure are shown. DETAILED DESCRIPTION

[0014] The technical matters, structural features, objectives and effects of the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. Specifically, the terms in the embodiments of the present invention are only used to illustrate the purpose of a certain embodiment and are not used to limit the present invention.

[0015] As used herein, the term "using" refers to a case where an object is directly used to perform a step, or a case where an object is modified through at least one intermediate step and the modified object is directly used to perform a step.

[0016] Figure 2 FIG. 2 is a block diagram showing input, processing and output hardware modules in a terminal 200 according to an embodiment of the present disclosure. Figure 2 , the terminal 200 includes a digital camera module 202, a processor module 204, a memory module 206, a display module 208, a storage module 210, a wired or wireless communication module 212 and a bus 214. The terminal 200 can be a mobile phone, a smart phone, a tablet computer, a laptop computer, a desktop computer or any electronic device with sufficient computing power for image processing.

[0017] The digital camera module 202 is an input hardware module for collecting input color images I1 ( Figure 3 ), and transmitted to the processor module 204 via the bus 214. The input color image I1 may be an original image whose pixels are arranged in a Bayer pattern. Alternatively, the input color image I1 may be obtained through other input hardware modules, such as the storage module 210, or the wired or wireless communication module 212. The storage module 210 is used to store the input color image I1 transmitted to the processor module 204 via the bus 214. The wired or wireless communication module 212 is used to receive the input color image I1 from the network via wired or wireless communication, wherein the input color image I1 is transmitted to the processor module 204 via the bus 214.

[0018] When the input color image is captured, for example, in low light conditions or underwater conditions, or when the exposure time is insufficient, the content of the image may be difficult to discern due to low signal-to-noise ratio (SNR), low contrast, and / or narrow dynamic range. The memory module 206 may be a temporary or non-temporary computer-readable medium, which includes at least one memory storing program instructions that, when executed by the processor module 204, cause the processor module 204 to process the input color image I1. The processor module 204 may be used as a codec network 300 (e.g., Figure 3 ), which performs image denoising and / or enhancement on the input color image I1 and generates an output color image I2 (in Figure 3 ). The processor module 204 includes at least one processor, which directly or indirectly sends and / or receives signals to the digital camera module 202, the memory module 206, the display module 208, the storage module 210, and the wired or wireless communication module 212 via the bus 214. The at least one processor may be a central processing unit (CPU(s)), a graphics processing unit (GPU(s)), and / or a digital signal processor (DSP(s)). The CPU may send the input color image I1, some program instructions, and other data or instructions to the GPU and / or the DSP via the bus 214.

[0019] The display module 208 is an output hardware module for displaying the output color image I2 received from the processor module 204 via the bus 214. Alternatively, the output color image I2 may be output using another output hardware module, such as the storage module 210, or the wired or wireless communication module 212. The storage module 210 is used to store the output color image I2 received from the processor module 204 via the bus 214. The wired or wireless communication module 212 is used to transmit the output color image I2 to a network via wired or wireless communication, wherein the output color image I2 is received from the processor module 204 via the bus 214.

[0020] Terminal 200 is a system in which all components are integrated together via bus 214. Other types of computing systems, such as systems having remote digital camera modules instead of digital camera module 202, are also within the contemplated scope of the present disclosure.

[0021] Figure 3 is a schematic diagram of a coding network 300 for image denoising and / or enhancement according to an embodiment of the present disclosure. Given an input image I, the coding network 300 learns a graph I'=f(I:w) that presents the input image denoised and / or enhanced by I to generate an output image I', where w is a set of learnable parameters of the coding network 300. The coding network 300 with the learned parameters performs image denoising and / or enhancement on the input color image I1 to generate an output color image I2.

[0022] The coding network 300 includes an encoder 302 and a decoder 304. The encoder 302 extracts the features of the input color image I1 to obtain an abstract representation with color information. In order to obtain the output color image I2, the decoder 304 needs to repair the color information of the abstract representation.

[0023] Reference Figure 3 In one embodiment, the encoding network 300 has a U-net architecture. The encoder 302 is used to output at least one first feature map, including fused global information and local information, so that whether there is a color consistency relationship between the first part of the input color image I1 and the second part of the input color image I1 is encoded, and the global information and the local information are fused. The first part of the input color image I1 and the second part of the input color image I1 are located at different positions of the input color image I1. In one embodiment, at least one first feature map is a multi-channel feature map. The encoder 302 includes a first block 306, a second block 308, and a jump connection 310. The first block 306 includes multiple stages from A to E. Each stage from A to E includes a convolution block X1 and a convolution block X2 after the convolution block X1, where X represents any stage from A to E. Each stage from A to D also includes a pooling layer X3 after the convolution block X2, where X is any stage from A to D. In one embodiment, each convolution block X1 and X2 includes a 3x3 convolution layer. Other kernel sizes of the convolutional layers of the convolutional blocks X1 and X2 are within the expected range of the present disclosure. Each convolutional block X1 and X2 also includes a nonlinear activation function after a 3x3 convolutional layer. In one embodiment, the nonlinear activation function is a Leaky ReLU operation. In one embodiment, the pooling layer X3 is a maximum pooling layer. Alternatively, the pooling layer X3 is an average pooling layer. In the first block 306, the size decreases step by step from stage A to stage D, so that the size of at least the first feature map is smaller than the size of the input color image I1. The number of channels increases step by step from stage A to stage E, so that the number of channels of at least one first feature map is greater than the number of channels of the input color image I1.

[0024] The convolution block E2 is used to output at least one second feature map including local information. The convolution block E2 has a first receptive field, that is, the 3x3 convolution layer of the convolution block E2 has a first receptive field.

[0025] The second block 308 includes a global pooling layer F1 and an amplification layer F2. The global pooling layer F1 is used to perform global pooling on at least one second feature map and output at least one third feature map containing global information. The global pooling layer F1 has a second receptive field wider than the first receptive field. In one embodiment, the pooling layer F1 is a global maximum pooling layer. Alternatively, the global pooling layer F1 is a global average pooling layer. The amplification layer F2 is used to amplify at least one third feature map and output at least one fourth feature map having the same size as at least one second feature map and including global information. In one embodiment, since the input color image I1 has an arbitrary size, if the amplification layer F2 is a deconvolution layer, the stride of the deconvolution layer is dynamic, which is undesirable. Therefore, the amplification layer F2 is an upsampling layer without learnable parameters, such as an upsampling layer that performs a bilinear upsampling operation, a linear upsampling operation, a bicubic upsampling operation, or a nearest upsampling operation on at least one third of the feature maps. In one embodiment, the amplification layer F2 is an upsampling layer that performs a bilinear upsampling operation on at least one third feature map.

[0026] The jump connection 310 is used to fuse at least one second feature map and at least one fourth feature map to generate at least one first feature map, so that the at least one first feature map has the same number of channels as the channels of the at least one second feature map. In one embodiment, the first jump connection is used to fuse at least one second feature map and at least one fourth feature map by performing an element-wise sum operation on the at least one second feature map and the at least one fourth feature map. Alternatively, an element-wise average operation is performed instead of an element-wise sum operation. The fused global information and local information in the at least one first feature map are obtained from the local information in the at least one second feature map and the global information in the at least one fourth feature map.

[0027] The decoder 304 is configured to output an output color image I2 generated from at least one first feature map, wherein the first part of the output color image I2 corresponding to the first part of the input color image I1 and the second part of the output color image I2 corresponding to the second part of the input color image I1 are color restored or not by determining whether a color consistency relationship exists between the first part of the input color image I1 and the second part of the input color image I1. In one embodiment, the output color image I2 is an RGB image. The decoder 304 includes a plurality of stages G to J. Each stage from G to J includes an upscaling layer X1, a convolution block X2 immediately after the upscaling layer X1, and a convolution block X3 immediately after the convolution block X2, wherein X is any one of the stages G to J. In one embodiment, the upscaling layer X1 is an upscaling layer that performs, for example, a linear upsampling operation, a bilinear upsampling operation, a bicubic upsampling operation, or a nearest upsampling operation. Alternatively, the upscaling layer X1 is a deconvolution layer. In one embodiment, each of the convolution blocks X2 and X3 includes a 3x3 convolution layer. Other kernel sizes of the convolutional layers of convolutional blocks X2 and X3 are within the expected range of the present disclosure. Each convolutional block X2 and X3 also includes a nonlinear activation function after the 3x3 convolutional layer. In one embodiment, the nonlinear activation function is a Leaky ReLU operation. The size increases step by step from stage G to stage J, so the size of the output color image I2 is larger than the size of at least one first feature map. The number of channels decreases step by step from stage G to stage J, so the number of channels of the output color image I2 is smaller than the number of channels of at least one first feature map.

[0028] In the above embodiment, the coding network 300 has a U-net architecture. Other architectures, such as coding networks having substantially the same size throughout the encoder and decoder, are within the contemplated scope of the present disclosure.

[0029] Figure 4 4 is a diagram schematically illustrating an input color image 402, an output color image 462, and intermediate details for color restoration performed by the compilation network 300 according to an embodiment of the present disclosure. Figure 3 and Figure 4 In the embodiment of the present invention, there is a chair with a flat backrest in the input color image 402. Due to low SNR, low contrast and / or narrow dynamic range, the chair in the input color image 402 is difficult to identify. The first part 404 of the input color image 402 is the edge of the chair backrest, and the second part 406 of the input color image 402 is the middle part of the chair backrest. The encoder 302 is used to output at least one first feature map 442, including fused global information and local information, so that whether there is a color consistency relationship between the first part 404 of the input color image 402 and the second part 406 of the input color image 402 is encoded into the fused global information and local information.

[0030] When the first portion 404 of the input color image 402 and the second portion 406 of the input color image 402 are being extracted by the encoder 302 for representation, the convolution block E2 directly or indirectly considers the first portion 404 of the input color image 402 in a view 422 having a first receptive field size, represented by a portion 424 drawn out of the view 422. The convolution block E2 directly or indirectly considers the second portion 406 of the input color image 402 in a view 426 having a first receptive field size, represented by a portion 428 drawn out of the view 426. For simplicity, in Figure 4 and Figure 5 In the convolution block E2, only at least one portion is processed, which corresponds to at least one of the first portion 404 of the input color image 402 and the second portion 406 of the input color image 402, which is displayed in the view, while other portions processed in the view are not displayed. When each of the first portion 404 of the input color image 402 and the second portion 406 of the input color image 402 is considered in the corresponding view 422 or 426, in at least one second feature map output by the convolution block E2, no semantics belonging to the first portion 404 of the input color image 402 and the second portion 406 of the input color image 402 are encoded as local information.

[0031] When the first portion 404 of the input color image 402 and the second portion 406 of the input color image 402 are extracted by the encoder 302 for representation, the global pooling layer F1 directly or indirectly considers the first portion 404 of the input color image 402 and the second portion 406 of the input color image 402 in the view 430 with the second receptive field size, represented by the portion 432 and the portion 434 drawn out of the view 430. When both the first portion 404 of the input color image 402 and the second portion 406 of the input color image 402 are considered in the view 430, a semantic meaning, or a "chair back" 444, attributed to the first portion 404 of the input color image 402 and the second portion 406 of the input color image 402 is encoded as global information in at least one third feature map output by the global pooling layer F1. The upscaling layer F2 outputs at least one fourth feature map including the global information.

[0032] The skip connection 310 fuses at least one second feature map and at least one fourth feature map to generate at least one first feature map 442 including fused global information and local information. The fused global information and local information are obtained from the local information and the global information. Because no semantics belonging to the first part 404 of the input color image 402 and the second part 406 of the input color image 402 are encoded into the fused global information and local information, and the semantics belonging to the first part 404 of the input color image 402 and the second part 406 of the input color image 402, i.e., "back of chair" 444, are encoded into the fused global information and local information, there is a color consistency relationship between the first part 404 of the input color image 402 and the second part 406 of the input color image 402. The existence of the color consistency relationship is demonstrated in the at least one first feature map 442, where a portion 446 corresponding to the first part 404 of the input color image 402 and a portion 448 corresponding to the second part 406 of the input color image 402 are circled together by the semantics, i.e., "back of chair", without being circled separately.

[0033] The decoder 304 is configured to output an output color image 462 generated from the at least one first feature map 442, wherein a first portion 464 of the output color image 462 corresponding to the first portion 404 of the input color image 402 and a second portion 466 of the output color image 462 corresponding to the second portion 406 of the input color image 402 are repaired by considering whether there is a color consistency relationship between the first portion 404 of the input color image 402 and the second portion 406 of the input color image 402. Figure 4 In the example of FIG. 4 , since there is a color consistency relationship between the first portion 404 of the input color image 402 and the second portion 406 of the input color image 402, the first portion 464 of the output color image 462 and the second portion 466 of the output color image 462 are restored to have the same color. The color consistency between the edge of the seat back and the middle portion of the seat back in the output color image 462 is represented by the edge of the seat back drawn with a solid line.

[0034] Figure 5 is a schematic diagram showing an input color image 502, an output color image 562, and intermediate details with color restoration performed by the compilation network 300 according to an embodiment of the present disclosure. Figure 3 and Figure 5, there is a chair with a backrest decorated with stars in the input color image 502. Due to low SNR, low contrast and / or narrow dynamic range, the chair in the input color image 502 is difficult to identify. The first part 504 of the input color image 502 is the edge of the chair back, and the second part 506 of the input color image 502 is the inside of the chair back with star decoration. The encoder 302 is used to output at least one first feature map 542 including fused global information and local information, so that whether there is a color consistency relationship between the first part 504 of the input color image 502 and the second part 506 of the input color image 502 is encoded as the fused global information and local information.

[0035] When the first portion 504 of the input color image 502 and the second portion 506 of the input color image 502 are extracted by the encoder 302 for representation, the convolution block E2 directly or indirectly considers the first portion 504 of the input color image 502 in a view 522 having a first receptive field size, represented by a cutout 524 in the view 522. The convolution block E2 directly or indirectly considers the second portion 506 of the input color image 502 in a view 526 having a first receptive field size, represented by a cutout 528 in the view 526. When processing the first portion 504 of the input color image 502 in the view 522, no semantics attributed to the first portion 504 of the input color image 502 are encoded as local information in at least one second feature map output by the convolution block E2. When considering the second portion 506 of the input color image 502 in the view 526, the semantics attributed to the second portion 506 of the input color image 502, namely, a "star" 544, are encoded into the local information in the at least one second feature map.

[0036] When the first portion 504 of the input color image 502 and the second portion 506 of the input color image 502 are being extracted by the encoder 302 for representation, the global pooling layer F1 considers the first portion 504 of the input color image 502 and the second portion 506 of the input color image 502 to be directly or indirectly in the view 530 having the second receptive field size and represented by the portion 532 and the other portion 534 drawn out of the view 530, respectively. When the first portion 504 of the input color image 502 and the second portion 506 of the input color image 502 are considered in the view 530, the semantics attributed to the first portion 504 of the input color image 502 and the second portion 506 of the input color image 502, i.e., "chair back" 548, is encoded as global information in at least one third feature map output by the global pooling layer F1. The upscaling layer F2 outputs at least one fourth feature map including the global information.

[0037] The skip connection 310 fuses at least one second feature map and at least one fourth feature map to generate at least one first feature map 542 including fused global information and local information. The fused global information and local information are obtained from the local information and the global information. Because the semantics attributed to the second part 506 of the input color image 502, i.e., "star" 544, is encoded as the fused global information and local information, and the semantics "chair back" 548 is also encoded as the fused global information and local information, there is no color consistency relationship between the first part 504 of the input color image 502 and the second part 506 of the input color image 502. In the at least one first feature map 542, the lack of color consistency relationship is manifested as: the part 546 corresponding to the second part 506 of the input color image 502 is circled separately by the aforementioned semantics, i.e., "star" 544; while the part 550 corresponding to the first part 504 of the input color image 502 and the part 552 corresponding to the second part 506 of the input color image 502 are circled together by the aforementioned semantics, i.e., "chair back" 548.

[0038] The decoder 304 is configured to output an output color image 562 generated from the at least one first feature map 442, wherein whether to repair the first part 564 of the output color image 562 corresponding to the first part 504 of the input color image 502 and the second part 566 of the output color image 562 corresponding to the second part 506 of the input color image 502 is determined based on whether there is a color consistency relationship between the first part 504 of the input color image 502 and the second part 566 of the output color image 562. Figure 5 In the example of FIG. 5 , because there is no color consistency relationship between the first portion 504 of the input color image 502 and the second portion 506 of the input color image 502, the first portion 564 of the output color image 562 and the second portion 566 of the output color image 562 are restored to have different colors. In the output color image 562, since fine details are locally presented, the color inconsistency between the edge of the seat back and the star-shaped portion of the seat back decoration is represented by the edge of the chair back drawn with a solid line and the star-shaped portion drawn with a dotted line.

[0039] In addition, refer to Figure 3In one embodiment, the feature maps output by the amplification layer G1 of the decoder 304 and the convolution block D2 of the encoder 302 have substantially the same size. Similarly, the feature maps of the amplification layer H1 and the convolution block C2, the feature maps of the amplification layer I1 and the convolution block B2, and the feature maps of the amplification layer J1 and the convolution block A2 have substantially the same corresponding sizes. The feature maps of the amplification layer G1 and the convolution block D2 are connected together and input to the convolution block G2. Similarly, the feature maps of the amplification layer H1 and the convolution block C2, the feature maps of the amplification layer I1 and the convolution block B2, and the feature maps of the amplification layer J1 and the convolution block A2 can be connected together accordingly and input to the corresponding convolution blocks H2, I2, and J2.

[0040] In addition, in one embodiment, during the training process, the input color image I1 of the compilation network 300 is a short-exposure color image captured under low-light or underwater conditions. The loss function between the output color image I2 of the compilation network 300 and the ground truth image as the corresponding long-exposure color image is calculated, where the loss function is and the weighted joint loss of the Multi-Scale Structural Similarity Index (MS-SSIM), defined by Equation (1):

[0041]

[0042] where λ is set to 0.16 based on experience. It is defined by equation (2) loss, represents the MS-SSIM loss obtained by equation (3). Equation (2) is as follows:

[0043]

[0044] in and I are the output color image and I2 base table real image respectively, and N is the total number of pixels of the input color image I1. Equation (3) is as follows:

[0045]

[0046] The MS-SSIM of the i-th pixel is defined by equations (4)-(6). Equations (4)-(6) are as follows:

[0047]

[0048]

[0049]

[0050] Where x and y represent two discrete non-negative signals that match each other (for example, two image blocks corresponding to the same real space position in two images); μx and μ y is the mean value, σ x and σ y standard deviation, M is the number of layers, α and β are weights to adjust the contribution of each quantity. The average value μ x and μ y and standard deviation σ x and σ y Through the Gaussian filter G g To calculate, where G g The mean is zero and the standard deviation is σ g .

[0051] Table 1 shows the Figures 3 to 5 The experimental results obtained in the described embodiments are given in the following sections. Figure 3 , by using the global pooling layer F1 and the skip connection 310, the fused global information and local information indicating whether the color consistency relationship between the first part of the input color image I1 and the second part of the first color image I2 exists or not is generated. The fused global information and local information can indicate Figure 4 There is a color consistency relationship in the situation described. Therefore, if Figure 1 As described above, the color inconsistency caused by the receptive field at the bottleneck of the compiled network SID-net being insufficient to cover the first part 104 of the input color image 102 and the second part 104 of the input color image 102 can be solved. Because the fused global information and local information can indicate such Figure 5 In the case described, there is no color consistency relationship, so a balance of color consistency can be achieved. Therefore, performance such as the peak signal-to-noise ratio (PSNR) of the output color image I2 can be improved. In addition, by using a jump connection 310 that fuses at least one second feature map and at least one fourth feature map, at least one first feature map is generated, so that at least one first feature map has the same number of channels as at least one second feature map, and the PSNR of the output color image I2 is improved without introducing additional learnable parameters to the compilation network 300. Table 1 shows the example. Figures 3 to 5 The described embodiments are used to denoise and enhance the images and are compared with the compiler network SID-net described in the prior art "Learning to see in the dark" by C. Chen, Q. Chen, J. Xu, and V. Koltun. Figures 3 to 5 The described embodiments and the compilation network SID-net can be found at Figure 2 Compared with the compiled network SID-net, Figures 3 to 5 The described embodiment can achieve a 3.5% increase in PSNR, where many of the learnable parameters are the same as the compiled network SID-net.

[0052] Table 1

[0053]

[0054] Those skilled in the art can understand that the various units, modules, layers, blocks, algorithms and steps of the system, or the computer-implemented methods described and disclosed in the embodiments of the present invention are implemented using hardware, firmware, software or a combination thereof. Whether the function is run in hardware, firmware or software depends on the application conditions and the design requirements of the technical solution. Those of ordinary skill in the art can use different methods to implement the function for each specific application, but such implementation should not exceed the scope of this disclosure.

[0055] It should be understood that the systems and computer-implemented methods disclosed in the embodiments of the present disclosure may be implemented in other ways. The above embodiments are exemplary only. The division of modules is based only on logical functions, and other divisions exist in implementation. Modules may or may not be physical modules. It is possible for multiple modules to be combined or integrated into one physical module. It is also possible to divide any module into multiple physical modules. It is also possible to omit or skip certain features. On the other hand, the mutual coupling, direct coupling or communication coupling shown or discussed is operated indirectly or communicatively through some ports, devices or modules in an electrical, mechanical or other form. Modules used as separate components for illustration may be physically separated or not. Modules are located in one place or distributed on multiple network modules. Some or all of the modules are used according to the purpose of the embodiment.

[0056] If the software function module is implemented as a product for use and sale, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution proposed in the present invention can be substantially or partially implemented in the form of a software product. Alternatively, a part of the technical solution that is beneficial to the prior art can be implemented in the form of a software product. The software product is stored in a computer-readable storage medium, including multiple commands for at least one processor of the system to run all or part of the steps disclosed in the embodiment of the present invention. The storage medium includes a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a floppy disk, or other medium capable of storing program instructions.

[0057] While the present disclosure has been described in connection with what is considered to be the most practical and preferred embodiments, it is to be understood that the disclosure is not limited to the disclosed embodiments but is intended to cover various arrangements embodied without departing from the scope of the broadest interpretation of the appended claims.

Claims

1. A method for improving image color quality, comprising: The encoder receives and processes a first color image, in, The first color image includes a first portion and a second portion of the first color image located at different positions of the first color image; The encoder is used to output at least one first feature map, the at least one first feature map includes fused global information and local information, and whether there is a color consistency relationship between the first part and the second part of the first color image is encoded into the fused global information and local information, wherein the encoder further includes: The first block; The second block; and First hop connection, in, The first block includes: A convolution block, configured to output at least one second feature map including the local information and including a first receptive field; The second block includes: a global pooling layer, configured to perform global pooling on the at least one second feature map, output at least one third feature map containing the global information, and including a second receptive field, wherein the second receptive field is wider than the first receptive field; and an amplification layer, configured to amplify the at least one third feature map and output at least one fourth feature map, wherein the at least one fourth feature map has the same size as the at least one second feature map and includes the global information; The first jump connection is used to fuse the at least one second feature map and the at least one fourth feature map to generate the at least one first feature map, and the at least one first feature map has the same number of channels as the at least one second feature map, wherein the fused global information and local information are obtained from the global information and the local information.

2. The method according to claim 1, wherein: The first block makes the size of the at least one first feature map smaller than the size of the first color image.

3. The method according to claim 1, wherein: When the convolutional block considers each of the first part and the second part of the first color image in a corresponding first field of view having the first receptive field size, semantics attributed to either of the first part and the second part of the first color image are not encoded into the local information; and When the global pooling layer simultaneously considers the first part and the second part of the first color image in the corresponding second field of view having the second receptive field size, the first semantics belonging to the first part and the second part of the first color image are encoded into the global information, wherein there is a color consistency relationship between the first part and the second part of the first color image.

4. The method according to claim 1, wherein: When the convolutional block considers one of the first part and the second part of the first color image in a first field of view having the first receptive field size, encoding a first semantic attributed to one of the first part and the second part of the first color image into the local information; and When the global pooling layer simultaneously considers the first part and the second part of the first color image in a corresponding second field of view having the second receptive field size, the second semantics attributed to the first part and the second part of the first color image are encoded into the global information, wherein the color consistency relationship does not exist between the first part and the second part of the first color image.

5. The method according to claim 1, wherein: The upscaling layer is an upsampling layer without learnable parameters.

6. The method according to claim 5, wherein: The upsampling layer is used to enlarge the at least one third feature map by performing a bilinear interpolation operation on the at least one third feature map.

7. The method according to claim 1, wherein: The first skip connection is used to fuse the at least one second feature map and the at least one fourth feature map by performing an element-wise sum operation on the at least one second feature map and the at least one fourth feature map.

8. The method according to claim 1, further comprising: The at least one first feature map is received and processed by a decoder, wherein the decoder is used to output a second color image generated from the at least one first feature map, wherein, in view of whether a color consistency relationship exists between the first part and the second part of the first color image, a first part of the second color image corresponding to the first part of the first color image and a second part of the second color image corresponding to the second part of the first color image are repaired, wherein the decoder includes multiple convolution blocks.

9. The method according to claim 8, wherein: The decoder makes the size of the second color image larger than the size of the at least one first feature map.

10. A system for improving image color quality, comprising: at least one memory for storing program instructions; At least one processor is used to execute the program instructions, so that the at least one processor performs the steps: The encoder receives and processes a first color image, in, The first color image includes a first portion and a second portion of the first color image located at different positions of the first color image; The encoder is used to output at least one first feature map, the at least one first feature map includes fused global information and local information, and whether there is a color consistency relationship between the first part and the second part of the first color image is encoded into the fused global information and local information, wherein the encoder further includes: The first block; The second block; and First hop connection, in, The first block includes: A convolution block, configured to output at least one second feature map including the local information and including a first receptive field; The second block includes: a global pooling layer, configured to perform global pooling on the at least one second feature map, output at least one third feature map containing the global information, and including a second receptive field, wherein the second receptive field is wider than the first receptive field; and an amplification layer, configured to amplify the at least one third feature map and output at least one fourth feature map, wherein the at least one fourth feature map has the same size as the at least one second feature map and includes the global information; The first jump connection is used to fuse the at least one second feature map and the at least one fourth feature map to generate the at least one first feature map, and the at least one first feature map has the same number of channels as the at least one second feature map, wherein the fused global information and local information are obtained from the global information and the local information.

11. The system according to claim 10, wherein: The first block makes the size of the at least one first feature map smaller than the size of the first color image.

12. The system according to claim 10, wherein: When the convolutional block considers each of the first part and the second part of the first color image in a corresponding first field of view having the first receptive field size, semantics attributed to either of the first part and the second part of the first color image are not encoded into the local information; and When the global pooling layer simultaneously considers the first part and the second part of the first color image in the corresponding second field of view having the second receptive field size, the first semantics belonging to the first part and the second part of the first color image are encoded into the global information, wherein there is a color consistency relationship between the first part and the second part of the first color image.

13. The system according to claim 10, wherein: When the convolutional block considers one of the first part and the second part of the first color image in a first field of view having the first receptive field size, encoding a first semantic attributed to one of the first part and the second part of the first color image into the local information; and When the global pooling layer simultaneously considers the first part and the second part of the first color image in the corresponding second field of view having the second receptive field size, the second semantics belonging to the first part and the second part of the first color image are encoded into the global information, wherein the color consistency relationship does not exist between the first part and the second part of the first color image.

14. The system according to claim 10, wherein: The upscaling layer is an upsampling layer without learnable parameters.

15. The system of claim 14, wherein: The upsampling layer is used to enlarge the at least one third feature map by performing a bilinear interpolation operation on the at least one third feature map.

16. The system of claim 10, wherein: The first skip connection is used to fuse the at least one second feature map and the at least one fourth feature map by performing an element-wise sum operation on the at least one second feature map and the at least one fourth feature map.

17. The system according to claim 10, the steps further comprising: The at least one first feature map is received and processed by a decoder, wherein the decoder is used to output a second color image generated from the at least one first feature map, wherein, in view of whether a color consistency relationship exists between the first part and the second part of the first color image, a first part of the second color image corresponding to the first part of the first color image and a second part of the second color image corresponding to the second part of the first color image are repaired, wherein the decoder includes multiple convolution blocks.

18. The system of claim 17, wherein: The decoder makes the size of the second color image larger than the size of the at least one first feature map.

19. A non-transitory computer readable medium storing program instructions, which, when executed by at least one processor, causes the at least one processor to perform the steps of: The encoder receives and processes a first color image, in, The first color image includes a first portion and a second portion of the first color image located at different positions of the first color image; The encoder is used to output at least one first feature map, the at least one first feature map includes fused global information and local information, and whether there is a color consistency relationship between the first part and the second part of the first color image is encoded into the fused global information and local information, wherein the encoder further includes: The first block; The second block; and First hop connection, in, The first block includes: A convolution block, configured to output at least one second feature map including the local information and including a first receptive field; The second block includes: a global pooling layer, configured to perform global pooling on the at least one second feature map, output at least one third feature map containing the global information, and including a second receptive field, wherein the second receptive field is wider than the first receptive field; and an amplification layer, configured to amplify the at least one third feature map and output at least one fourth feature map, wherein the at least one fourth feature map has the same size as the at least one second feature map and includes the global information; The first jump connection is used to fuse the at least one second feature map and the at least one fourth feature map to generate the at least one first feature map, and the at least one first feature map has the same number of channels as the at least one second feature map, wherein the fused global information and local information are obtained from the global information and the local information.