Semantic Knowledge-Guided Low-Light Image Enhancement Method

Abstract

This invention discloses a low-light image enhancement method based on semantic knowledge guidance, belonging to the field of low-light image enhancement technology. By introducing semantic information, this invention addresses issues neglected by previous methods. Furthermore, it can be applied to image enhancement networks with arbitrary encoder-decoder structures, enabling models that previously lacked semantic information to learn more. Specifically, through a semantically guided embedding module, combined with semantically guided color histogram loss and semantically guided adversarial loss, this invention can focus on semantically relevant knowledge from multiple perspectives. This invention improves the capabilities of low-light image enhancement networks, resulting in more realistic and natural enhancement results.

Description

Technical Field

[0001] This invention belongs to the field of low-light image enhancement technology, specifically relating to a low-light image enhancement method based on semantic knowledge guidance. Background Technology

[0002] Due to unavoidable environmental and / or technical limitations, such as insufficient lighting and limited exposure time, images are often captured under suboptimal lighting conditions, affected by backlighting, non-uniform illumination, and low light. The aesthetic quality of such images is compromised, and information transmission is unsatisfactory for high-level tasks such as object tracking, recognition, and detection. Low-light (image brightness below or equal to a specified value) enhancement has wide applications in various fields, including visual surveillance, autonomous driving, and computational photography. In particular, smartphone photography has become ubiquitous and popular. Taking photos in dimly lit environments is especially challenging due to limitations in smartphone camera aperture, real-time processing requirements, and memory. Enhancing low-light images and videos in such applications is a worthwhile research area to explore. Traditional low-light image enhancement methods include histogram equalization and Retinex model-based methods; however, these traditional models do not adapt well to varying environments, generally have long runtimes, and it is difficult to obtain optimal parameters. In recent years, combined with advancements in deep learning technology, deep learning-based low-light image enhancement has achieved remarkable success.

[0003] Currently, deep learning-based low-light image enhancement methods are mainly divided into two categories: end-to-end methods and Retinex-based methods. Inspired by the classic LLNet, researchers have proposed various end-to-end methods, including end-to-end parameter filter estimation networks, recurrent neural networks, multiple exposure fusion networks, deep stacked Laplacian enhancement networks, and wavelet transform-based enhancement networks. Compared to enhancement effects directly learned in end-to-end networks, Retinex theory, due to its physical interpretability, generally leads to better results for deep low-light image enhancement methods. The first Retinex-based method, called Retinex-Net, decomposes low-light images into illumination and reflection components, enhances the illumination component, and then fuses them into a normal-light image. Later, researchers proposed KinD based on Retinex-Net, adding enhancement and denoising operations for the reflection component, thus improving the enhancement effect. In addition, there are KinD++, enhancement networks based on Retinex and neural architecture search, deep unfolding enhancement networks based on Retinex, and enhancement networks based on regularized flow. It's worth noting that these methods tend to enhance low-light images without considering the semantic information of different regions. When there are objects that are originally black in a low-light image, such as a person's black hair or a black vehicle, these enhancement methods often enhance these parts to gray, leading to color distortion. To address this issue, the enhancement network needs to learn semantically relevant information. Some researchers have proposed preliminary solutions, including fusing the prediction results of semantic segmentation networks into Retinex-based networks and using the loss function of semantic segmentation networks to constrain the parameter updates of image enhancement networks. Both methods achieve the combination of semantic and image information through carefully designed networks and training methods. However, they do not fully utilize the information provided by semantic segmentation networks and do not consider the differences between semantic information and the original image enhancement task. For the former, the difference between semantic segmentation results and intermediate features in image enhancement is relatively large, inevitably damaging the original image information during fusion. For the latter, directly constraining the two different tasks through loss can affect the original optimization process of the image enhancement network parameters, thus affecting the final enhancement result. In summary, current solutions fail to effectively incorporate semantic information into image enhancement tasks. Furthermore, the carefully designed interaction between semantic and image information not only has limited generalization capabilities but also results in abnormal colors and details in the generated normal light images, affecting the visual quality of the images and the effectiveness of subsequent image processing tasks. Summary of the Invention

[0004] This invention provides a low-light image enhancement method based on semantic knowledge, which can be used to improve the image enhancement effect of low-light images.

[0005] The technical solution adopted in this invention is as follows:

[0006] A low-light image enhancement method guided by semantic knowledge, the method comprising:

[0007] Step 1: Construct an image enhancement processing network model;

[0008] The image enhancement processing network model includes two branches: one branch is a semantic segmentation network and the other branch is an image enhancement network. N (N≥2) semantic embedding modules are set between the two branches.

[0009] The semantic segmentation network comprises, in sequence, a first encoder, a first decoder, and a prediction head. The first encoder is used to process the input image I. l Feature extraction is performed on the low-light image to obtain the input image I. l The first initial feature map;

[0010] The first decoder is used to decode the first initial feature map at multiple scales to obtain deep feature maps at different scales, i.e., semantic segmentation features F. i Furthermore, the number of scales M for semantic segmentation features is greater than N;

[0011] Semantic segmentation features F at different scales of the prediction head compared to the output of the first decoder i Perform pixel-level semantic category prediction and output the input image I. l Semantic prediction graph I seg (Semantic segmentation result);

[0012] From the deep feature maps of M scales, select semantic segmentation features of N scales of continuous scale as one of the two inputs of each semantic embedding module, and define the N semantic embedding modules as the 1st to the Nth semantic embedding modules in ascending order of scale.

[0013] The image enhancement network includes a second encoder and a second decoder, wherein the second encoder is used to process the input image I. l Feature extraction is performed on the low-light image to obtain the input image I. l The second initial feature map; the decoder consists of N+1 convolutional blocks, each used to upsample its input and output image enhancement features F at different scales. i The output of the last convolutional block is the input image I. l Predictive augmented image Furthermore, the input of the first convolutional block is the second initial feature map, and the output of the first convolutional block is used as another input of the first semantic embedding module. For any i-th (i = 1, ..., N) semantic embedding module, the output of the i-th semantic embedding module is used as the input of the (i+1)-th convolutional block.

[0014] Step 2: Train the network parameters of the image enhancement processing network model based on the training samples. Stop when the preset training termination condition is met, and obtain the trained image enhancement processing network model.

[0015] The loss function for training the image augmentation network model is set as follows:

[0016]

[0017] in, Indicates the predicted enhanced image With input image I l Label Image I h The reconstruction losses between This represents semantically guided color histogram loss, i.e., predicting enhanced images. Histogram and label image I h L1 norm loss between histograms, λ SCH Represents semantically guided color histogram loss. The weight, This represents semantically guided adversarial loss, λ. SA Semantic guidance adversarial loss The weights;

[0018] Step 3: Input the image to be enhanced, which matches the input of the image enhancement processing network model, into the trained image enhancement network model, and obtain the enhancement result of the image to be enhanced based on the output of the last convolutional block of the image enhancement network.

[0019] Furthermore, in step 2, semantically guided adversarial loss. For global combat losses Localized combat losses The summation is achieved by introducing a discriminator during training to obtain the global adversarial loss. Localized combat losses Specifically:

[0020] Semantic prediction graph I based on the output of a semantic segmentation network seg For predictive augmented images The image is divided into blocks, with each block corresponding to a semantic category, and P is defined. k Represents any k-th image patch;

[0021] Calculate local adversarial loss

[0022]

[0023] x f =P t ,D(P t )=min{D(P k )}

[0024] Where G represents the generator, i.e., the image enhancement processing network model, D represents the generator, D() represents the generator's output, and x r p represents a real image patch real The data distribution representing the real image patch, x f p represents a fake image block fake Data distribution representing fake image patches, Represents the mathematical expectation of a real image patch;

[0025] The input to the prediction head of the semantic segmentation network is denoted as feature map I. s ′ eg In the channel dimension, the predicted augmented image In feature map I s ′ eg The images are stitched together to form a new spoof image patch x′. f And calculate the overall combat losses.

[0026]

[0027] in, This represents the mathematical expectation of the new fake image patch.

[0028] Furthermore, the semantic segmentation network is a pre-trained network that remains unchanged during the learning and training of the network parameters of the image enhancement processing network model; that is, only the network parameters of the image enhancement network and the N semantic embedding modules are learned and updated.

[0029] Furthermore, the network structure of the semantic embedding module is as follows:

[0030] Input semantic segmentation features F s and image enhancement features F i After passing through a normalization layer and a convolutional layer, semantic feature maps and image enhancement feature maps with consistent dimensions are obtained. The semantic feature maps and image enhancement feature maps are flattened in the channel dimension, and then the attention map between the two flattened feature maps is calculated through the transpose attention mechanism to obtain the semantic related attention map A.

[0031] Image enhancement features F are adjusted using semantically relevant attention map A. i The output features F of the semantic embedding module are obtained. o :

[0032] F o =FN(W v (F i )×A+F i ),

[0033] Among them, W v The value represents the weights embedded in the convolutional layer, and FN() represents the output of the feedforward neural network.

[0034] Furthermore, the semantically guided color histogram loss is calculated. When the histogram is estimated, a differentiable method is used, specifically:

[0035] Semantic prediction graph I based on the output of a semantic segmentation network seg For the predicted enhanced image, respectively and label image I h The image is divided into blocks, with each block corresponding to a semantic category.

[0036] Estimate the predicted augmented image separately and label image I h Semantic-guided color histogram:

[0037] For each color channel of each image block, category edge pixel adjustment is performed to obtain the high and low anchor point values ​​of each pixel grayscale value of each pixel in each image block;

[0038] For the same pixel grayscale value of the same color channel within the same semantic category, the high and low anchor values ​​are multiplied by a preset scaling factor and used as input to the Sigmoid activation function. The cumulative difference between the high and low anchor values ​​of all pixels under the scaling factor is used to obtain an estimate of the number of pixels for the current pixel grayscale value of the current semantic category. Based on this estimate, an estimated histogram of the current color channel for the current semantic category is obtained. Finally, the predicted enhanced image is obtained from the estimated histograms of all color channels. Or label image I h The semantic-guided color histogram.

[0039] The technical solution provided by this invention brings at least the following beneficial effects:

[0040] By introducing semantic information, the semantic knowledge-guided low-light image enhancement method of this invention can address issues neglected by previous methods. Furthermore, this invention can be applied to image enhancement networks with any encoder-decoder structure, enabling these models, which originally lacked semantic information, to learn more knowledge. Specifically, through a semantically guided embedding module, combined with semantically guided color histogram loss and semantically guided adversarial loss, this invention can focus on semantically relevant knowledge from multiple different perspectives.

[0041] The semantic-guided embedding module of this invention processes at the feature level, corresponding the multi-scale features (semantic segmentation features) extracted by the semantic segmentation network to the multi-scale features in the original low-light image enhancement network decoder. Through similarity calculation, the deep coding information of semantic segmentation is introduced into the image features, and transformation is performed in the feature representation space to optimize the output. The semantic-guided color histogram loss is processed at the output level. After obtaining the semantic segmentation prediction result, the final enhanced image output by the image enhancement network is segmented by category, and color histograms are estimated for different image patches and compared with the histogram of the real label image to achieve more accurate color constraints. This allows the network to learn color information related to semantic categories, ensuring color consistency in the enhancement results. The semantic-guided adversarial loss is also processed at the output level, again utilizing the semantic segmentation prediction result and combining it with global and local adversarial losses. In the local adversarial loss, the most "fake" image patch is found by comparing the image patches through the discriminator's output, causing the generator (i.e., the image enhancement network) to focus on the "fake" parts. In addition, in the global adversarial loss, the segmentation result and the enhancement result are concatenated and input into the discriminator, so that the discriminator gives a global discrimination result with reference to semantic information. Together with the aforementioned local adversarial loss, the discriminator and the generator are constrained, which improves the ability of the low-light image enhancement network and obtains more realistic and natural enhancement results. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A schematic diagram illustrating the processing steps of the low-light image enhancement method based on semantic knowledge provided in this embodiment of the invention;

[0044] Figure 2 This is a schematic diagram of the network structure of the semantic guidance and semantic embedding module used in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0046] This invention addresses the problem of color deviation and abnormal details in enhanced images due to a lack of semantic information guidance. The objective of this invention is to identify which semantic information can be utilized for image enhancement and how this semantic information positively impacts low-light image enhancement tasks. For semantic segmentation networks, there are many multi-scale features (semantic segmentation features) output from intermediate layers. These features have different receptive fields and representational capabilities, allowing for optimization of the intermediate layer features in the representation space. Furthermore, the prediction results of the semantic segmentation network can also serve as prior information to guide the image enhancement network in learning semantically relevant mapping relationships.

[0047] Because there are certain differences between the intermediate layer features of semantic segmentation networks and the intermediate layer features of image enhancement networks, directly using concatenation or multiplication operations can lead to feature degradation, thus affecting the image enhancement effect. Therefore, this invention proposes a semantic-guided embedding module, which establishes cross-modal interaction information to reasonably embed semantic features into image enhancement features. Secondly, for color optimization, this invention preserves the color consistency of the output image through color histogram constraints. However, as a global statistical feature, the histogram cannot guarantee local consistency, thus limiting its ability to preserve color. Therefore, this invention proposes a semantic-guided color histogram loss, which uses the semantic segmentation results to segment each region, calculates the histogram and loss separately, and constrains the output color characteristics at the semantic level. Finally, current loss functions do not well represent the visual effect of the image and cannot capture the internal structure of the image, resulting in visually unsatisfactory results. To further improve the quality of the output image, some researchers have improved image quality through global and local adversarial training. However, random selection of local image patches cannot fully utilize the capabilities of local adversarial loss. Therefore, the method of this invention proposes semantically guided adversarial loss, which obtains image patches corresponding to different categories through semantic segmentation results, identifies the most false image patch as a local image patch for parameter update, thereby improving the ability of local loss and improving the quality of the final output image.

[0048] As one possible implementation, the specific implementation process of the low-light image enhancement method based on semantic knowledge guidance provided in this embodiment of the invention includes:

[0049] First, input a low-light image (I l The image is fed into the image enhancement network and the semantic segmentation network. After multiple layers of feature interaction, the output image enhancement is obtained. and semantic segmentation results (Iseg And, guided by the semantic segmentation results, implement color histogram loss and adversarial loss to constrain the training of the image augmentation network, such as... Figure 1 As shown.

[0050] In this method, the semantically guided low-light image enhancement problem can be described by the following formula:

[0051] M = F segment (I l ;θ s ),

[0052] Among them, F segment Let M be a pre-trained semantic segmentation network, and I be the semantic prior information obtained from the semantic segmentation network. l Given the input low-light image, θ s These are the parameters of the semantic segmentation network. Because this semantic segmentation network is pre-trained on a large-scale dataset, it can provide rich and varied semantic prior information, which is referred to as the semantic knowledge base in this embodiment. After obtaining the semantic prior information, it is input together with the low-light image into the image enhancement network:

[0053]

[0054] Among them, F enhance For low-light image enhancement networks, θ e For the parameters of the image enhancement network, The output is the normal light image, i.e., the predicted enhanced image. In this embodiment, only the parameters of the image enhancement network are updated during training, while the semantic segmentation network remains fixed, as shown below:

[0055]

[0056] Among them, I h To correspond with I l The normal light image (i.e., the label image) is used as an update for the label-constrained image augmentation network.

[0057] To address the impact of the differences between semantic segmentation and image enhancement on feature fusion, this invention establishes a reasonable interaction between the semantic segmentation network and the image enhancement network through a constructed semantic-guided embedding module. In this embodiment, HRNet (High-Resolution Net) is selected as the semantic knowledge base to provide semantic prior information. In HRNet, multi-scale intermediate layer features, output features, and prediction results are used as semantic information in the image enhancement task. For better illustration, the number of semantic-guided embedding modules in this embodiment is set to three, such as... Figure 1 As shown.

[0058] Figure 2 A schematic diagram of the network structure of each semantically guided semantic embedding module is given. The input of this module is semantic segmentation features and image enhancement features. After convolution, layer normalization, and flattening operations, they are transformed into feature maps of the same dimension. Then, an attention map is calculated, and the information contained in the semantic segmentation features is integrated into the image enhancement features. Finally, the optimized features are output, realizing the corresponding feature interaction operation. That is, in each semantically guided semantic embedding module of this invention, its input is the corresponding semantic segmentation features and image enhancement features. After entering the module, the features are first preprocessed through convolutional layers and layer normalization to transform the dimensions of the two features to be consistent, and their corresponding dimensions are represented as H×W×C. Next, the features are flattened in the channel dimension to obtain two HW×C feature maps. Then, based on the transposed attention mechanism, the attention map between the two feature maps is calculated and computational resources are saved. The final semantically related attention map A is shown below:

[0059]

[0060] Among them, W k and W q For key embedding and query embedding convolutional layers, F i and F s For image enhancement features and semantic segmentation features, C represents the number of channels, and Softmax is the activation function. The resulting semantically relevant attention map... It indicates F i and F s The intrinsic correlation between them, and then use A to adjust F i As shown below:

[0061] F o =FN(W v (F i )×A+F i ),

[0062] Among them, W v For value embedding, a convolutional layer is used; FN is a feedforward neural network, F o The optimized features are the output features of the semantically guided semantic embedding module. Therefore, this invention optimizes image enhancement features through semantic segmentation features, enabling them to focus on semantically relevant information in the representation subspace.

[0063] That is, W v (F i )×A is then convolved with image enhancement features F i The sum is then passed through a normalization layer and a convolutional layer, followed by matrix multiplication and another convolutional layer. Finally, it is added back to the sum to obtain the output features of the semantically guided semantic embedding module.

[0064] Color histograms contain important image statistical information and are well-suited for preserving color consistency in images. To optimize color, affinity matrices can be used to combine learned color histograms with image content. However, histograms describe global statistics, which differ significantly from content; direct fusion can negatively impact the restoration of detail and texture. Furthermore, color features for each category are ignored when calculating the global histogram, limiting color optimization capabilities. Therefore, this invention proposes a semantically guided color histogram loss method to achieve local color adjustment, thereby improving the color preservation capabilities of the image enhancement framework.

[0065] First, in this embodiment of the invention, the image is segmented into image patches using semantic segmentation results, and each image patch contains content of only one category. Therefore, the image patch generation process is as follows:

[0066] P = {P} 0 ,P 1 ,…,P class},

[0067] Where ⊙ represents matrix dot product, I out This represents the enhanced output (i.e., the predicted enhanced image). ), P represents the prediction result of the c-th category output by the semantic segmentation network. c Let P represent the image patch of category c, and let P represent the group of image patches. Thus, we obtain the image patches for each category.

[0068] Since color histograms are discrete statistical features, this embodiment of the invention estimates the histogram in a differentiable manner, thus enabling its use in model training. Considering the error in semantic segmentation results, pixels at class edges are ignored during calculation to reduce the impact of segmentation errors on training. Each image patch in group P is adjusted based on class edge pixels, resulting in an adjusted image patch group P′. To facilitate explanation of the histogram estimation process, the adjusted image patch P is described using the R channel of the c-th class. c′ (R) will be used as an example for explanation:

[0069]

[0070] Where, x j P represents c′ The j-th pixel of (R), i∈[0,255] represents the pixel grayscale value. and These represent the high anchor value and the low anchor value, respectively, which are used as features of the current pixel in subsequent calculations, as shown below:

[0071]

[0072]

[0073] Among them, H c P represents c′ Differentiable histogram estimation of (R), This represents the estimated number of pixels with grayscale value i, where α is the scaling factor, which is set to 400 in this embodiment. The two anchor values ​​mentioned above are scaled and then output using the Sigmoid activation function; the difference between the outputs is taken as pixel x. j Contribution to the pixel count estimate, x j The closer x is to i, the larger the difference becomes. j When the value of is exactly equal to i, the difference is 1, meaning it contributes one pixel. Finally, the l1 loss is used as the final constraint on the estimated color histogram. Therefore, the semantically guided color histogram loss is as follows:

[0074]

[0075] in, and I h H represents the output image and the ground truth label image, respectively. c (·) indicates the histogram estimation process.

[0076] In image completion tasks, global and local discriminators are used to obtain more realistic completion results. In low-light image enhancement tasks, this invention introduces semantic information to guide the discriminator to focus on regions of interest. To achieve this, embodiments of this invention optimize the global and local adversarial losses, incorporating semantic prediction graphs I... seg A semantically guided adversarial loss is proposed by incorporating image patch group P′ into the calculation of the loss function.

[0077] For the local adversarial loss, the aforementioned image patch group P′ is first used as candidate fake image patches. These candidate fake image patches are then input into the discriminator to obtain a discrimination result (the probability of whether it is a labeled image). The image patch with the smallest output result is considered the most fake part, and the gradient obtained from this output is used to update the parameters of the discriminator and generator. In this way, the discriminator reasonably uses semantic prior information to find the fake target region. For real image patches, a random cropping method is still used to obtain them from the dataset. Therefore, the local adversarial loss can be described as:

[0078]

[0079] x f =P t ,D(P t )=min(D(P0 ),…,D(P class ))

[0080] Where MSE(·) represents the mean square error, P t x represents a candidate fake image patch. r Represents a real image patch, x f This indicates a fake image block.

[0081] For global adversarial loss, this embodiment of the invention employs a simple design to implement semantically relevant guidance. I is concatenated at the channel dimension. out (Output of the N+1th convolutional block) and I s ′ eg As the new x f , among which, I s ′ eg This refers to the output features before the final Softmax activation function of the semantic segmentation network, i.e., the output features of the prediction head of the semantic segmentation network. Real images are still sampled randomly; therefore, the final global adversarial loss can be described as:

[0082]

[0083] That is, the semantic guidance adversarial loss in the embodiments of the present invention can be described as:

[0084]

[0085] Furthermore, this embodiment of the invention also defines the original loss function of the augmented network as... (Predictive Enhanced Image) With input image I l Label Image I h The reconstruction loss function (reconstruction loss) of augmented networks can generally be the first-order difference loss, mean squared error loss, or structural similarity loss, etc.

[0086] In summary, in this embodiment of the invention, the loss function for semantic knowledge-guided low-light image enhancement can be described as:

[0087]

[0088] Where, λ sCH and λ SA To balance the weights of each loss function, empirical values ​​are used.

[0089] Based on total loss The image enhancement processing network model of the present invention is trained, and training is stopped when the loss value tends to converge and remain stable, thus obtaining the trained image enhancement processing network model, and the enhancement processing result of the image to be enhanced (low-light image) is obtained based on its output.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0091] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A low-light image enhancement method guided by semantic knowledge, characterized in that, Includes the following steps: Step 1: Construct an image enhancement processing network model; The image enhancement processing network model includes two branches: one branch is a semantic segmentation network and the other branch is an image enhancement network. N semantic embedding modules are set between the two branches, where N≥2. The semantic segmentation network comprises, in sequence, a first encoder, a first decoder, and a prediction head. The first encoder is used to process the input image I. l Feature extraction is performed to obtain the input image I. l The first initial feature map, the input image I l Low-light image; The first decoder is used to decode the first initial feature map at multiple scales to obtain deep feature maps at different scales, i.e., semantic segmentation features F. i Furthermore, the number of scales M for semantic segmentation features is greater than N; Semantic segmentation features F at different scales of the prediction head compared to the output of the first decoder i Perform pixel-level semantic category prediction and output the input image I. l Semantic prediction graph I seg ; From the deep feature maps of M scales, select semantic segmentation features of N scales of continuous scale as one of the two inputs of each semantic embedding module, and define the N semantic embedding modules as the 1st to the Nth semantic embedding modules in ascending order of scale. The image enhancement network includes a second encoder and a second decoder, wherein the second encoder is used to process the input image I. l Feature extraction is performed on the low-light image to obtain the input image I. l The second initial feature map; the decoder consists of N+1 convolutional blocks, each used to upsample its input and output image enhancement features F at different scales. i The output of the last convolutional block is the input image I. l Predictive augmented image Furthermore, the input of the first convolutional block is the second initial feature map, and the output of the first convolutional block is used as another input of the first semantic embedding module. For any i-th semantic embedding module, the output of the i-th semantic embedding module is used as the input of the (i+1)-th convolutional block, where i = 1, ..., N; Step 2: Train the network parameters of the image enhancement processing network model based on the training samples. Stop when the preset training termination condition is met, and obtain the trained image enhancement processing network model. The loss function for training the image augmentation network model is set as follows: in, Indicates the predicted augmented image With input image I l Label Image I h The reconstruction losses between them This represents semantically guided color histogram loss, i.e., predicting enhanced images. Histogram and label image I h L1 norm loss between histograms, λ SCH Represents semantically guided color histogram loss. The weight, This represents semantically guided adversarial loss, λ. SA Semantic guidance adversarial loss The weights; Step 3: Input the image to be enhanced, which matches the input of the image enhancement processing network model, into the trained image enhancement network model, and obtain the enhancement result of the image to be enhanced based on the output of the last convolutional block of the image enhancement network.

2. The method as described in claim 1, characterized in that, In step 2, semantically guided adversarial loss For global combat losses Localized combat losses The summation is achieved by introducing a discriminator during training to obtain the global adversarial loss. Localized combat losses Specifically: Semantic prediction graph I based on the output of a semantic segmentation network seg For predictive augmented images The image is divided into blocks, with each block corresponding to a semantic category, and P is defined. k Represents any k-th image patch; Calculate local adversarial loss x f =P t ,D(P t )=min{D(P k )} Where G represents the generator, i.e., the image enhancement processing network model, D represents the generator, D(·) represents the generator output, and x r p represents a real image patch real The data distribution representing the real image patch, x f p represents a fake image block fake Data distribution representing fake image patches, Represents the mathematical expectation of a real image patch; Let the input of the prediction head of the semantic segmentation network be denoted as feature map I′. seg In the channel dimension, the predicted augmented image In feature map I′ seg The images are stitched together to form a new spoof image patch x′. f And calculate the overall combat losses. in, This represents the mathematical expectation of the new fake image patch.

3. The method as described in claim 1, characterized in that, The semantic segmentation network is a pre-trained network that remains unchanged during the learning and training of the network parameters of the image enhancement processing network model.

4. The method as described in claim 1, characterized in that, The network structure of the semantic embedding module is as follows: Input semantic segmentation features F s and image enhancement features F i After passing through a normalization layer and a convolutional layer respectively, we obtain semantic feature maps and image enhancement feature maps with consistent dimensions. The semantic feature map and the image enhancement feature map are flattened in the channel dimension, and then the attention map between the two flattened feature maps is calculated through the transpose attention mechanism to obtain the semantic related attention map A. Image enhancement features F are adjusted using semantically relevant attention map A. i The output features F of the semantic embedding module are obtained. o : F o =FN(W v (F i )×A+F i ), Among them, W v The value represents the weights embedded in the convolutional layer, and FN() represents the output of the feedforward neural network.

5. The method as described in claim 1, characterized in that, Calculate semantically guided color histogram loss When the histogram is estimated, a differentiable method is used, specifically: Semantic prediction graph I based on the output of a semantic segmentation network seg For the predicted enhanced image, respectively and label image I h The image is divided into blocks, with each block corresponding to a semantic category. Estimate the predicted augmented image separately and label image I h Semantic-guided color histogram: For each color channel of each image block, category edge pixel adjustment is performed to obtain the high and low anchor point values ​​of each pixel grayscale value of each pixel in each image block; For the same pixel grayscale value of the same color channel within the same semantic category, the high and low anchor values ​​are multiplied by a preset scaling factor and used as input to the Sigmoid activation function. The cumulative difference between the high and low anchor values ​​of all pixels under the scaling factor is used to obtain an estimate of the number of pixels for the current pixel grayscale value of the current semantic category. Based on this estimate, an estimated histogram of the current color channel for the current semantic category is obtained. Finally, the predicted enhanced image is obtained from the estimated histograms of all color channels. Or label image I h The semantic-guided color histogram.

Images