Image enhancement processing method based on deep learning

Abstract

The present invention proposes an image enhancement processing method based on deep learning, including: extracting multi-scale features of the input image; recalibrating and fusing the multi-scale features to obtain a comprehensive feature map; performing nonlinear transformation; using a decoder network to reconstruct the transformed features into a preliminary enhanced image; designing a generative adversarial network to generate a content-aware weight map, in which each pixel value in the weight map reflects the importance of the corresponding position in terms of content; identifying key areas and associated areas based on the content-aware weight map, and further enhancing the overall effect of the image through interaction and collaboration between regions. The present invention improves the pertinence and effectiveness of image enhancement by extracting multi-scale features and performing recalibration and fusion, and then guides the identification of different areas based on the content-aware weight map, achieving collaborative enhancement between regions, and improving the overall effect of the image, so that the enhanced image is more natural and realistic in terms of details, color, contrast, etc.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image enhancement processing method based on deep learning. Background Art

[0002] Image enhancement technology is an image processing method in the field of computer vision. Its main goal is to improve image quality by processing the input image, thereby enhancing the recognition and detection performance of the computer vision system. It can be applied to various computer vision tasks, such as image recognition, image classification, and object detection.

[0003] Traditional image enhancement processing technology usually has the following defects: 1. Lack of adaptability, making it difficult to perform targeted enhancement based on different image contents. The enhancement process often ignores the differences in the importance of content in different areas of the image, resulting in insufficiently refined enhancement effects; 2. It often ignores the correlation and interactivity between different areas of the image, resulting in localized and incoherent enhancement effects, making it difficult to achieve global optimization of the enhancement process, and thus difficult to improve the overall effect and beauty of the image. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to propose an image enhancement processing method based on deep learning to solve the above-mentioned problems.

[0005] According to the image enhancement processing method based on deep learning proposed by the present invention, the method includes:

[0006] Extract multi-scale features of the input image;

[0007] Recalibrate and fuse multi-scale features to obtain a comprehensive feature map;

[0008] Perform nonlinear transformation on the comprehensive feature map;

[0009] Use the decoder network to reconstruct the transformed features into a preliminary enhanced image;

[0010] Design a generative adversarial network that takes the preliminary enhanced image as input and generates a content-aware weight map, where each pixel value in the content-aware weight map reflects the importance of the corresponding position in the content;

[0011] According to the content-aware weight map, key areas and related areas are identified, and the overall effect of the image is further enhanced through interaction and collaboration between regions.

[0012] Furthermore, the steps of designing a generative adversarial network to generate a content-aware weight map using the preliminary enhanced image as input include:

[0013] Determine the structure of the generative adversarial network, including the generator and discriminator;

[0014] A training set is constructed based on a large amount of historical data. Each sample data in the training set contains a preliminary enhanced image and a corresponding real content-aware weight map to adaptively adjust the parameters of the generator to generate an image that better meets the discriminator's expectations. The content-aware weight map is marked with a label of content importance.

[0015] Train the generator based on the generator loss;

[0016] Train the discriminator based on the discriminator loss;

[0017] Through adversarial training between the generator and the discriminator, the parameters of the generator are adaptively adjusted to generate images that are more in line with the discriminator's expectations;

[0018] After the training is completed, the preliminary enhanced image is input into the trained generator, and a content-aware weight map with the same size as the preliminary enhanced image is generated.

[0019] Furthermore, the step of training the generator based on the generator loss includes:

[0020] Randomly sample preliminary enhanced images from the training set;

[0021] Use the generator to generate content-aware weight maps;

[0022] The generated content-aware weight map is input into the discriminator to calculate the generator loss, which is calculated as follows:

[0023]

[0024] in, is the generator loss, is the content-aware weight map generated by the generator, z is the input of the generator, and is random noise. For the discriminator pair The output of is the probability of the true content-aware weight map;

[0025] Use the back-propagation algorithm to update the generator's network parameters.

[0026] Furthermore, the step of training the generator based on the discriminator loss includes:

[0027] Initialize the network parameters of the generator and discriminator;

[0028] Randomly extract preliminary enhanced images and labeled true content-aware weight maps from the training set;

[0029] Use the generator to generate fake content-aware weight maps;

[0030] The real and fake content-aware weight maps are input into the discriminator and the discriminator loss is calculated as:

[0031] ,

[0032] in, is the discriminator loss, p is the true content-aware weight map, and are the outputs of the discriminator for p and G(z), respectively, indicating the probability that the discriminator believes that p and G(z) are true content-aware weight maps;

[0033] The network parameters of the discriminator are updated through the back-propagation algorithm.

[0034] Furthermore, the step of identifying key areas and associated areas based on the content-aware weight map and enhancing the overall effect of the image through interaction and collaboration between the areas includes:

[0035] Define key areas and associated areas based on weight values ​​in the content-aware weight map;

[0036] Binarize the content-aware weight map to obtain a binary image containing only key areas and associated areas;

[0037] Segment the binary image into different regions and assign unique region labels;

[0038] Identify key regions and related regions from the content-aware weight map based on region labels;

[0039] Calculate the similarity between two key regions with a connection relationship and the associated region;

[0040] If the similarity is higher than a first threshold, the visual connection between the two regions is strengthened;

[0041] If the similarity is lower than the second threshold, the visual interference between the two regions is reduced or eliminated.

[0042] Furthermore, the step of defining the key areas and associated areas according to the weight values ​​in the content-aware weight map includes:

[0043] The area with a weight value higher than the third threshold is defined as a key area;

[0044] The regions whose weight values ​​are lower than the third threshold and are connected to the key regions are defined as associated regions.

[0045] Furthermore, the step of recalibrating and fusing the multi-scale features to obtain a comprehensive feature map includes:

[0046] Input the multi-scale features into the deep clustering network, divide the scale features into corresponding feature clusters, and recalibrate the scale features according to the importance weight of the corresponding feature cluster;

[0047] The recalibrated features of each scale are fused to obtain a comprehensive feature map.

[0048] Furthermore, the multi-scale features are input into the deep clustering network, each scale feature is divided into a corresponding feature cluster, and each scale feature is recalibrated according to the importance weight of the corresponding feature cluster, including:

[0049] The extracted multi-scale features are input into the encoder of the deep clustering network to obtain the low-dimensional representation of the features at each scale;

[0050] In the low-dimensional representation space, clustering algorithms are used to divide the scale features into corresponding feature clusters;

[0051] Predict the importance weight of each feature cluster through the MLP model;

[0052] Recalibrate each scale feature according to the importance weight corresponding to the feature cluster divided by each scale feature to obtain the recalibrated scale features. The formula is:

[0053]

[0054] in, is the i-th scale feature after recalibration, is a scale feature Importance weights corresponding to the divided feature clusters;

[0055] The step of fusing the recalibrated scale features to obtain a comprehensive feature includes:

[0056] The recalibrated scale features are fused to obtain the fused feature X, which is formulated as follows:

[0057]

[0058] in, are the scale features after recalibration, and M is the number of scale features.

[0059] Furthermore, the importance weight of each feature cluster is predicted by the MLP model, including:

[0060] Get the representative feature Y of each feature cluster from the deep clustering network and construct the input matrix ,in, , n is the number of feature clusters, d is the dimension of the representative feature of each feature cluster;

[0061] Labeling each representative feature in the input matrix according to true label data related to the importance of the feature cluster;

[0062] Propagate the input features forward through the MLP model to obtain the output prediction value;

[0063] Use the loss function to calculate the error between the predicted value and the true label;

[0064] Calculate the gradient of the loss function with respect to the model parameters using the chain rule, and update the model parameters using the gradient descent method;

[0065] For new feature clusters, their representative features are extracted and input into the trained MLP model to output the importance weight prediction results of each new feature cluster through the model.

[0066] In summary, the image enhancement processing method based on deep learning of the present invention extracts multi-scale features so that the detail information in the image can be fully retained and enhanced, and recalibrates and fuses the multi-scale features to obtain a comprehensive feature map. The recalibration is to ensure that important features are enhanced and secondary features are properly suppressed, thereby improving the pertinence and effectiveness of image enhancement. Next, a nonlinear transformation is performed on the comprehensive feature map to enhance the contrast and color saturation of the image. The decoder network is then used to reconstruct the transformed features into a preliminary enhanced image to achieve automated and accurate mapping from feature space to image space. The decoder network can optimize the efficiency and quality of image reconstruction so that the preliminary enhanced image has better visual effects while maintaining high fidelity. In order to further enhance the overall effect of the image, a generative adversarial network is used to generate a content-aware weight map. The content-aware weight map can accurately reflect the importance of each pixel in the image in terms of content, provide guidance for further image enhancement, and make the image enhancement process more intelligent and adaptive. Then, according to the value of the content-aware weight map, the key areas and related areas are divided, and global optimization is achieved through interaction and collaborative enhancement between areas. The key areas are enhanced and the related areas are adjusted appropriately, so that the various parts of the image can be more harmonious and unified, thereby improving the overall effect and beauty of the image and obtaining a higher quality image.

[0067] Additional aspects and advantages of the present invention will be set forth in part in the following description and, in part, will be obvious from the following description, or may be learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0069] Figure 1This is a flowchart of an image enhancement processing method based on deep learning according to the first embodiment of the present invention;

[0070] Figure 2 This is a system block diagram of an image enhancement processing system based on deep learning according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0071] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0072] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0074] Example 1

[0075] See also Figure 1 The present invention proposes an image enhancement processing method based on deep learning, which includes steps S101 to S106:

[0076] S101, extracting multi-scale features of the input image.

[0077] It should be noted that multi-scale features of the input image are extracted to capture different details and structures in the image, including edges, textures, shapes, etc., providing rich feature information for subsequent image enhancement. Before feature extraction, the input image can be preprocessed, such as normalization and denoising.

[0078] Convolutional neural networks (CNNs) can be used to extract multi-scale features from input images. When building a CNN model, you can choose a CNN architecture such as ResNet or VGG and adjust the parameters. For example, to extract multi-scale features, you need to use convolution kernels of different sizes to capture information at different scales in the image. The preprocessed image is then input into the CNN model, and feature maps for each layer are calculated through forward propagation. When extracting multi-scale features, attention is paid to the feature maps output by different convolutional layers. Feature maps output by shallow convolutional layers typically contain more detailed information (such as edges and textures), while feature maps output by deep convolutional layers focus more on the overall structure and semantic information of the image. After feature extraction, multi-scale features are output. These features are subsequently recalibrated and fused for image enhancement.

[0079] S102, recalibrate and fuse the multi-scale features to obtain a comprehensive feature map.

[0080] It should be noted that before fusing multi-scale features, it is necessary to recalibrate the features of each scale to adjust the importance of features at different scales, highlight important features, suppress minor features, and fuse the recalibrated features at different scales to obtain a comprehensive feature map. This feature map integrates information at multiple scales and has stronger expressiveness and robustness.

[0081] S103, performing nonlinear transformation on the comprehensive feature map.

[0082] It should be noted that the fused comprehensive feature map is subjected to nonlinear transformation to enhance the expressiveness and flexibility of the features, making subsequent image reconstruction and enhancement more accurate and diverse.

[0083] This can be achieved through activation functions (such as ReLU, Sigmoid, etc.) to perform nonlinear mapping of features. In the image enhancement task of this embodiment, a simple, effective, and easily calculated gradient ReLU function can be used. The comprehensive feature map obtained in step S102 is used as the input of the nonlinear transformation function, and the nonlinear transformation function is applied to each element in the comprehensive feature map to obtain the transformed features. During the training process, the effect of the nonlinear transformation can be evaluated based on the change in the loss function. If the loss function value continues to decrease, it means that the nonlinear transformation helps improve the model performance. Otherwise, the nonlinear transformation function or parameters need to be adjusted.

[0084] S104, using a decoder network to reconstruct the transformed features into a preliminary enhanced image.

[0085] It should be noted that the decoder network is used to map the information in the feature space back to the image space to obtain a preliminarily enhanced image.

[0086] Construct a decoder network. Since the decoder network is used to map features back to the image space, a structure symmetrical to the encoder network (e.g., the CNN in step S101) can be chosen. The decoder network can consist of multiple upsampling and convolutional layers to gradually restore the spatial resolution and detail of the image. The decoder network parameters (such as weights and biases) are then initialized. The nonlinearly transformed features are then input into the decoder network, ensuring that the input feature format meets the decoder network's input requirements. Through the forward propagation of the decoder network, feature maps are calculated layer by layer to gradually restore the spatial resolution and detail of the image. After layer-by-layer calculations in the decoder network, a preliminary enhanced image is ultimately output. Ensure that the output image's shape and type meet the requirements. For example, the image pixel values ​​can be normalized to the range of 0 to 255 and converted to RGB format.

[0087] S105, design a generative adversarial network, take the preliminary enhanced image as input, and generate a content-aware weight map, wherein each pixel value in the content-aware weight map reflects the importance of the corresponding position in the content.

[0088] It should be noted that a generative adversarial network is designed to generate a content-aware weight map based on the content of the preliminary enhanced image, reflecting the importance of each pixel in terms of content.

[0089] A generative adversarial network (GAN) consists of a generator and a discriminator. The generator is responsible for generating content-aware weight maps, while the discriminator determines whether the generated weight maps are realistic. As GANs continuously optimize the generator and discriminator during training to produce more realistic images, their ability to understand image content improves. Compared to traditional image processing models such as convolutional neural networks and autoencoders, they demonstrate higher accuracy in content perception.

[0090] The content-aware weight map generated by GANs allows for more precise image enhancement. For example, in image super-resolution reconstruction, key areas of the image can be targeted for enhancement based on the weight map, thereby improving the quality of the reconstructed image. Traditional image enhancement methods, however, often rely on global statistical features or simple local features, making it difficult to accurately capture key information and details in the image.

[0091] The generated content-aware weight map can be applied to a variety of image processing tasks, such as image restoration, image style transfer, image segmentation, etc., and has broad application prospects in the field of image processing.

[0092] The recalibration and fusion in step S102 act on the feature level to optimize feature representation, which can improve the precise expression ability and robustness of features, thereby indirectly improving the image enhancement effect. Then, by generating a content-aware weight map, it acts on the image pixel level to guide further enhancement of the image, which can improve the overall quality and overall visual effect of the image, making the processed image clearer, more natural, and with better visual effects.

[0093] Further optionally, the step of designing a generative adversarial network to generate a content-aware weight map using the preliminary enhanced image as input includes:

[0094] Determine the structure of the generative adversarial network, including the generator and discriminator;

[0095] A training set is constructed based on a large amount of historical data. Each sample data in the training set contains a preliminary enhanced image and a corresponding real content-aware weight map to adaptively adjust the parameters of the generator to generate an image that better meets the discriminator's expectations. The content-aware weight map is marked with a label of content importance.

[0096] Train the generator based on the generator loss;

[0097] Train the discriminator based on the discriminator loss;

[0098] Through adversarial training between the generator and the discriminator, the parameters of the generator are adaptively adjusted to generate images that are more in line with the discriminator's expectations;

[0099] After the training is completed, the preliminary enhanced image is input into the trained generator, and a content-aware weight map with the same size as the preliminary enhanced image is generated.

[0100] It is understandable that GAN can learn the deep features of image content through adversarial training of the generator and the discriminator. This deep learning ability enables GAN to more accurately capture the key information and details in the image when generating content-aware weight maps, thereby generating more precise content-aware weight maps.

[0101] GAN's adversarial training mechanism enables it to adaptively adjust the generator's parameters to produce images that better meet the discriminator's expectations. This adaptability allows GAN to more flexibly adapt to different types of image content and styles when generating content-aware weight maps. By training on large amounts of data, GAN can learn universal patterns in image content, allowing it to better generalize to unseen image data when generating content-aware weight maps.

[0102] Further optionally, the step of training the generator based on the generator loss includes:

[0103] Randomly sample preliminary enhanced images from the training set;

[0104] Use the generator to generate content-aware weight maps;

[0105] The generated content-aware weight map is input into the discriminator to calculate the generator loss, which is calculated as follows:

[0106] ,

[0107] in, is the generator loss, is the content-aware weight map generated by the generator, z is the input of the generator, and is random noise. For the discriminator pair The output of is the probability of the true content-aware weight map;

[0108] Use the back-propagation algorithm to update the generator's network parameters.

[0109] Understandably, through the adversarial training process, the generator is able to generate increasingly realistic content-aware weight maps that are difficult for the discriminator to distinguish. The generator and the discriminator compete with each other and evolve together, and eventually the generator is able to generate high-quality forged images.

[0110] The formula of the generator loss indicates that the generator hopes that the discriminator will mistake the generated weight map G(z) as real, that is, D(G(z)) is as close to 1 as possible. Therefore, the generator loss is the negative logarithm of D(G(z)), and when D(G(z)) is close to 1, the loss is close to 0.

[0111] Further optionally, the step of training the generator based on the discriminator loss includes:

[0112] Initialize the network parameters of the generator and discriminator;

[0113] Randomly extract preliminary enhanced images and labeled true content-aware weight maps from the training set;

[0114] Use the generator to generate fake content-aware weight maps;

[0115] The real and fake content-aware weight maps are input into the discriminator and the discriminator loss is calculated as:

[0116] ,

[0117] in, is the discriminator loss, p is the true content-aware weight map, and are the outputs of the discriminator for p and G(z), respectively, indicating the probability that the discriminator believes that p and G(z) are true content-aware weight maps;

[0118] The network parameters of the discriminator are updated through the back-propagation algorithm.

[0119] It is understandable that by training the discriminator to distinguish between real and fake content-aware weight maps, and at the same time using this adversarial training process to indirectly train the generator, it can generate more realistic content-aware weight maps that are difficult for the discriminator to distinguish.

[0120] The formula of the discriminator loss indicates that the discriminator hopes to judge the probability D(p) of the true weight map p as true as close to 1 as possible, while judging the probability of the false weight map G(z) as false is also as close to 1 as possible. Therefore, the discriminator loss is D(p) and The sum of the negative logarithms of .

[0121] S106 , identifying key areas and associated areas based on the content-aware weight map, and further enhancing the overall effect of the image through interaction and collaboration between areas.

[0122] It should be noted that the content-aware weight map is used to identify key areas and related areas in the image, and the overall effect and beauty of the image are improved through interaction and collaborative enhancement between regions.

[0123] The image is divided into key regions and associated regions based on the values ​​in the weight map. Adaptive enhancement can be performed based on the difference between the two regions. For example, key regions can be enhanced by increasing brightness or color saturation, while associated regions can be adjusted appropriately to maintain overall image harmony and unity. Inter-regional interaction and collaboration can also be achieved through methods such as regional fusion, transition, or collaborative enhancement.

[0124] Further optionally, the step of identifying key areas and associated areas according to the content-aware weight map and enhancing the overall effect of the image through interaction and collaboration between the areas includes:

[0125] Define key areas and associated areas based on weight values ​​in the content-aware weight map;

[0126] Binarize the content-aware weight map to obtain a binary image containing only key areas and associated areas;

[0127] Segment the binary image into different regions and assign unique region labels;

[0128] Identify key regions and related regions from the content-aware weight map based on region labels;

[0129] Calculate the similarity between two key regions with a connection relationship and the associated region;

[0130] If the similarity is higher than a first threshold, the visual connection between the two regions is strengthened;

[0131] If the similarity is lower than the second threshold, the visual interference between the two regions is reduced or eliminated.

[0132] It can be understood that the content-aware weight map provides the importance or relevance information of each area in the image. By analyzing the weight values, key areas and associated areas can be identified. Key areas are usually important or prominent parts of the image, while associated areas are areas that are connected to the key areas and may affect them.

[0133] The weight map is converted into a binary image containing only key areas and associated areas through binarization. In the binary image, key areas and associated areas are clearly distinguished, facilitating subsequent segmentation and recognition. The image segmentation algorithm is then used to segment the binary image into different areas and assign unique area labels, providing a basis for subsequent area recognition and similarity calculation. Area labeling can be performed in the following ways:

[0134] Use a connected component labeling algorithm (such as the scanline algorithm) to label the binary image, where each connected component is assigned a unique region label. During or after labeling, traverse the image and record each region's adjacent regions. Specifically, an adjacency matrix can be used to represent the connectivity between regions. The adjacency matrix is ​​a two-dimensional array with rows and columns representing different regions, and each element in the matrix indicates whether two regions are adjacent. When calculating similarity, for each key region, traverse its adjacency matrix, find a corresponding region adjacent to it, and calculate the similarity between the two regions.

[0135] After region labeling, when calculating similarity, for each key region, its adjacency matrix is ​​traversed to find a related region adjacent to it. Feature descriptors (such as color histograms, texture features, and shape features) are then extracted and converted into feature vectors to quantify the similarity between regions. Based on the feature vectors, the similarity between two regions is calculated using similarity calculation methods (such as Euclidean distance and cosine similarity). When the similarity is above a first threshold, it means that the two regions are highly related in content. Therefore, strengthening the connection between them can enhance the overall coordination and unity of the image. If the similarity is below a second threshold, it means that the two regions are significantly different in content. Therefore, reducing or eliminating the interference between them can make the image more harmonious.

[0136] Key regions are the most important parts of an image and have a direct impact on the overall image quality. Key regions are emphasized by strengthening the visual connection between them and similar, related regions. If related regions are highly correlated with the key region, they complement or complement it. Adjusting the color, texture, and other features of the related regions can enhance their harmony with the key region, thereby enhancing the overall image quality. For related regions with low similarity to the key region, smoothing can be used to reduce or eliminate their visual interference.

[0137] This embodiment uses a content-aware weight map to identify and process key areas and related areas in an image. It enhances the overall effect of the image by strengthening the visual connection between similar areas and reducing the visual interference between dissimilar areas. This image enhancement processing method takes into account both the importance of the area and the correlation between the areas.

[0138] Further optionally, the step of defining the key areas and associated areas according to the weight values ​​in the content-aware weight map includes:

[0139] The area with a weight value higher than the third threshold is defined as a key area;

[0140] The regions whose weight values ​​are lower than the third threshold and are connected to the key regions are defined as associated regions.

[0141] Further optionally, the step of recalibrating and fusing the multi-scale features to obtain a comprehensive feature map includes:

[0142] Input the multi-scale features into the deep clustering network, divide the scale features into corresponding feature clusters, and recalibrate the scale features according to the importance weight of the corresponding feature cluster;

[0143] The recalibrated features of each scale are fused to obtain a comprehensive feature map.

[0144] As can be understood, the deep clustering network divides each scale feature into its corresponding feature cluster, and recalibrates each scale feature according to the importance weight of the corresponding feature cluster to emphasize important features and suppress unimportant features. Importance weights can be used to adjust the weights of each scale feature to improve feature representation before feature fusion.

[0145] This embodiment uses a deep clustering network to divide features at each scale into corresponding feature clusters. Each feature cluster represents a specific pattern, structure, or semantic information in the image. Features at each scale are recalibrated based on the importance weight of the corresponding feature cluster. The importance weight of the feature cluster can reflect the contribution or significance of the feature cluster in a specific task (such as classification, detection, segmentation, etc.). The recalibration process can involve scaling, weighting, or selecting features to enhance the influence of important features and weaken the influence of unimportant features. The recalibrated features at each scale are fused to obtain a comprehensive feature map that not only incorporates information from multi-scale features but also takes into account the importance of feature clusters (i.e., the contribution or significance of different feature clusters in the current task), making it more expressive and discriminative.

[0146] Further optionally, the step of inputting the multi-scale features into a deep clustering network, dividing each scale feature into a corresponding feature cluster, and recalibrating each scale feature according to the importance weight of the corresponding feature cluster includes:

[0147] The extracted multi-scale features are input into the encoder of the deep clustering network to obtain the low-dimensional representation of the features at each scale;

[0148] In the low-dimensional representation space, clustering algorithms are used to divide the scale features into corresponding feature clusters;

[0149] Predict the importance weight of each feature cluster through the MLP model;

[0150] Recalibrate each scale feature according to the importance weight corresponding to the feature cluster divided by each scale feature to obtain the recalibrated scale features. The formula is:

[0151] ,

[0152] in, is the i-th scale feature after recalibration, is a scale feature The importance weights corresponding to the divided feature clusters.

[0153] Further optionally, the step of fusing the recalibrated scale features to obtain a comprehensive feature includes:

[0154] The recalibrated scale features are fused to obtain the fused feature X, which is formulated as follows:

[0155] ,

[0156] in, , ,… , are the scale features after recalibration, and M is the number of scale features.

[0157] Understandable and comprehensible, by assigning importance weights through a deep clustering network and recalibrating multi-scale features, it is possible to fully utilize the information of multi-scale features and consider the importance of feature clusters, thereby obtaining a more expressive and discriminative comprehensive feature map.

[0158] The goal of the clustering algorithm is to make the features within the same feature cluster as similar as possible and the features between different feature clusters as different as possible. K-means, spectral clustering, etc. can be used.

[0159] Further optionally, predicting the importance weight of each feature cluster by the MLP model includes:

[0160] Get the representative feature Y of each feature cluster from the deep clustering network and construct the input matrix ,in, , n is the number of feature clusters, d is the dimension of the representative feature of each feature cluster, and the representative feature can be the centroid, mean, or feature representation after some transformation of the feature cluster;

[0161] Labeling each representative feature in the input matrix according to true label data related to the importance of the feature cluster;

[0162] Propagate the input features forward through the MLP model to obtain the output prediction value;

[0163] Use the loss function to calculate the error between the predicted value and the true label;

[0164] Calculate the gradient of the loss function with respect to the model parameters using the chain rule, and update the model parameters using the gradient descent method;

[0165] For new feature clusters, their representative features are extracted and input into the trained MLP model to output the importance weight prediction results of each new feature cluster through the model.

[0166] As can be understood, using an MLP (Multi-Layer Perceptron) model to predict the importance weights of each feature cluster makes the determination of importance weights automated, efficient, and accurate. Furthermore, through training on large amounts of data, the MLP model can learn underlying patterns and laws within the data. These patterns and laws are not only applicable to the training data but can also be generalized to new feature clusters, thereby enabling accurate prediction of the importance weights of feature clusters in new data.

[0167] In summary, the image enhancement processing method based on deep learning of the present invention extracts multi-scale features so that the detail information in the image can be fully retained and enhanced, and recalibrates and fuses the multi-scale features to obtain a comprehensive feature map. The recalibration is to ensure that important features are enhanced and secondary features are properly suppressed, thereby improving the pertinence and effectiveness of image enhancement. Next, a nonlinear transformation is performed on the comprehensive feature map to enhance the contrast and color saturation of the image. The decoder network is then used to reconstruct the transformed features into a preliminary enhanced image to achieve automated and accurate mapping from feature space to image space. The decoder network can optimize the efficiency and quality of image reconstruction so that the preliminary enhanced image has better visual effects while maintaining high fidelity. In order to further enhance the overall effect of the image, a generative adversarial network is used to generate a content-aware weight map. The content-aware weight map can accurately reflect the importance of each pixel in the image in terms of content, provide guidance for further image enhancement, and make the image enhancement process more intelligent and adaptive. Then, according to the value of the content-aware weight map, the key areas and related areas are divided, and global optimization is achieved through interaction and collaborative enhancement between areas. The key areas are enhanced and the related areas are adjusted appropriately, so that the various parts of the image can be more harmonious and unified, thereby improving the overall effect and beauty of the image and obtaining a higher quality image.

[0168] Example 2

[0169] See also Figure 2 The present invention proposes an image enhancement processing system based on deep learning, which includes:

[0170] Feature extraction module: used to extract multi-scale features of the input image;

[0171] Feature fusion module: used to recalibrate and fuse multi-scale features to obtain a comprehensive feature map;

[0172] Feature transformation module: used to perform nonlinear transformation on the comprehensive feature map;

[0173] Preliminary enhancement module: used to reconstruct the transformed features into a preliminary enhanced image using the decoder network;

[0174] Weight map module: This module is used to design a generative adversarial network that takes the initial enhanced image as input and generates a content-aware weight map, where each pixel value in the content-aware weight map reflects the importance of the corresponding position in the content.

[0175] Collaborative enhancement module: used to identify key areas and related areas based on the content-aware weight map, and further enhance the overall effect of the image through interaction and collaboration between regions.

[0176] Further optionally, the weight map module is further used to:

[0177] Determine the structure of the generative adversarial network, including the generator and discriminator;

[0178] A training set is constructed based on a large amount of historical data. Each sample data in the training set contains a preliminary enhanced image and a corresponding real content-aware weight map to adaptively adjust the parameters of the generator to generate an image that better meets the discriminator's expectations. The content-aware weight map is marked with a label of content importance.

[0179] Train the generator based on the generator loss;

[0180] Train the discriminator based on the discriminator loss;

[0181] Through adversarial training between the generator and the discriminator, the parameters of the generator are adaptively adjusted to generate images that are more in line with the discriminator's expectations;

[0182] After the training is completed, the preliminary enhanced image is input into the trained generator, and a content-aware weight map with the same size as the preliminary enhanced image is generated.

[0183] Further optionally, the weight map module is further used to:

[0184] Randomly sample preliminary enhanced images from the training set;

[0185] Use the generator to generate content-aware weight maps;

[0186] The generated content-aware weight map is input into the discriminator to calculate the generator loss, which is calculated as follows:

[0187] ,

[0188] in, is the generator loss, is the content-aware weight map generated by the generator, z is the input of the generator, and is random noise. For the discriminator pair The output of is the probability of the true content-aware weight map;

[0189] Use the back-propagation algorithm to update the generator's network parameters.

[0190] Further optionally, the weight map module is further used to:

[0191] Initialize the network parameters of the generator and discriminator;

[0192] Randomly extract preliminary enhanced images and labeled true content-aware weight maps from the training set;

[0193] Use the generator to generate fake content-aware weight maps;

[0194] The real and fake content-aware weight maps are input into the discriminator and the discriminator loss is calculated as:

[0195] ,

[0196] in, is the discriminator loss, p is the true content-aware weight map, and are the outputs of the discriminator for p and G(z), respectively, indicating the probability that the discriminator believes that p and G(z) are true content-aware weight maps;

[0197] The network parameters of the discriminator are updated through the back-propagation algorithm.

[0198] Further optionally, the collaborative enhancement module is further configured to:

[0199] Define key areas and associated areas based on weight values ​​in the content-aware weight map;

[0200] Binarize the content-aware weight map to obtain a binary image containing only key areas and associated areas;

[0201] Segment the binary image into different regions and assign unique region labels;

[0202] Identify key regions and related regions from the content-aware weight map based on region labels;

[0203] Calculate the similarity between two key regions with a connection relationship and the associated region;

[0204] If the similarity is higher than a first threshold, the visual connection between the two regions is strengthened;

[0205] If the similarity is lower than the second threshold, the visual interference between the two regions is reduced or eliminated.

[0206] Further optionally, the collaborative enhancement module is further configured to:

[0207] The area with a weight value higher than the third threshold is defined as a key area;

[0208] The regions whose weight values ​​are lower than the third threshold and are connected to the key regions are defined as associated regions.

[0209] Further optionally, the feature fusion module is further configured to:

[0210] Input the multi-scale features into the deep clustering network, divide the scale features into corresponding feature clusters, and recalibrate the scale features according to the importance weight of the corresponding feature cluster;

[0211] The recalibrated features of each scale are fused to obtain a comprehensive feature map.

[0212] Further optionally, the feature fusion module is further configured to:

[0213] The extracted multi-scale features are input into the encoder of the deep clustering network to obtain the low-dimensional representation of the features at each scale;

[0214] In the low-dimensional representation space, clustering algorithms are used to divide the scale features into corresponding feature clusters;

[0215] Predict the importance weight of each feature cluster through the MLP model;

[0216] Recalibrate each scale feature according to the importance weight corresponding to the feature cluster divided by each scale feature to obtain the recalibrated scale features. The formula is:

[0217] ,

[0218] in, is the i-th scale feature after recalibration, is a scale feature Importance weights corresponding to the divided feature clusters;

[0219] Further optionally, the feature fusion module is further configured to:

[0220] The recalibrated scale features are fused to obtain the fused feature X, which is formulated as follows:

[0221] ,

[0222] in, , ,… , are the scale features after recalibration, and M is the number of scale features.

[0223] Further optionally, the feature fusion module is further configured to:

[0224] Get the representative feature Y of each feature cluster from the deep clustering network and construct the input matrix ,in, , n is the number of feature clusters, d is the dimension of the representative feature of each feature cluster;

[0225] Labeling each representative feature in the input matrix according to true label data related to the importance of the feature cluster;

[0226] Propagate the input features forward through the MLP model to obtain the output prediction value;

[0227] Use the loss function to calculate the error between the predicted value and the true label;

[0228] Calculate the gradient of the loss function with respect to the model parameters using the chain rule, and update the model parameters using the gradient descent method;

[0229] For new feature clusters, their representative features are extracted and input into the trained MLP model to output the importance weight prediction results of each new feature cluster through the model.

[0230] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. An image enhancement processing method based on deep learning, characterized in that: The method comprises: Extract multi-scale features of the input image; Recalibrate and fuse multi-scale features to obtain a comprehensive feature map; Perform nonlinear transformation on the comprehensive feature map; Use the decoder network to reconstruct the transformed features into a preliminary enhanced image; Design a generative adversarial network that takes the preliminary enhanced image as input and generates a content-aware weight map, where each pixel value in the content-aware weight map reflects the importance of the corresponding position in the content; Based on the content-aware weight map, key areas and related areas are identified, and the overall effect of the image is further enhanced through interaction and collaboration between areas; The step of recalibrating and fusing the multi-scale features to obtain a comprehensive feature map includes: Input the multi-scale features into the deep clustering network, divide the scale features into corresponding feature clusters, and recalibrate the scale features according to the importance weight of the corresponding feature cluster; The recalibrated features of each scale are fused to obtain a comprehensive feature map; The steps of identifying key areas and associated areas based on the content-aware weight map and enhancing the overall effect of the image through interaction and collaboration between the areas include: Define key areas and associated areas based on weight values ​​in the content-aware weight map; Binarize the content-aware weight map to obtain a binary image containing only key areas and associated areas; Segment the binary image into different regions and assign unique region labels; Identify key regions and related regions from the content-aware weight map based on region labels; Calculate the similarity between two key regions with a connection relationship and the associated region; If the similarity is higher than a first threshold, the visual connection between the two regions is strengthened; If the similarity is lower than the second threshold, the visual interference between the two regions is reduced or eliminated.

2. The image enhancement processing method based on deep learning according to claim 1, characterized in that: The steps of designing a generative adversarial network to generate a content-aware weight map using a preliminary enhanced image as input include: Determine the structure of the generative adversarial network, including the generator and discriminator; A training set is constructed based on a large amount of historical data. Each sample data in the training set contains a preliminary enhanced image and a corresponding real content-aware weight map to adaptively adjust the parameters of the generator to generate an image that better meets the discriminator's expectations. The content-aware weight map is marked with a label of content importance. Train the generator based on the generator loss; Train the discriminator based on the discriminator loss; Through adversarial training between the generator and the discriminator, the parameters of the generator are adaptively adjusted to generate images that are more in line with the discriminator's expectations; After the training is completed, the preliminary enhanced image is input into the trained generator, and a content-aware weight map with the same size as the preliminary enhanced image is generated.

3. The image enhancement processing method based on deep learning according to claim 2, characterized in that: The step of training the generator based on the generator loss includes: Randomly sample preliminary enhanced images from the training set; Use the generator to generate content-aware weight maps; The generated content-aware weight map is input into the discriminator to calculate the generator loss, which is calculated as follows: ， in, is the generator loss, is the content-aware weight map generated by the generator, z is the input of the generator, and is random noise. For the discriminator pair The output of is the probability of the true content-aware weight map; Use the back-propagation algorithm to update the generator's network parameters.

4. The image enhancement processing method based on deep learning according to claim 2, characterized in that: The step of training the generator based on the discriminator loss includes: Initialize the network parameters of the generator and discriminator; Randomly extract preliminary enhanced images and labeled true content-aware weight maps from the training set; Use the generator to generate fake content-aware weight maps; The real and fake content-aware weight maps are input into the discriminator and the discriminator loss is calculated as: ， in, is the discriminator loss, p is the true content-aware weight map, and are the outputs of the discriminator for p and G(z), respectively, indicating the probability that the discriminator believes that p and G(z) are true content-aware weight maps; The network parameters of the discriminator are updated through the back-propagation algorithm.

5. The image enhancement processing method based on deep learning according to claim 1, characterized in that: The step of defining the key areas and associated areas according to the weight values ​​in the content-aware weight map comprises: The area with a weight value higher than the third threshold is defined as a key area; The regions whose weight values ​​are lower than the third threshold and are connected to the key regions are defined as associated regions.

6. The image enhancement processing method based on deep learning according to claim 1, characterized in that: The multi-scale features are input into the deep clustering network, each scale feature is divided into a corresponding feature cluster, and each scale feature is recalibrated according to the importance weight of the corresponding feature cluster, including: The extracted multi-scale features are input into the encoder of the deep clustering network to obtain the low-dimensional representation of the features at each scale; In the low-dimensional representation space, clustering algorithms are used to divide the scale features into corresponding feature clusters; Predict the importance weight of each feature cluster through the MLP model; Recalibrate each scale feature according to the importance weight corresponding to the feature cluster divided by each scale feature to obtain the recalibrated scale features. The formula is: ， in, is the i-th scale feature after recalibration, is a scale feature Importance weights corresponding to the divided feature clusters; The step of fusing the recalibrated scale features to obtain a comprehensive feature includes: The recalibrated scale features are fused to obtain the fused feature X, which is formulated as follows: ， in, , ,… , are the scale features after recalibration, and M is the number of scale features.

7. The image enhancement processing method based on deep learning according to claim 6, characterized in that: The MLP model is used to predict the importance weight of each feature cluster, including: Get the representative feature Y of each feature cluster from the deep clustering network and construct the input matrix ,in, , n is the number of feature clusters, d is the dimension of the representative feature of each feature cluster; Labeling each representative feature in the input matrix according to true label data related to the importance of the feature cluster; Propagate the input features forward through the MLP model to obtain the output prediction value; Use the loss function to calculate the error between the predicted value and the true label; Calculate the gradient of the loss function with respect to the model parameters using the chain rule, and update the model parameters using the gradient descent method; For new feature clusters, their representative features are extracted and input into the trained MLP model to output the importance weight prediction results of each new feature cluster through the model.

Images