Scene-adaptive and self-distillation based low-light scene image enhancement method and system

Abstract

The application discloses a low-light scene image enhancement method and system based on scene self-adaptation and self-distillation, and the method comprises the following steps: acquiring a low-light scene image to be enhanced; inputting the low-light scene image to be enhanced into a trained scene classification model to output a scene classification result; and inputting the low-light scene image to be enhanced into a luminance enhancement network corresponding to the scene category to output an enhanced image. In an unknown low-light environment, the method can quickly classify the environment, perform luminance enhancement according to the classification result, and finally achieve the effect of image luminance enhancement in a low-light environment. The method uses a residual network and a self-distillation system, can accelerate the inference speed of the model, improve the enhancement effect of the system, reduce the size of the system, enable the system to be better deployed on an embedded device, and realize real-time low-light environment enhancement.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, and in particular to a method and system for enhancing low-light scene images based on scene adaptation and self-distillation. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] In recent years, with the rapid development of computer vision technology and the deepening of machine learning research, people have increasingly higher requirements for images. Due to environmental and equipment limitations, people often acquire low-light images, which not only affect their visual experience but also significantly reduce the reliability of computer vision systems. Therefore, low-light image enhancement is crucial for improving image visual quality and the performance of computer vision systems under extreme lighting conditions.

[0004] Many photographs are typically taken in low-light conditions due to environmental factors and hardware limitations. Key factors include: 1. Insufficient light: Low-light images are usually taken in dimly lit or poorly lit environments. This can be due to factors such as nighttime, insufficient indoor lighting, or cloudy days. 2. High dynamic range (HDR) scenes: Some scenes exhibit significant differences in brightness, known as HDR scenes. Examples include bright lights and details in shadows during night photography, or backlighting and foreground in backlit situations. 3. Shutter speed limitations: In low-light conditions, photographers may need to use longer shutter speeds to capture sufficient light and avoid blur. However, long exposures can easily lead to image blur or motion blur in video. 4. Camera sensor limitations: The sensitivity and noise control capabilities of the camera sensor also affect the quality of low-light images. Some camera sensors may produce more noise in low-light conditions, affecting image sharpness and detail.

[0005] The main impact of low-light images is that under low-light conditions, images have lower brightness, poorer contrast, and lost details. This makes details in the image difficult to discern, affecting the observer's ability to understand and identify the scene or object. Low-light images often contain more noise and grain, which affects the image's sharpness and smoothness, making the image appear rough and unclear. In high dynamic range scenes, images under low-light conditions may fail to capture details and brightness differences within the scene. This results in insufficient representation of details in both bright and dark areas, and an inability to realistically reproduce the dynamic range of the scene.

[0006] Currently, low-light image enhancement technology still faces several challenges and problems: 1. In the process of low-light image enhancement, increasing image brightness may also increase noise and distortion. Maintaining good noise control and detail preservation is a challenge; 2. The effect of low-light image enhancement often involves subjective evaluation of image quality and visual perception. Different people have different preferences for image quality and visual experience, so balancing enhancement effect and user experience is a problem; 3. In some scenarios, such as autonomous driving and real-time monitoring, low-light image enhancement requires real-time processing. How to meet real-time requirements while maintaining high-quality enhancement is a challenge; 4. Low-light image enhancement needs to cope with different lighting conditions, including different brightness levels, color temperatures, and light source types. How to adapt to different lighting conditions and perform effective enhancement is a difficult problem. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a low-light scene image enhancement method and system based on scene adaptation and self-distillation. In an unknown low-light environment, this method can quickly classify the environment and enhance brightness based on the classification results, ultimately achieving image brightness enhancement in low-light environments. This method utilizes residual networks and a self-distillation system, which can accelerate model inference, improve system enhancement performance, and reduce system size, making it easier to deploy on embedded devices and achieve real-time low-light environment enhancement.

[0008] On the one hand, it provides a low-light scene image enhancement method based on scene adaptation and self-distillation;

[0009] Low-light scene image enhancement methods based on scene adaptation and self-distillation include:

[0010] Acquire low-light scene images to be enhanced;

[0011] The low-light scene image to be enhanced is input into the trained scene classification model, and the scene classification result is output.

[0012] The low-light scene image to be enhanced is input into the brightness enhancement network corresponding to the scene category, and the enhanced image is output.

[0013] On the other hand, a low-light scene image enhancement system based on scene adaptation and self-distillation is provided;

[0014] A low-light scene image enhancement system based on scene adaptation and self-distillation includes:

[0015] The acquisition module is configured to acquire images of low-light scenes to be enhanced.

[0016] The scene classification module is configured to: input the low-light scene image to be enhanced into the trained scene classification model and output the scene classification result;

[0017] The image enhancement module is configured to input the low-light scene image to be enhanced into a brightness enhancement network corresponding to the scene category, and output the enhanced image.

[0018] Furthermore, an electronic device is also provided, including:

[0019] Memory, used for non-transitory storage of computer-readable instructions; and

[0020] Processor, for executing the computer-readable instructions,

[0021] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0022] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0023] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0024] One of the above technical solutions has the following advantages or beneficial effects:

[0025] In terms of speed, the use of the lightweight ResNet-18 and the brightness enhancement network enables fast inference speed, which can meet the requirements of engineering implementation and embedded deployment.

[0026] In terms of application scenarios, this invention first uses the ResNet-18 residual network to perform the first step of scene classification on the input image, so that the network trained on a training set with more similar scenes can perform inference, thereby improving the brightness enhancement effect in different scenes.

[0027] This invention employs a self-distillation method tailored to different scenarios for this brightness enhancement curve, resulting in better training results and thus improved performance. Attached Figure Description

[0028] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0029] Figure 1 This is a flowchart of the method in Example 1;

[0030] Figure 2 This is a diagram of the image brightness enhancement network structure in Example 1;

[0031] Figure 3 This is a diagram of the activation functions in the hidden layer of Example 1;

[0032] Figure 4 This is a structural diagram of the self-distillation system of Example 1;

[0033] Figure 5 This is a schematic diagram of the scaling criteria in Example 1;

[0034] Figure 6 This is a schematic diagram of the weight calculation results for Example 1;

[0035] Figure 7 This is a flowchart illustrating the overall process of the present invention as described in Embodiment 1. Detailed Implementation

[0036] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0037] Terminology Explanation:

[0038] 1. Jump connection:

[0039] Skip connections, also known as residual connections, are a technique that introduces short-circuit connections into deep neural networks. By introducing direct connections into the network, skipp connections allow input information to be directly passed to deeper layers, thus skipping some non-linear transformations. Specifically, skip connections directly add the input and output, allowing the network to learn the transformations of the input data while simultaneously learning the residual. This makes it easier for gradients to flow through the network during backpropagation, mitigating the vanishing gradient problem. Skip connections help the network better capture the connections between low-level features and high-level semantics, thereby improving the model's performance and generalization ability.

[0040] 2. Shallow feature reuse:

[0041] Shallow feature reuse is a technique in deep neural networks that utilizes features from earlier layers for auxiliary training or prediction. By using auxiliary classifiers or feature reuse, it can fully leverage the specificity and locality of lower-level features to improve model performance and generalization ability, and enhance the understanding and processing of complex data.

[0042] 3. Batch standardization:

[0043] Batch normalization is a technique used in deep neural networks to accelerate training and improve model performance. It normalizes the input data to a standard normal distribution with a mean of 0 and a variance of 1 by normalizing the intermediate layers of the neural network, thus addressing the internal covariate shift problem. Internal covariate shift refers to the change in the distribution of inputs to each layer during training due to parameter updates. This shift in the input distribution of each layer increases the difficulty of training the network. The existence of internal covariate shift leads to the network requiring a smaller learning rate and longer training time. Batch normalization normalizes the inputs of each neural network layer, ensuring that the input data in each batch has a similar distribution.

[0044] 4. ONNX Models: ONNX (Open Neural Network Exchange) is an open deep learning model exchange format designed to enable seamless sharing and use of models across different deep learning frameworks. ONNX models are a format for describing deep learning models, containing the model's structure and parameters. These models can be trained using various deep learning frameworks and then exported in ONNX format. After export, ONNX models can be loaded and executed in other ONNX-supporting deep learning frameworks without requiring retraining or significant modifications. In summary, ONNX models are a neutral deep learning model representation format that can be converted and shared between different deep learning frameworks. It provides cross-platform and cross-framework model deployment capabilities and supports model optimization and inference acceleration.

[0045] Example 1

[0046] This embodiment provides a low-light scene image enhancement method based on scene adaptation and self-distillation;

[0047] Low-light scene image enhancement methods based on scene adaptation and self-distillation include:

[0048] S101: Acquire the low-light scene image to be enhanced;

[0049] S102: Input the low-light scene image to be enhanced into the trained scene classification model and output the scene classification result;

[0050] S103: Input the low-light scene image to be enhanced into the brightness enhancement network of the corresponding scene category, and output the enhanced image.

[0051] Furthermore, the training process of the trained scene classification model includes:

[0052] Construct a first training set, which includes: low-light scene images with known scene classification labels; the scene classification labels include: no light outdoors at night, dim light outdoors at night, no light indoors, and weak light indoors.

[0053] The first training set is input into the scene classification model to train the model and obtain the trained scene classification model.

[0054] Understandably, in order to better improve image enhancement and increase inference speed, this method uses a ResNet18 residual network to classify the input images into the following scenarios: (1) outdoor nighttime with no light, (2) outdoor nighttime with dim light, (3) indoor nighttime with no light, and (4) indoor nighttime with dim light. Images of these four scenarios are collected and batch-labeled to create a training set.

[0055] To improve the training speed and performance of the model, reduce the risk of overfitting, and achieve better learning and convergence, this method preprocesses the dataset as follows: all obtained datasets are standardized and normalized so that the features can be mapped to the required range through linear scaling, and their size is set to conform to the shape of the operation.

[0056] First, take photos of the following four scenarios: 1. No light outdoors at night; 2. Low light outdoors at night; 3. No light indoors; 4. Low light indoors. It is important to note that when collecting outdoor photos, the outdoor scenes should be roughly similar, such as streets or buildings, and the brightness difference between the no light and low light images should be distinct.

[0057] Next, the captured images need to be preprocessed. Without preprocessing, problems such as unstable training, small feature value range, unbalanced gradient updates, gradient explosion, gradient vanishing, model non-convergence, and performance degradation will occur. Therefore, preprocessing is required. The detailed steps are as follows: (1) Use the OpenCV library to read the images. (2) Use the resize function to set the image size to 3*224*224 for input into the ResNet-18 residual network. (3) Normalize the images to scale the pixel values ​​to [0,1], which helps improve the training stability and convergence speed of the model. (4) Label the images of the four scenes accordingly.

[0058] Furthermore, the trained scene classification model is implemented using ResNet-18.

[0059] It should be understood that this method uses a ResNet-18 residual network to classify the input low-light images. ResNet-18 is a relatively shallow model with fewer layers, resulting in fewer parameters and lower computational complexity. By introducing skip connections, ResNet-18 allows information to be transmitted more directly within the network, facilitating efficient gradient propagation and accelerating model training. Multi-scale feature extraction is achieved by stacking multiple residual blocks, allowing ResNet-18 to gradually extract features at different scales, giving the network a better receptive field and abstraction capabilities. ResNet-18 is used instead of ResNet-50 in this scenario because the data is relatively simple, ResNet-18 is sufficient for the task, and its shallower depth and fewer parameters make it less prone to overfitting and faster to train. Furthermore, this application requires a faster and smaller model for industrial deployment, hence the choice of ResNet-18.

[0060] ResNet-18 consists of 18 layers, including convolutional layers, batch normalization layers, activation functions, and pooling layers. Its core component is the residual block, each containing two convolutional layers for feature extraction. Additionally, ResNet-18 includes a global average pooling layer and a fully connected layer for final classification. The residual block bypasses the operations of some layers by adding a skip connection to the network. The skip connection directly adds the input signal to the output, forming a residual—the difference between the network's output and input. This allows the network to learn variations in the residual rather than completely replacing the feature representation of the previous layer. A residual block typically consists of two convolutional layers, with a batch normalization layer and activation function added between them for feature extraction. The dimensionality of the skip connection can be adjusted by adding an identity mapping or using additional convolutional layers. Through the stacking of residual blocks, deep networks can be trained more easily and achieve better performance.

[0061] ResNet-18 is a classic deep convolutional neural network architecture. It is widely used in visual tasks, especially in image classification, object detection, and semantic segmentation. The main feature of ResNet-18 is the introduction of residual connections, which solve the gradient vanishing and gradient explosion problems in deep neural networks through skip connections. This residual connection makes it easier for the network to learn identity mappings, thus allowing for deeper network layers. The residual block consists of two convolutional layers and one skip connection, divided into six parts, in the following order: (1) Input: The input of the residual block is the output of the previous layer (or the output of the previous residual block). (2) Convolutional layer: The first convolutional layer of the residual block is responsible for feature extraction, using a 3x3 convolutional kernel to learn features. Convolutional operations can change the depth and spatial dimension of the feature map. (3) Batch normalization: After the convolutional layer, batch normalization is usually applied to normalize the output of the convolutional layer, which helps to accelerate training and improve the robustness of the model. (4) ReLU: The ReLU activation function is applied to the normalized feature map to increase the non-linear expressive power of the network. (5) Second Convolutional Layer: After the ReLU activation function, the second convolutional layer of the residual block further extracts features using a 3x3 convolutional kernel. (6) Skip Connection (Identity Mapping): Skip connections are a key component of the residual block. They enable cross-layer information transfer by directly adding the input to the output of the convolutional layer. If the dimensions of the input and output match, the skip connection uses an identity mapping, that is, directly adding the input to the output. If the dimensions of the input and output do not match, the skip connection uses an additional 1x1 convolutional layer to adjust the dimensions to ensure that they can be added together.

[0062] The input layer sets the shape of the input image to 3*224*224, where 3 represents the three color channels (R, G, B) and 224*224 represents the image size. Following the input layer is a convolutional layer using a 7*7 kernel and a sliding window operation to extract features from the image, focusing on local features. A stride of 2 is set to reduce the size of the feature map. After the convolutional layer is a pooling layer using 3*3 max pooling, selecting the maximum value from each pooling window as the output, with a stride of 2 to further reduce the feature map size. This is followed by four residual blocks. The first residual block consists of two 3*3 convolutional layers with a stride of 1 and padding of 1. The second residual block has an input size of 64*56*56, consisting of two convolutional layers. To reduce the input size and data volume, the stride of the first convolutional layer is 2, resulting in an output size of 128*28*28. The next two residual blocks are the same as the second, with a stride of 1 for each convolutional layer. Each convolutional layer is followed by batch normalization and a ReLU activation function. After the last residual block, ResNet-18 uses a global average pooling layer to reduce the dimensionality of the feature map, converting it into a fixed-size vector. By calculating the average value of each feature map channel, the spatial dimension of the feature map is reduced to one dimension. Following the global average pooling layer is an output layer. The output layer size is set to 3*512*512, which provides good classification performance and is beneficial for input to brightness enhancement algorithms.

[0063] Further, in step S103: the low-light scene image to be enhanced is input into a brightness enhancement network corresponding to the scene category, and the enhanced image is output, including:

[0064] The image of the low-light scene to be enhanced in the dark outdoors at night is input into the brightness enhancement network in the dark outdoors at night, and the enhanced image is output.

[0065] The low-light scene image to be enhanced in the dark outdoor environment at night is input into the brightness enhancement network in the dark outdoor environment at night, and the enhanced image is output.

[0066] The image of the low-light scene to be enhanced in a dark room is input into the brightness enhancement network in the dark room, and the enhanced image is output.

[0067] The low-light scene image to be enhanced in a dimly lit indoor environment is input into a brightness enhancement network in the dimly lit indoor environment, and the enhanced image is output.

[0068] Further, in step S103: the low-light scene image to be enhanced is input into a brightness enhancement network corresponding to the scene category, and the enhanced image is output; wherein, the brightness enhancement network has the following network structure:

[0069] The first convolutional module, the second convolutional module, the third convolutional module, the fourth convolutional module, the fifth convolutional module, the sixth convolutional module, and the seventh convolutional module are connected in sequence;

[0070] The output of the first convolutional module is residually connected to the input of the sixth convolutional module;

[0071] The output of the second convolutional module is residually connected to the input of the fifth convolutional module;

[0072] The output of the third convolution module is residually connected to the input of the fourth convolution module.

[0073] Furthermore, the internal structures of the first convolutional module, the second convolutional module, the third convolutional module, the fourth convolutional module, the fifth convolutional module, and the sixth convolutional module are the same. The first convolutional module includes a convolutional layer and an activation function layer ReLU connected in sequence.

[0074] Furthermore, the seventh convolutional module includes: a convolutional layer and an activation function layer tanh connected in sequence.

[0075] like Figure 2 As shown, the overall architecture of the neural network for fitting the curve consists of seven convolutional layers (six hidden layers and one output layer). All layers are 3x3 convolutional layers with 32 kernels and an equal stride (1). To maintain the connection between adjacent pixels, batch normalization is not used after the convolutional layers. The activation function for the hidden layers is ReLU, as shown... Figure 3 Since the output falls within [-1, 1], the activation function of the output layer is tanh. Repeating conv-ReLU six times with conv-Tanh results in a symmetrical concatenation, meaning the outputs of layers 1 / 2 / 3 are channel-concatenated with the outputs of layers 6 / 5 / 4. For example... Figure 2 As shown, the six hidden layers use symmetrical skip connections similar to U-Net for shallow feature reuse. The outputs of layers 3 and 4 are concatenated before being fed into layer 5, the outputs of layers 2 and 5 are concatenated before being fed into layer 6, and the outputs of layers 1 and 6 are concatenated before being fed into layer 7. After the output layer, each pixel has 24 channels—3 color channels (RGB), and each channel has 8 parameters. The output is the A value obtained by iterating through the R, G, and B channels.

[0076] Further, in step S103: the low-light scene image to be enhanced is input into a brightness enhancement network corresponding to the scene category, and the enhanced image is output; wherein, the brightness enhancement network is expressed by the following formula:

[0077] T n (G(x); γ)=G n-1 (x) (i)(1+γ n )-γ n G n-1 (x) (i)

[0078] Where x is the pixel coordinate, G(x) is the brightness value, γ is the learning parameter with a value range of (-1, 1), and to ensure good performance, prevent overfitting, and satisfy monotonically increasing, i is set to 2, T n (G(x); γ) represents the enhanced image, G n-1 (x) (i) γ represents the brightness value after n-1 iterations. n This represents the learning parameters after n iterations.

[0079] It should be understood that the following three points must be met during the fitting of the brightness curve: 1. The brightness values ​​should fall within the interval [0,1], and the curve should take a value of 0 at 0 and a value of 1 at 1, in order to ensure that the range of brightness values ​​remains unchanged. 2. The curve must be monotonically increasing. If it is not monotonically increasing, bright areas in the image will become darker. 3. It must be differentiable everywhere.

[0080] This method uses the following brightness curve formula:

[0081] T(G(x); γ)=G(x) (i) (1+γ)-γG(x) (i)

[0082] Where x is the pixel coordinate, G(x) is the brightness value, and γ is the learning parameter with a value range of (-1, 1). To ensure the effect, prevent overfitting, and satisfy monotonically increasing, i is set to 2 here.

[0083] To complicate matters, this method iterates the function using nested iterations and determines the number of iterations by analyzing its performance curve:

[0084] T n (G(x); γ)=G n-1 (x) (i) (1+γ n )-γ n G n-1 (x) (i) .

[0085] Furthermore, the method also includes: using a total loss function to limit the number of iterations during the training of the brightness enhancement network; stopping the iteration when the total loss function reaches its minimum value, thus obtaining the trained brightness enhancement network; wherein, the total loss function is specifically expressed as:

[0086]

[0087] Among them, L c For color constant error, L e For exposure control error, L s For spatial consistency error, L t Let W be the illumination smoothing error, and W be the weight of each error constraint.

[0088] Among them, the color constant error is:

[0089]

[0090] Here, (i,j) iterates through all pairwise combinations of the three color channels. The value of a certain color channel in the image should not be significantly higher than that of the other two channels. P i This represents the average brightness of color channel i.

[0091] Among them, exposure control error:

[0092]

[0093] Where, constant This is the median brightness value; based on testing, we've chosen 0.55 here. Z is the average brightness across a large pixel area. N is the total number of large pixels. This error is designed to control brightness levels, preventing some areas from being too bright or too dark, and minimizing extreme brightness, even if each pixel's brightness is closer to the median value.

[0094] Spatial consistency error:

[0095]

[0096] Where M is the number of large pixels, and i is the number of times the pixels are traversed. Ω(i) is the four-neighborhood of the i-th pixel, E is the pooled image of the enhanced image, and I is the pooled image of the input image; Spatial consistency error aims to prevent the difference between a pixel value and its neighboring pixel values ​​from changing too much;

[0097] Among them, the illumination smoothing error is:

[0098]

[0099] Where N is the number of iterations, It is a horizontal gradient operator. It's the vertical gradient operator. The illumination smoothing error aims to make the brightness changes between adjacent pixels less significant, thus making it smoother.

[0100] Further, in step S103: the low-light scene image to be enhanced is input into a brightness enhancement network corresponding to the scene category, and the enhanced image is output; wherein, the training process of the brightness enhancement network includes:

[0101] S103-1: Construct teacher and student networks for four scene categories; the internal structure of the teacher and student networks is consistent with the brightness enhancement network; under the same scene category, there is a one-to-one correspondence between the teacher network and the student network;

[0102] S103-2: Construct a second training set, which includes low-light scene images of known enhanced images;

[0103] S103-3: Input the second training set into the teacher network of the first scene category, train the teacher network of the first scene category, and obtain the trained teacher network of the first scene category;

[0104] S103-4: Input the second training set into the student network of the first scene category to train the student network of the first scene category;

[0105] S103-5: The first scenario category teacher network will use the trained parameters γ t The first iteration training results of the student network in the first scenario category γ s _1. Calculate the cross-entropy, use the cross-entropy as an influence factor and assign it a weight to obtain the first generation result γ after distillation learning. * 1;

[0106] S103-6: The first generation result γ * 1 as the next generation of teacher network parameter γ t The result of the next iteration of training of the student network γ s _2 Calculate the cross-entropy and obtain the second-generation distillation result γ * 2;

[0107] S103-7: Continue in this manner until the cross-entropy loss of the nth generation falls below a set value. Finally, the distillation results γ of each generation are calculated. * The summation and averaging of i yields the parameter γ for the final network requirements of students in the first scenario category;

[0108] S103-8: Apply the final trained parameters γ to the brightness enhancement network as the brightness enhancement network for the first scene category;

[0109] S103-9: Similarly, obtain the brightness enhancement network for four scene categories.

[0110] Further, in S103-3: the second training set is input into the teacher network of the first scene category, and the teacher network of the first scene category is trained to obtain the trained teacher network of the first scene category; wherein, the teacher network of the first scene category uses the analytic hierarchy process to calculate the weights of four constraints: exposure control error, color constant error, illumination smoothing error and spatial consistency error, including exposure control error, color constant error, illumination smoothing error and spatial consistency error.

[0111] like Figure 6 As shown, the weight calculation formula is as follows:

[0112] First, calculate the m-th power of the product of each row to obtain an m-dimensional vector:

[0113]

[0114] Here, the loss constraints of the brightness enhancement algorithm are four, so m is set to 4.

[0115] Standardizing the vectors gives us the weight vector, which in turn gives us the weights:

[0116]

[0117] It should be understood that, firstly, the teacher model is trained on the training set to obtain the output target γ1 of the teacher model. Then, the student model is trained on the same training set to obtain the output target γ2 of the student model. From this, the cross-entropy formula of the teacher-student network can be obtained:

[0118]

[0119] The teacher model's parameters are set to γ1, and the student model's parameters are set to γ2. Therefore, the student model's loss function not only depends on the true labels but also fits the parameter γ through cross-entropy. Based on this, the principle of continuous distillation is used to use the learning outcomes γ of the previous generation of students as the new generation's teacher to train the next generation of students.

[0120]

[0121] By iterating continuously until the cross-entropy is lower than the set parameter β, the learned parameter γ of each generation is obtained. Finally, the outputs of all generations are summed and averaged to obtain the final parameter γ.

[0122]

[0123] When calculating the cross-entropy gradient, the cross-entropy is calculated for the probabilities of each sub-class:

[0124]

[0125] To distinguish between correct labels (*) and incorrect labels (i), the formula can be broken down as follows:

[0126]

[0127] This formula can be broken down into two parts: the correction of the gradient for the negative label in the latter part, and the correct label in the former part.

[0128] This allows for results that are better than those of a single network. This invention utilizes a self-distillation system instead of directly changing the network weights, avoiding the uncontrollable training direction and inferior results caused by directly setting weights based on the scene. This makes the training direction controllable, and the training effect improves based on the scene.

[0129] It should be understood that the teacher network adopted the following measures: using the analytic hierarchy process (AHP) to set corresponding constraint weights for different scenarios, thereby achieving a more scenario-specific purpose.

[0130] The Analytic Hierarchy Process (AHP) is a decision-making method that decomposes decision-related elements into multiple levels, such as objectives, criteria, and alternatives. It then performs qualitative and quantitative analysis based on these levels. The core steps are constructing a comparison matrix and conducting consistency checks.

[0131] Constructing a comparison matrix involves comparing each element pairwise to determine the weight of each criterion layer relative to the target layer. A scale from 1 to 9 is used to represent their relative importance. The comparison matrix is ​​a square matrix where each element represents the relative importance of one criterion or choice to another. The scaling criteria are determined by… Figure 5 As shown, a comparison matrix can be created using scaling criteria.

[0132] In the four scenarios that have been set up, the relative importance levels of the four error constraints are different. Therefore, four corresponding comparison matrices can be listed for these four scenarios to obtain weight parameters that are more suitable for the scenarios.

[0133] Knowledge distillation, as you can understand it, is a machine learning technique used to transfer knowledge from a complex model to a simpler one. This approach aims to simplify the model while preserving the original model's performance and predictive ability. In knowledge distillation, a large, complex model is called the teacher model, and a smaller, simpler model is called the student model. The teacher model is first trained, and its output is used as the objective to train the student model. By using the teacher model's output as an additional objective or leveraging the teacher model's soft objective (probability distribution), the student model can benefit from the teacher model's knowledge and learn better feature representations and generalization capabilities. It can compress a complex model into a smaller, lighter model, thus reducing model size and computational resource requirements. Secondly, the student model can gain better performance and generalization ability by learning from the teacher model's knowledge. Furthermore, knowledge distillation can help the student model combat noise and uncertainty and improve its predictive ability on unseen data. In this invention, the aim is to enhance the effect based on different scenarios. However, directly setting weights based on scenarios can lead to uncontrollable training direction, resulting in a situation where the effect is not as good as the initial setting. Using knowledge distillation can ensure that the training direction is controllable and can also improve the effect. Therefore, this invention uses a knowledge distillation system instead of directly changing the network weights.

[0134] Since the brightness enhancement network itself is not very deep or complex, this method uses self-distillation. Self-distillation refers to knowledge distillation learning where the teacher network and student network have the same structure or the same network complexity but different structures. This method designs different teacher networks for different scenarios and uses continuous knowledge distillation, constantly using the previous generation of students as new teachers to train the next generation of students, thus iterating to obtain a more accurate algorithm model.

[0135] Self-distillation learning is a technique for knowledge distillation that aims to improve the performance of a model by having it itself or a network of similar complexity act as a teacher. Self-distillation learning achieves knowledge distillation by having the student model generate its own target distribution, without requiring an additional teacher model. The basic idea is to have the student model generate a soft target during training and use it as its own supervisory signal for learning. The soft target is typically a probability distribution obtained by smoothing the student model's output; it can be seen as the student model's confidence level for different classes. This approach allows the student model to learn from its own predictions while incorporating knowledge from the teacher model. The structure of a self-distillation system is as follows: Figure 4 As shown.

[0136] Using the previously prepared datasets for the following four scenarios: 1. Outdoor nighttime with no light; 2. Outdoor nighttime with dim light; 3. Indoor nighttime with no light; 4. Indoor nighttime with low light), a self-distillation system designed for each scenario was used to train the corresponding datasets, obtaining training parameters for these four different scenarios. After an image is classified by the ResNet-18 network, the image is used to perform inference using the parameters corresponding to the scenario, finally obtaining the inference result.

[0137] Finally, in practical applications, it is deployed on embedded devices. Because the model of this invention is lightweight, it is easy to deploy on embedded devices, has a fast inference speed, and good results, meeting real-world needs. Low-light video is decoded, input into the model, and after inference, an enhanced image is obtained.

[0138] like Figure 1 As shown, the input low-light image first passes through the ResNet-18 residual network to classify the image into similar environments. Then, it passes through a brightness enhancement network with a self-distillation system within this environment for brightness enhancement. The algorithm utilizes the relationship between the brightness curve and the image to enhance the low-light image, and improves the image processing speed and enhancement effect through a self-distillation method.

[0139] Since setting weights directly based on the scenario can lead to uncontrollable training direction and the results may be worse than the initial settings, the self-distillation method can ensure that the training direction is controllable and the results are better. Therefore, this invention uses a self-distillation system instead of directly changing the network weights.

[0140] Embedded device deployment:

[0141] (1) First, convert the network model into an ONNX model. At this time, the network model needs to be properly pruned, the size of the input image is set, and the pruned network model and the trained weight file are combined to convert it into an ONNX model.

[0142] (2) Convert the ONNX model to a format required by different embedded systems, such as the OM offline model required by Ascend devices. At this time, it is necessary to set the model's running mode, original framework type, input data format, input data size, output file name, and the version of the running chip.

[0143] (3) Write the runtime file, including calling the decoding interface, importing relevant libraries, writing pre-processing and post-processing, importing the model for use, setting the inference data type, allocating the memory and functions of operators in the embedded device, and scheduling available computing power. In this invention, the convolution operations of ResNet-18 and the brightness enhancement algorithm can be processed in parallel using a dedicated module responsible for convolution. This not only increases speed but also makes full use of the resources of the embedded device. Kernel fusion can also be performed. Conv only uses matrix calculation units, and ReLU only uses vector calculation units. If each is implemented using a separate kernel function, each kernel can only use a portion of the calculation units and will cause a large amount of memory movement, which will introduce high overhead. The matrix operation of Conv and the vector operation of ReLU are completed in a single data transfer process. Implementing kernel fusion can save memory reading, writing, and movement, and improve inference performance.

[0144] (4) Perform inference. The input image is first classified into a scene by ResNet-18 to achieve the first step of scene classification. Then it is passed through the low-light enhancement network of the corresponding scene, and output after inference.

[0145] (5) Set the save path and display effect for the inference results. The results can be saved in a local folder, displayed on an external monitor, or wirelessly transmitted via Bluetooth.

[0146] The overall process of this invention is as follows: Figure 7 As shown. This invention, based on existing residual network technology, combines a brightness enhancement algorithm and a self-distillation system to propose solutions to some problems encountered in low-light images. Based on a lightweight network model, it proposes using a simple ResNet-18 model and a low-light enhancement model to address the need for fast, real-time inference in practical applications. Based on knowledge distillation technology, it proposes using a self-distillation method to learn the network, addressing issues related to the effectiveness and accuracy of image enhancement. Based on the idea of ​​scene-specific enhancement, it further improves the image enhancement effect in different scenes by dividing the training set into different scenes, training with the corresponding training set for each scene, and using the corresponding self-distillation system for distillation learning. Based on the idea of ​​adaptability to different scenes, it proposes using ResNet-18 for classification followed by a brightness enhancement network for brightness enhancement, addressing the need for low-light image enhancement to adapt to different lighting conditions, including different brightness levels, color temperatures, and light source types, and effectively enhancing the image under different lighting conditions. Based on the above method, this invention builds a complete and optimized low-light image brightness enhancement system. It does not require additional redundant modules to complete the system performance optimization, and can achieve efficient and accurate image brightness enhancement. It is also more suitable for embedded implementation.

[0147] Example 2

[0148] This embodiment provides a low-light scene image enhancement system based on scene adaptation and self-distillation;

[0149] A low-light scene image enhancement system based on scene adaptation and self-distillation includes:

[0150] The acquisition module is configured to acquire images of low-light scenes to be enhanced.

[0151] The scene classification module is configured to: input the low-light scene image to be enhanced into the trained scene classification model and output the scene classification result;

[0152] The image enhancement module is configured to input the low-light scene image to be enhanced into a brightness enhancement network corresponding to the scene category, and output the enhanced image.

[0153] It should be noted that the acquisition module, scene classification module, and image enhancement module described above correspond to steps S101 to S103 in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.

[0154] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0155] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0156] Example 3

[0157] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0158] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0159] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0160] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0161] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0162] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0163] Example 4

[0164] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0165] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A low-light scene image enhancement method based on scene adaptation and self-distillation, characterized by: include: Acquire low-light scene images to be enhanced; The low-light scene image to be enhanced is input into the trained scene classification model, and the scene classification result is output. The low-light scene image to be enhanced is input into a brightness enhancement network corresponding to the scene category, and the enhanced image is output. The training process of the brightness enhancement network includes: 3-1: Construct teacher and student networks for four scene categories; the internal structure of the teacher and student networks is consistent with the brightness enhancement network; within the same scene category, there is a one-to-one correspondence between the teacher network and the student network; 3-2: Construct a second training set, which includes low-light scene images with known augmented images; 3-3: Input the second training set into the teacher network of the first scene category, train the teacher network of the first scene category, and obtain the trained teacher network of the first scene category; 3-4: Input the second training set into the student network of the first scene category to train the student network of the first scene category; 3-5: The first scenario category teacher network will use the trained parameters The first iteration training results of the student network in the first scenario category Calculate the cross-entropy, use the cross-entropy as an influence factor and assign it a weight to obtain the first generation result after distillation learning. ; 3-6: Results of the first generation As a parameter for the next generation of teachers' networks The results of the next iteration of training of the student network Calculate the cross-entropy and obtain the second-generation distillation results. ; 3-7: Continue in this manner until the cross-entropy loss of the nth generation falls below a set value. Finally, record the distillation results of each generation. The summation and averaging of i yields the parameter γ for the final network requirements of students in the first scenario category; 3-8: The final parameters γ after training are applied to the brightness enhancement network as the brightness enhancement network for the first scene category; 3-9: Similarly, obtain brightness enhancement networks for four scene categories.

2. The low-light scene image enhancement method based on scene adaptation and self-distillation as described in claim 1, characterized in that, The training process for the trained scene classification model includes: Construct a first training set, which includes: low-light scene images with known scene classification labels; the scene classification labels include: no light outdoors at night, dim light outdoors at night, no light indoors, and weak light indoors. The first training set is input into the scene classification model to train the model and obtain the trained scene classification model.

3. The low-light scene image enhancement method based on scene adaptation and self-distillation as described in claim 1, characterized in that, The low-light scene image to be enhanced is input into a brightness enhancement network corresponding to the scene category, and the enhanced image is output, including: The image of the low-light scene to be enhanced in the dark outdoors at night is input into the brightness enhancement network in the dark outdoors at night, and the enhanced image is output. The low-light scene image to be enhanced in the dark outdoor environment at night is input into the brightness enhancement network in the dark outdoor environment at night, and the enhanced image is output. The image of the low-light scene to be enhanced in a dark room is input into the brightness enhancement network in the dark room, and the enhanced image is output. The low-light scene image to be enhanced in a dimly lit indoor environment is input into a brightness enhancement network in the dimly lit indoor environment, and the enhanced image is output.

4. The low-light scene image enhancement method based on scene adaptation and self-distillation as described in claim 1, characterized in that, The low-light scene image to be enhanced is input into a brightness enhancement network corresponding to the scene category, and the enhanced image is output; wherein, the brightness enhancement network has the following network structure: The first convolutional module, the second convolutional module, the third convolutional module, the fourth convolutional module, the fifth convolutional module, the sixth convolutional module, and the seventh convolutional module are connected in sequence; The output of the first convolutional module is residually connected to the input of the sixth convolutional module; The output of the second convolutional module is residually connected to the input of the fifth convolutional module; The output of the third convolution module is residually connected to the input of the fourth convolution module; The internal structures of the first convolutional module, the second convolutional module, the third convolutional module, the fourth convolutional module, the fifth convolutional module, and the sixth convolutional module are the same. The first convolutional module includes a convolutional layer and an activation function layer ReLU connected in sequence. The seventh convolutional module includes: a convolutional layer and an activation function layer tanh connected in sequence.

5. The low-light scene image enhancement method based on scene adaptation and self-distillation as described in claim 1, characterized in that, The low-light scene image to be enhanced is input into a brightness enhancement network corresponding to the scene category, and the enhanced image is output; the brightness enhancement network is expressed by the following formula: in, These are pixel coordinates. It's the brightness value. The learning parameters take values ​​in the range (-1, 1). To ensure good performance, prevent overfitting, and satisfy the condition of monotonically increasing, the value of i is set to 2. This represents the enhanced image. This represents the brightness value after n-1 iterations. This represents the learned parameters after n iterations.

6. The low-light scene image enhancement method based on scene adaptation and self-distillation as described in claim 1, characterized in that, The method further includes: using a total loss function to limit the number of iterations during the training of the brightness enhancement network; stopping the iteration when the total loss function reaches its minimum value, thus obtaining the trained brightness enhancement network; wherein, the total loss function is specifically expressed as: + + + ； in, For color consistency error, To control exposure error, For spatial consistency error, W represents the illumination smoothing error, and W represents the weight of each error constraint. Among them, the color constant error is: Here, (i,j) iterates through all pairwise combinations of the three color channels, ensuring that the value of one color channel in the image should not be significantly higher than the other two channels. This represents the average brightness of color channel i; Among them, exposure control error: Where, constant Z is the median value of brightness, which is taken as 0.55 based on the effect test. Z is the average brightness of a large pixel area, and N is the total number of large pixels. Spatial consistency error: Where M is the number of large pixels, i is the number of times the pixels are traversed; Ω(i) is the four-neighborhood of the i-th pixel, E is the image after pooling the enhanced image, and I is the image after pooling the input image. Among them, the illumination smoothing error is: Where N is the number of iterations, It is a horizontal gradient operator. It is the vertical gradient operator.

7. A low-light scene image enhancement system based on scene adaptation and self-distillation, characterized by: include: The acquisition module is configured to acquire images of low-light scenes to be enhanced. The scene classification module is configured to: input the low-light scene image to be enhanced into the trained scene classification model and output the scene classification result; The image enhancement module is configured to: input the low-light scene image to be enhanced into a brightness enhancement network corresponding to the scene category, and output the enhanced image; wherein, the training process of the brightness enhancement network includes: 3-1: Construct teacher and student networks for four scene categories; the internal structure of the teacher and student networks is consistent with the brightness enhancement network; within the same scene category, there is a one-to-one correspondence between the teacher network and the student network; 3-2: Construct a second training set, which includes low-light scene images with known augmented images; 3-3: Input the second training set into the teacher network of the first scene category, train the teacher network of the first scene category, and obtain the trained teacher network of the first scene category; 3-4: Input the second training set into the student network of the first scene category to train the student network of the first scene category; 3-5: The first scenario category teacher network will use the trained parameters The first iteration training results of the student network in the first scenario category Calculate the cross-entropy, use the cross-entropy as an influence factor and assign it a weight to obtain the first generation result after distillation learning. ; 3-6: Results of the first generation As a parameter for the next generation of teachers' networks The results of the next iteration of training of the student network Calculate the cross-entropy and obtain the second-generation distillation results. ; 3-7: Continue in this manner until the cross-entropy loss of the nth generation falls below a set value. Finally, record the distillation results of each generation. The summation and averaging of i yields the parameter γ for the final network requirements of students in the first scenario category; 3-8: The final parameters γ after training are applied to the brightness enhancement network as the brightness enhancement network for the first scene category; 3-9: Similarly, obtain brightness enhancement networks for four scene categories.

8. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-6.

9. A storage medium, characterized in that, The computer-readable instructions are stored non-transitory, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-6 are executed.

Images