Method for enhancing nighttime concrete images based on state space encoding

Abstract

The present invention discloses a method for enhancing nighttime concrete images based on state-space coding, which relates to the field of digital image processing and includes the following steps: S1, constructing a nighttime concrete image enhancement algorithm based on state-space convolutional deep network technology, which is composed of a state-space module and a convolutional module; S2, constructing a training model based on the nighttime concrete image enhancement algorithm in S1; and S3, inputting a test sample into the training model constructed in S2 to obtain an enhanced concrete image, wherein the training model evaluates the enhancement effect based on a loss function. The present invention provides a nighttime concrete image enhancement method based on state-space coding. The proposed nighttime concrete image enhancement algorithm is used for feature extraction and multi-level feature fusion of images at different levels, thereby improving performance, reducing the number of model parameters, and accelerating inference speed compared to low-light enhancement algorithms based on Transformers.

Description

Technical Field

[0001] The present invention relates to the field of digital image processing, and more particularly to a method for enhancing nighttime concrete images based on state space coding. Background Art

[0002] With the increasing popularity of artificial intelligence algorithms, deep learning methods have garnered significant attention in both industry and academia. During concrete quality testing, cameras can be used to capture the flow of concrete to determine whether its slump meets production standards. However, during nighttime concrete production, due to factors such as insufficient exposure, concrete images captured at night can often appear dark in whole or in parts, with difficulty capturing detailed information, color distortion, and significant noise. In low-light environments, long exposure times or insufficient sensitivity can produce low-quality images with high noise, low contrast, and poor visibility, which can affect subsequent concrete slump measurements. Therefore, low-light image enhancement technology is needed to improve image quality and provide a reliable foundation for subsequent concrete slump testing.

[0003] At present, low-light image enhancement algorithms can be roughly divided into image enhancement algorithms based on traditional technologies and image enhancement algorithms based on deep learning, among which:

[0004] Traditional methods include grayscale transformation, histogram equalization, and Retinex. Grayscale transformation is a spatial domain processing method. Its essence is to modify the grayscale of each pixel in the image according to certain rules to change the image's grayscale range. Typical grayscale transformations can be divided into linear and nonlinear transformations. Histogram equalization uses the cumulative distribution function to adjust the output grayscale, making the image's grayscale distribution more uniform. Retinex theory is based on the idea that the color of an object depends on its ability to reflect long-wave (red), medium-wave (green), and short-wave (blue) illumination, rather than the absolute value of the reflected light. Image enhancement is achieved by estimating the incident component from the original image and then processing it. The most classic algorithms currently include the single-level Retinex algorithm, the multi-level Retinex algorithm, and the MSR algorithm with color restoration. However, these methods suffer from the problem of over-enhancement, resulting in significant color deviation.

[0005] Deep learning-based methods include Jain et al., who used a variant of stacked sparse denoising autoencoders to brighten and denoise low-light images. Lu et al. proposed an end-to-end multi-branch enhancement network that extracts effective features through a feature extraction module, an enhancement module, and a fusion module to improve the performance of low-light enhancement models. To reduce the computational burden of deep learning models based on Retinex theory, Li et al. proposed a lightweight LightenNet network for low-light image enhancement. Some teams have proposed using generative adversarial networks (GANs) to enhance low-light images. Through the adversarial process, the generator learns to produce images that are more natural and closer to real-world lighting conditions. Jiang et al. proposed EnlightenGAN, which provides effective enhancement without relying on paired training data. Yu et al. learned exposure photos through reinforcement adversarial learning, segmenting the input image into sub-images based on exposure. For each sub-image, a policy network sequentially learned the local exposure using reinforcement learning. Yang et al. proposed a semi-supervised deep recurrent band network (DRBN). DRBN first restores the linear band representation of the enhanced image under supervised learning, and then reorganizes the given bands through a learnable linear transformation based on unsupervised adversarial learning to obtain the improved band representation. However, these methods cannot effectively deal with problems such as severe noise and color distortion in enhanced images.

[0006] Therefore, although some existing methods can achieve good performance, they still have some limitations: For unsupervised and semi-supervised learning methods, achieving stable training, avoiding color bias, and establishing cross-domain information relationships still pose certain challenges. For reinforcement learning, designing an effective reward mechanism and implementing it effectively and stably are complex and difficult. Most existing nighttime single-frame image enhancement algorithms are only effective for low-light images without noise. However, the enhancement results for particularly dark areas often suffer from severe noise and color distortion, making it difficult to meet practical needs.

[0007] Furthermore, application number 202310828756.4 is a low-light image enhancement method based on feature fusion and attention embedding. It introduces attention and feature fusion to effectively solve the problem that convolutional networks have difficulty in capturing global image information. However, the introduction of the attention mechanism leads to a huge increase in the number of model parameters, which easily makes the model difficult to deploy on some low-computing power devices.

[0008] For example, application number 201810246429.7 is a low-light image enhancement method based on convolutional neural network. It uses the convolution method to enhance the image, but the convolution is poor in capturing the global information between image contents, and artifacts and overexposure are prone to occur during the image enhancement process.

[0009] Another example is a multi-exposure image fusion method based on a multi-level autoencoder in application number 202211424921.1, which only enhances grayscale low-light images but does not solve the problem of enhancing low-light images.

[0010] As can be seen from the above, all existing methods are prone to local over-enhancement, under-enhancement, and color distortion, resulting in poor robustness of traditional methods. In addition, existing methods based on Transformer and convolutional neural networks have problems such as large number of model parameters, high computing power requirements, and convolution blocks can only capture local image information. In summary, there is currently no effective enhancement method for night-time concrete images. Summary of the Invention

[0011] An object of the present invention is to solve at least the above problems and / or disadvantages and to provide at least the advantages which will be described hereinafter.

[0012] In order to achieve these objectives and other advantages of the present invention, a method for enhancing nighttime concrete images based on state-space coding is provided, comprising:

[0013] S1. Using state-space convolutional deep network technology, a nighttime concrete image enhancement algorithm combining state-space modules and convolution modules is constructed.

[0014] S2. Build and train a neural network model based on the nighttime concrete image enhancement algorithm in S1;

[0015] S3: Input the test sample into the trained neural network model to obtain the enhanced image, and evaluate the enhancement effect through the overall loss function in the model. If the enhancement effect meets the expected requirements, the neural network model training is considered complete; if not, return to S2 for further adjustment;

[0016] S4. Deploy the trained neural network model to the edge device to enhance the nighttime concrete falling image through the neural network model and feed it back to the host computer for real-time display;

[0017] In S2, the nighttime concrete image enhancement algorithm is trained based on the deep learning framework Pytorch platform. The overall loss function used in model training is: L total It is expressed by the following formula:

[0018]

[0019] In the above formula, 、 、 RespectivelyL 2. L SSIM 、 L color The weight coefficient of L 2 is the Euclidean distance loss, L SSIM is the structural similarity loss, L color is the color loss, and L 2. L SSIM 、 L color They are expressed by the following formulas:

[0020]

[0021] In the above formula, x i 、 y i They represent the RGB values of the model output pixels and the RGB values of the normal exposure image pixels, respectively. X 、 Y Represent the enhanced image and normal exposure image output by the model, respectively. n is the number of image pixels, SSIM is the structural similarity formula, G high To capture the color distribution of images during the day with normal exposure, G pred To capture the color distribution of images with normal exposure through training, and enhance the color distribution of night images, H is the height of a single frame image, W The width of a single frame image.

[0022] Preferably, in S1, the method further includes: using a camera to capture the concrete falling process during the day and at night to form a video dataset, and cropping images in the video dataset to generate an image dataset for training:

[0023] In S2, the daytime concrete image dataset in the training dataset is used as the true value, and the nighttime concrete image dataset is used as the data to be enhanced. After initializing the neural network parameters, the overall loss function is used. L total Iteratively train the neural network.

[0024] Preferably, the nighttime concrete image enhancement algorithm includes a multi-level downsampling module, a multi-level encoder, and a multi-level fusion decoder, and its processing flow includes:

[0025] S10, obtain images at three different levels: detail, local, and global from a single-frame nighttime image through a multi-level downsampling module;

[0026] S11, processing the images at the detail, local, and global levels obtained in S10 respectively by the corresponding detail encoder module, local encoder module, and global encoder module in the multi-level encoder to obtain a preprocessed concrete image;

[0027] S12, the pre-processed images obtained by the encoder modules at each level in S11 are spliced using a multi-level fusion decoder, and up-sampled after the splicing process to obtain an enhanced image.

[0028] Preferably, in S11, a standardized convolution layer and a state space image processing layer are defined as a standard state space module, the convolution kernel size of the standardized convolution layer is 1×1, and the convolution step size is 1;

[0029] Wherein, the detail encoder module includes a standard state space module;

[0030] The local encoder module includes three standard state space modules and residual mapping;

[0031] The global encoder module includes a standard state-space module and a state-space module, and the state-space module includes two state-space image processing layers.

[0032] Preferably, in S12, the splicing process of the multi-level fusion decoder is:

[0033] S120. In the global encoder module, the output after a standard state space module is defined as feature image I, the output after a global encoder module is defined as feature image II, and the output not after a global encoder module is defined as feature image III. Feature image I, feature image II, and feature image III are spliced according to the number of channels, and the spliced result is then quadrupled and the number of channels is reduced through a convolution layer to obtain a global feature image extraction result. F high ;

[0034] S121. In the local encoder module, the output after one standard state space module is defined as a feature image V, the output after two standard state space modules is defined as a feature image VI, the output after three standard state space modules is defined as a feature image VII, and the output not after the local encoder module is defined as a feature image VIII. Feature images V, VI, VII, VIII, and VIII are sorted by the number of channels. F highSplicing is performed, and then the splicing result is passed through two convolutional layers to expand the number of convolution channels to obtain the feature image extraction result F middle ;

[0035] S122, the S121 F middle After being concatenated with the feature image X output by the detail encoder module, the enhanced image is obtained by upsampling by a factor of two.

[0036] Preferably, the processing flow of the multi-level fusion decoder includes:

[0037] S130, using Pixel-Shuffle to upsample the output of the global encoder module by a factor of 4, and connect it to the output of the local encoder module according to the depth, so as to fuse the feature information extracted from the global encoder module and the local encoder module through a convolution operation;

[0038] S131, upsample the convolution result in S130 by four times, connect it to the output of the detail fusion module according to the depth, and then pass it through a convolution layer to generate the restored RGB image.

[0039] Preferably, in S2, the calculation formula of SSIM is:

[0040]

[0041] in, , , ;

[0042] In the above formula, μ X for X The average value of μ Y for Y The average value of for X and Y The standard deviation of C 1. C 2 is a constant, for X The standard deviation of for Y The standard deviation of for Y The standard deviation of is the weighting coefficient, N is the number of image pixels, for X Standard deviation.

[0043] The present invention has at least the following beneficial effects: the present invention adopts a nighttime concrete image enhancement method based on state-space equations, which can well capture global and local information of the image, ensure the quality of image enhancement, further improve the quality of the enhanced image, and has low computing power requirements. The present invention can be applied to a color enhancement system based on state-space coding, enabling the system to be used for nighttime concrete image enhancement based on deep learning, greatly improving the enhancement quality of nighttime concrete image enhancement, and bringing innovative solutions to the field of nighttime image enhancement. Specifically, it has the following effects:

[0044] (1) Reducing artifacts after nighttime image enhancement: Introducing deep learning and state-space coding to reduce artifacts caused by model enhancement;

[0045] (2) Using the optimized loss function: By designing the color loss function, the overexposure of the enhanced image can be reduced.

[0046] (3) Multi-scenario support: Compared with traditional image enhancement methods, this method can be trained and improved for different scenarios, which improves the generalization ability of the model and can be optimized for different scenarios.

[0047] (4) Continuous optimization and improvement: By collecting concrete image datasets during the day and at night in real time, in-depth data analysis can be performed, and continuous training can be performed on night images to improve night enhancement effects.

[0048] (5) Using state-space model encoding, low computational complexity: Compared with the night enhancement model based on the Transformer attention mechanism, the model has lower parameter count and computational complexity, and the model inference time is short, which can achieve real-time effects.

[0049] (6) Real-time detection: The device can enhance nighttime concrete images in real time, avoiding the timeliness problem of traditional methods.

[0050] Other advantages, objectives and features of the present invention will be reflected in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The schematic diagram of the processing flow after the nighttime concrete image enhancement system based on state space coding of the present invention is built;

[0052] Figure 2 Schematic diagram of the processing flow of the multi-level nighttime concrete image enhancement algorithm of the present invention;

[0053] Figure 3 Schematic diagram of the basic architecture of the state space model in the present invention;

[0054] Figure 4 Schematic diagram of the final architecture of the state space model in the present invention. DETAILED DESCRIPTION

[0055] The present invention will be described in further detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0056] A real-time enhancement method for nighttime surveillance video based on state-space coding, the technical solution of which mainly includes:

[0057] 1. Concrete images captured by camera: Daytime and nighttime videos of falling concrete are captured by camera as video dataset.

[0058] 2. Dataset processing: Crop the collected dataset to a suitable image size.

[0059] 3. Train the neural network: Use the daytime video dataset as the true value and the nighttime video as the data to be enhanced, initialize the neural network parameters, and then iteratively train the neural network using the designed loss function.

[0060] 4. Deploy the detection device: Deploy the trained neural network model to the device to achieve real-time detection function.

[0061] 5. Real-time detection and feedback: The camera transmits video data to the neural network model, which enhances the nighttime video stream data in real time and feeds it back to the host computer for real-time display.

[0062] 6. Maintenance and calibration: Regularly maintain the camera device and neural network model to ensure normal operation, and calibrate the device to adapt to different nighttime conditions.

[0063] Specifically, a nighttime concrete image enhancement method based on state-space coding is proposed. By introducing a state-space model and a convolutional network fusion method to enhance nighttime concrete images, the method can better capture the global and local information of the image. In addition, the method uses multiple layers of encoders for parallel computing, which enables the model to obtain enhanced images more quickly on some devices capable of parallel computing. The model has relatively few parameters and faster inference speed. The method specifically includes the following steps:

[0064] S1. Device Construction

[0065] Install the camera device to ensure that it can capture the video stream and transmit it to the host computer for display; pass the captured night-time concrete image through the state space image enhancement network to obtain the enhanced single-frame image, which is then displayed on the host computer, such as Figure 1 shown.

[0066] S2. Design an image enhancement algorithm consisting of a multi-level encoder and a single-frame image multi-level fusion decoder module.

[0067] Based on the state-space convolutional deep network technology, a nighttime concrete image enhancement algorithm consisting of a state-space module and a convolution module is designed. The multi-level image downsampling module in S1 uses pooling kernel sizes of 2×2, 4×4, and 16×16. When a single-frame image of the original size is input, images downsampled 4 times, 16 times, and 256 times can be obtained, which are detail, local, and global images, respectively. Specifically, the nighttime concrete image enhancement algorithm includes the following:

[0068] 1. Multi-level downsampling module for single-frame images: This module consists of different pooling kernel sizes to obtain low-light images at different levels. The pooling method used is max pooling.

[0069] 2. Standard state-space module: The state-space module consists of a standardized convolution layer and a state-space image processing layer. The convolution kernel size of the standardized convolution layer is 1×1 and the convolution stride is 1.

[0070] 3. Detail encoder module: The detail encoder module consists of a standard state space module.

[0071] 4. Local Encoder Module: The local encoder module is composed of three standard state-space modules concatenated at the last layer.

[0072] 5. Global Encoder Module: The global encoder module consists of a standard state-space module and a state-space module consisting of two state-space image processing layers.

[0073] 6. Multi-layer fusion decoder module: In the global encoder module, the outputs of the standard state-space module, the global encoder module, and the outputs that have not passed through the global encoder module are concatenated according to the number of channels. The concatenated result is then upsampled by a factor of four. Finally, the number of channels is reduced through a convolutional layer. The result is then concatenated with the output of the local encoder module according to the number of channels. The concatenated result is passed through two convolutional layers to increase the number of channels. The concatenated result is then concatenated with the output of the detail encoder module and finally upsampled by a factor of two to obtain the enhanced image.

[0074] It should be further explained that the multi-level encoder described in S1 is composed of a standard state space module, a detail encoder module, a local encoder module and a global encoder module;

[0075] The multi-level encoder structure adopts a pyramid parallel encoder structure. The length and width of the input original image are scaled by 4 times, 8 times, and 16 times to obtain detailed, local, and global images. When processing the image at each level, each level is an independent encoder structure.

[0076] The standard state-space module data processing pipeline is as follows: the input image is divided into 1×1 non-overlapping patches, and then the image dimensions are mapped to C′ . The generated embedded image is normalized using hierarchical normalization and then fed back to the state space module for feature extraction. The state space module is divided into two branches: the first branch is processed by a linear layer and an activation function, and the second branch is processed by a linear layer, a depthwise separable convolution and an activation function, and then enters 2D-Selective-Scan (SS2D) for processing. SS2D consists of three steps: (1) Flatten a two-dimensional feature into a one-dimensional vector along four different directions (upper left, lower right, lower left, upper right). (2) The four one-dimensional vectors obtained in the previous step are sent to the selective state space image processing layer for operation. (3) The four one-dimensional vectors obtained are fused into a two-dimensional feature as output. After feature normalization, they are merged by element-wise multiplication with the output of the first branch, and then a linear layer is used to mix the features and added with the residual connection to form the output of the VSS block. LeakyReLU is used as the activation function.

[0077] The multi-layer fusion decoder consists of the following steps: First, the feature information extracted by the global and local modules is fused. The output of the global encoder module is upsampled by a factor of 4 using pixel shuffle and concatenated depthwise to the output of the local encoder module, followed by a convolution operation. Finally, the convolution result is upsampled by a factor of 4 and concatenated depthwise to the output of the detail fusion module. A convolution layer then generates the restored RGB image.

[0078] S3. Build a neural network model and train it

[0079] Based on the image enhancement algorithm designed in S2, a multi-level state-space convolutional deep network low-light enhancement model was built. The model was trained using the deep learning framework Pytorch platform. It was iterated 100,000 times on the LOL-v2 training set. The learning rate started from 1e-4 and decreased by 0.1 times after every 1,000 iterations.

[0080] It should be noted that the loss function used by the deep learning framework Pytorch platform in training the model in S2 includes Euclidean distance loss L 2. Structural Similarity Loss L SSIM and color loss Lcolor :

[0081] The overall loss function is L total :

[0082]

[0083]

[0084] The calculation formula of SSIM is:

[0085] in, , , ;

[0086] In the above formula, μ X for X The average value of μ Y for Y The average value of for X and Y The standard deviation of C 1 、 C 2 is a constant, for X The standard deviation of for Y The standard deviation of for Y The standard deviation of is the weighting coefficient, N is the number of image pixels, for X Standard deviation.

[0087] for G high In order to obtain the color distribution of daytime images with normal exposure by training the UNet network used to capture the color distribution of images, G pred In order to obtain the color distribution of the image after normal exposure and enhancement of the night image by training the UNet network for capturing the image color distribution, H is the height of a single frame image, W The width of a single frame image.

[0088] S4. Output results

[0089] The low-light single-frame image in the night video of the test set of the dataset is input into the low-light enhancement model trained in S3, and the corresponding enhanced image is output.

[0090] Example:

[0091] like Figure 2 This paper proposes a multi-level nighttime concrete image enhancement algorithm based on a state space and convolution hybrid model. With almost no loss of image enhancement effect, it has fewer parameters and a faster model inference speed than other deep learning methods. The algorithm mainly includes the following steps:

[0092] S1, a RGB three-channel length H , width is W Low-light images P The length and width are downsampled by 2 times, 4 times, and 16 times respectively, and the 2 times downsampled image is output P 2x , 4x downsampled image P 4x , 16x downsampled image P 16x The role of downsampling is to reduce the size of the image, reduce the amount of calculation, and extract features from images at three different levels, obtaining feature information at different levels and improving the generalization ability of the model.

[0093] S2. Feature extraction is performed on images at different levels through different encoder structures, and the parameters of the encoders at each level are shown in Table 1:

[0094] Table 1

[0095]

[0096] The specific contents of encoders at different levels are as follows:

[0097] S20, 2x downsampled image P 2x First, a detail encoder is used to extract image feature information. This detail encoder is a standard state-space module with two main components: a 1×1 convolutional layer that extracts image feature information; and a state-space image processing layer that further processes and optimizes the extracted features. This structural design not only effectively reduces the number of model parameters but also ensures the efficiency and quality of feature extraction. The resulting image features can be used in subsequent tasks such as image analysis and restoration, improving image processing performance and effectiveness.

[0098] S21, 4x downsampled image P 4xImage features are obtained through a local encoder. The local encoder consists of three standard state-space modules and a residual map. In this encoder, the input image is first processed by the first standard state-space module, and the output is then passed through two more standard state-space modules in sequence. After processing by each standard state-space module, the outputs of each stage are concatenated according to the number of channels.

[0099] S22, 16x downsampled image P 16x The image features are obtained through the global encoder.

[0100] The global encoder module consists of a standard state-space module and a state-space module consisting of two state-space image processing layers.

[0101] The specific details of the state-space image processing layer mentioned above are as follows:

[0102] like Figure 3 The state-space image processing layer shown, considering the input , x is the input sequence, Indicates a C′ Channel, the height of each channel single frame image is H , width is W Tensor of .

[0103] Establish the following continuous state space model : , ;

[0104] Among them, in the continuous state space, is the intermediate state, is the input sequence, is the output sequence, A 、 B 、 C is the learnable parameter matrix, A Size is ( D , N ), B Size is ( b,l,N ), C Size is ( b,l,N ),in, D is the dimension of the input vector, N is the dimension of the hidden layer, b is the batch size, l is the sequence length.

[0105] As the parameters are continuously updated in the iteration rounds, the above continuous state space is discretized and can be expressed as: , ,in, 、 is the discretized learnable parameter matrix, h k for k The intermediate state of the moment, h k-1 for k -1 moment intermediate state, x k for k The moment input sequence; the discrete rule is:

[0106] ,

[0107] Among them, ∆ is the step size, which is ( b,l,N ),and , h 0 is the initial intermediate state, x 0 is the initial input sequence, is the identity matrix, and exp(·) is the exponential function.

[0108] After deducing multiple time series inputs, the convolution representation can be obtained:

[0109]

[0110] in, , Used to represent the learnable parameter matrix. Usually is the HiPPO matrix, where 、 、 .

[0111] In order to make the model selective and input-dependent, the final model is:

[0112] ,

[0113] in, 、 、 In order to make the discretized learnable matrix 、 、 A regular function that changes dynamically with the input parameters, namely: , , ,

[0114] x t for t Input at the moment, ht-1 for t -1 moment intermediate state, h t for t The intermediate state at the moment. Matrix A 、 B 、 C Both ∆ and ∆ are learnable parameters that are iterated during training. The input image is divided into 1×1 non-overlapping patches and then the image dimensions are mapped to C′ The resulting embedded image is normalized using hierarchical normalization and then fed back to the state space module for feature extraction. Figure 4 The final architecture diagram shown ( Figure 4 In the figure, the multiplication sign indicates the element-by-element multiplication of the matrix, and σ indicates the SiLu activation function). The state space processing module is divided into two branches: the first branch is processed by the linear mapping layer and the activation function SiLu, and the second branch is processed by the linear mapping layer, depth-wise separable convolution and the activation function SiLu, and then enters the state space model for processing.

[0115] S3. Concatenate the feature map of the image after passing through the global encoder module and the output of the image before passing through the global encoder module according to the number of channels. The specific contents are as follows:

[0116] Assume the input is , divide the input into 4×4 non-overlapping patches, and then x The dimensions are mapped to C′ , this process produces an embedded image , and finally use Layer Normalization to x′ Normalized and then sent to the SSM encoder for feature extraction. The SSM encoder consists of four stages, and a patch merging operation is applied at the end of the first three stages to reduce the height and width of the input features while increasing the number of channels. [2,2,2,2] VSS blocks are used in the four stages, and the number of channels in each stage is [ C′′ ,2 C′′ ,4 C′′ , 8 C′′ ]. Assume that the number of channels after encoding is C 1, the number of channels that have not passed through the encoder is C 2, then the final number of output channels is ( C 1+ C 2) Then, a convolutional layer is used to reduce the number of channels. The concatenated result is quadrupled to obtain the global encoder feature extraction result. F high The upsampling method is the PixelShuffle method provided by Pytorch.

[0117] S4. F high The output of the local encoder module is spliced with the number of channels, and the spliced result is passed through two convolutional layers to expand the number of convolution channels. It is then spliced with the output of the detail encoder module and finally upsampled by a factor of two to obtain the enhanced image.

[0118] The weight parameters of each network designed in the low-light image enhancement method of this invention are continuously updated during the training process based on the training data and loss function. The parameters of each network model are saved until the training process converges. In practical applications, only a low-light image and a lighting adjustment parameter need to be input to obtain the enhanced effect at the output.

[0119] Specifically, for a dataset containing hundreds of paired images with different exposure times, the normal exposure images are marked as P h , low-light images are marked as P l . The paired images [ P h , P l ] is input into the network to obtain the final enhanced image. In this invention, a new loss function is constructed, which includes three constraints, which are specifically expressed as follows:

[0120] in, , , , x i 、 y i They represent the RGB values of the pixels output by the model and the RGB values of the pixels in the normal exposure image respectively. X 、 Y Represent the enhanced image and normal exposure image output by the model respectively. G high 、 G pred The color distribution of the normally exposed daytime picture and the color distribution of the enhanced nighttime image are obtained by training the UNet network for capturing the image color distribution. L 2. The purpose of this item is to make the model results as consistent as possible with the normal exposure image results. L SSIM The role of is to enhance without losing the structural information of the image. L color The purpose of is to allow the model to learn the color distribution of the image.

[0121] S5. Input the paired training sets of normal exposure and low light images into the enhanced model network designed in steps 2-4 for training. The specific parameters for model training in this example are shown in Table 2:

[0122] Table 2

[0123]

[0124] S5. Input the test set into the model trained in steps 2-4 to test and verify the model's effectiveness. The training set uses a mixture of original and augmented data, while the test set uses the original data to ensure the credibility of the model's effectiveness on the original data. On the League of Legends-v2 dataset, all models use the same training and test sets.

[0125] It can be seen from the above examples that the present invention has the following effects:

[0126] 1. Low number of model parameters

[0127] The method of the present invention adopts multiple parallel coding methods at different levels. Compared with the U-Net linear coding method, it reduces the number of model parameters and improves the model calculation speed. The final model parameter number is about 1.40M, and the model floating-point operation number is about 4.16G.

[0128] In the existing technology, the deep learning technology with better image restoration performance, such as the KinD model, has 8.02M parameters and about 34.99G of model floating-point operations, while the EnlightGAN model has 114.35M parameters and 61.01G of model floating-point operations.

[0129] 2. More effective loss function

[0130] The method of the present invention uses a method in which three types of loss functions interact to enable the model to learn information in the image, namely the L2 loss function, the ms-SSIM loss function and the color loss function, thereby improving the enhancement effect of the model.

[0131] In the existing technology, traditional methods usually only use ms-SSIM loss and L2 norm loss function as the total loss function, and lack loss functions involving image color information.

[0132] 3. Data Parallel Processing

[0133] The present invention processes data in parallel on different downsampling modules through multi-level downsampling, and finally fuses the data processed in parallel by the multi-level downsampling modules through a first-level feature fusion module, thereby improving the running speed of the model.

[0134] Existing Technology: Currently, many image enhancement models use linear feature extraction and processing models such as U-Net. This structure has been proven to achieve good image enhancement results, but due to the linear processing, the image enhancement speed is generally slow.

[0135] 4. Supports optimized training for various complex scenarios

[0136] When the present invention needs to be transferred to other scenes, daytime and nighttime concrete videos shot in other scenes can be collected, and the enhanced model can be fine-tuned and retrained based on the original trained model to adapt to the current scene.

[0137] In the existing technology, the current night-time concrete image enhancement model is enhanced for the current complex night-time scene. When migrating to a different scene, the model parameters need to be reinitialized and retrained.

[0138] 5. Capturing global image features

[0139] The present invention improves the global capture capability of an image by designing a multi-level feature extraction module and a state space model.

[0140] However, in the existing technology, traditional image enhancement methods based on convolutional networks can only capture local features of the image and have weak global image capture capabilities.

[0141] 6. Fast model inference speed

[0142] Through reasonable network structure design, the model of this method only needs 34ms to process a 1024×1024 low-light image.

[0143] However, in the existing technology, traditional methods may have slow reasoning speed due to problems such as the model being too large.

[0144] 7. Continuous optimization and improvement

[0145] The present invention can conduct in-depth data analysis by collecting data sets of concrete falling during the day and at night in real time, and can continuously train nighttime surveillance videos to improve nighttime enhancement effects.

[0146] However, in existing technologies, traditional methods are usually unable to provide large-scale data for analysis and improvement.

[0147] The above solution is only an illustration of a preferred embodiment, but is not limited thereto. When implementing the present invention, appropriate replacements and / or modifications can be made according to user needs.

[0148] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and exemplary embodiments. They can be applied to a variety of fields suitable for the present invention. Further modifications will be readily apparent to those skilled in the art. Therefore, the present invention is not limited to the specific details and illustrations shown and described herein without departing from the general concept defined by the claims and their equivalents.

Claims

1. A method for enhancing nighttime concrete images based on state space coding, characterized in that: include: S1. Using state-space convolutional deep network technology, a nighttime concrete image enhancement algorithm combining state-space modules and convolution modules is constructed. S2. Build and train a neural network model based on the nighttime concrete image enhancement algorithm in S1; S3: Input the test sample into the trained neural network model to obtain the enhanced image, and evaluate the enhancement effect through the overall loss function in the model. If the enhancement effect meets the expected requirements, the neural network model training is considered complete; if not, return to S2 for further adjustment; S4. Deploy the trained neural network model to the edge device to enhance the nighttime concrete falling image through the neural network model and feed it back to the host computer for real-time display; In S2, the nighttime concrete image enhancement algorithm is trained based on the deep learning framework Pytorch platform. The overall loss function used in model training is: L total It is expressed by the following formula: In the above formula, L 2 is the Euclidean distance loss, L SSIM is the structural similarity loss, L color is the color loss, and L 2. L SSIM 、 L color They are expressed by the following formulas: In the above formula, x i 、 y i They represent the RGB values of the model output pixels and the RGB values of the normal exposure image pixels, respectively. X 、 Y Represent the enhanced image and normal exposure image output by the model, respectively. n is the number of image pixels, SSIM is the structural similarity formula, G high To capture the color distribution of images during the day with normal exposure, G pred To capture the color distribution of images with normal exposure through training, and enhance the color distribution of night images, H is the height of a single frame image, W is the width of a single frame image; The nighttime concrete image enhancement algorithm includes a multi-level downsampling module, a multi-level encoder, and a multi-level fusion decoder. The processing flow includes: S10, obtain images at three different levels: detail, local, and global from a single-frame nighttime image through a multi-level downsampling module; S11, processing the images at the detail, local, and global levels obtained in S10 respectively by the corresponding detail encoder module, local encoder module, and global encoder module in the multi-level encoder to obtain a preprocessed concrete image; S12, splicing the pre-processed images obtained by the encoder modules at each level in S11 using a multi-level fusion decoder, and up-sampling them after the splicing process to obtain an enhanced image; In S11, a standardized convolution layer and a state space image processing layer are defined as a standard state space module, wherein the convolution kernel size of the standardized convolution layer is 1×1 and the convolution step size is 1; Wherein, the detail encoder module includes a standard state space module; The local encoder module includes three standard state space modules and residual mapping; The global encoder module includes a standard state space module and a state space module, wherein the state space module includes two state space image processing layers; The state space image processing layer establishes the following continuous state space model : ， in, 、 、 In order to make the discretized learnable matrix 、 、 Regular functions that change dynamically with input parameters, x t for t Input at the moment, h t-1 for t -1 moment intermediate state, h t for t The intermediate state of the moment.

2. The method for enhancing nighttime concrete images based on state space coding according to claim 1, characterized in that: S1 also includes: using a camera to capture the concrete falling process during the day and at night to form a video dataset, and cropping the images in the video dataset to generate an image dataset for training: In S2, the daytime concrete image dataset in the training dataset is used as the true value, and the nighttime concrete image dataset is used as the data to be enhanced. After initializing the neural network parameters, the overall loss function is used. L total Iteratively train the neural network.

3. The method for enhancing nighttime concrete images based on state space coding according to claim 1, characterized in that: In S12, the splicing process of the multi-level fusion decoder is as follows: S120. In the global encoder module, the output after a standard state space module is defined as feature image I, the output after a global encoder module is defined as feature image II, and the output not after a global encoder module is defined as feature image III. Feature image I, feature image II, and feature image III are spliced according to the number of channels, and the spliced result is then quadrupled and the number of channels is reduced through a convolution layer to obtain a global feature image extraction result. F high ; S121. In the local encoder module, the output after one standard state space module is defined as a feature image V, the output after two standard state space modules is defined as a feature image VI, the output after three standard state space modules is defined as a feature image VII, and the output not after the local encoder module is defined as a feature image VIII. Feature images V, VI, VII, VIII, and VIII are sorted by the number of channels. F high Splicing is performed, and then the splicing result is passed through two convolutional layers to expand the number of convolution channels to obtain the feature image extraction result F middle ; S122, the S121 F middle After being concatenated with the feature image X output by the detail encoder module, the enhanced image is obtained by upsampling by a factor of two.

4. The method for enhancing nighttime concrete images based on state space coding according to claim 1, characterized in that: The processing flow of the multi-level fusion decoder includes: S130, using Pixel-Shuffle to upsample the output of the global encoder module by a factor of 4, and connect it to the output of the local encoder module according to the depth, so as to fuse the feature information extracted from the global encoder module and the local encoder module through a convolution operation; S131, upsample the convolution result in S130 by four times, connect it to the output of the detail fusion module according to the depth, and then pass it through a convolution layer to generate the restored RGB image.

5. The method for enhancing nighttime concrete images based on state space coding according to claim 1, characterized in that: In S2, the calculation formula of SSIM is: in, , , ; In the above formula, μ X for X The average value of μ Y for Y The average value of for X and Y The standard deviation of C 1. C 2 is a constant, for X The standard deviation of for Y The standard deviation of for Y The standard deviation of is the weighting coefficient, N is the number of image pixels, for X Standard deviation.

Images