Extremely weak light imaging method based on deep neural network cutting and quantification technology

Abstract

The invention discloses an extremely-weak light imaging method based on a deep neural network cutting and quantification technology, and the method comprises the following steps: collecting a plurality of images in an extremely-weak light environment, and obtaining an original image; performing image enhancement on the original image and converting the original image into a full-color image by using a deep neural network model which is subjected to network cutting and quantization processing on the evaluation result of the importance of the neurons; wherein the neuronal importance evaluation result is obtained based on activation value distribution, gradient information and weight of neurons in the deep neural network model; and outputting the full-color image. By optimizing the network structure and processing the parameters, the collected extremely-weak light image is processed to generate a clearer and more real full-color image, meanwhile, the calculation efficiency is improved, the storage requirement is reduced, and the defects in the prior art are overcome.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to an extremely weak light imaging method based on deep neural network clipping and quantization technology. Background Art

[0002] Obtaining high-quality full-color images in extremely low-light environments has always been a challenging problem. Current imaging technologies have the following main problems:

[0003] (1) Limitations of traditional imaging methods: Under low-light conditions, traditional imaging methods based on optics and sensors often have difficulty capturing sufficient light signals, resulting in insufficient brightness of the obtained images and blurred details. For example, at night or in dim indoor environments, photos taken with ordinary cameras often have a large amount of noise and obvious color distortion. Traditional imaging technology has poor adaptability to extremely weak light environments and cannot effectively process weak light signals, making it difficult to meet the needs of some special scenarios such as deep-sea exploration and nighttime monitoring for high-quality images.

[0004] (2) Limitations of existing low-light imaging technologies: Although some existing low-light imaging technologies have made some progress in improving image quality, their effects are still limited. For example, some image enhancement algorithms may cause over-enhancement, which in turn leads to image quality problems such as halo and artifacts. These technologies are often accompanied by high computational complexity and require a lot of computing resources and time. Unoptimized networks will degrade in performance under extremely weak light, such as poor noise robustness and inaccurate color reproduction. In addition, the model is large in scale and computationally complex, which is a huge challenge for resource-constrained devices.

[0005] (3) Challenges of deep neural networks in extremely weak light imaging: Although deep neural networks have achieved remarkable results in the field of image processing, there are still many challenges in extremely weak light full-color imaging. In existing low-light imaging technologies, algorithms based on histogram equalization can enhance image contrast, but are prone to over-stretching, resulting in loss of details and color distortion; algorithms based on Retinex theory often produce halos in practical applications due to parameter settings and illumination estimation problems. In terms of computational complexity, some traditional algorithms take a long time to process high-resolution extremely weak light images. For example, a traditional algorithm takes more than 5 seconds to process a 1080p extremely weak light image. The present invention aims to solve these problems and provide an efficient extremely weak light imaging.

[0006] In summary, there is an urgent need for a new method that can effectively solve the problem of full-color imaging in extremely weak light environments. Summary of the invention

[0007] The purpose of the present invention is to provide an extremely weak light imaging method based on deep neural network cropping and quantization technology. By optimizing the network structure and parameter processing, the collected extremely weak light images are processed to generate clearer and more realistic full-color images, while improving computing efficiency and reducing storage requirements, so as to overcome the shortcomings of the prior art.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] An extremely weak light imaging method based on deep neural network clipping and quantization technology comprises the following steps:

[0010] Collect multiple images in extremely weak light environments to obtain original images;

[0011] Using the deep neural network model after network clipping and quantization processing based on the evaluation result of neuron importance, the original image is enhanced and converted into full-color imaging; wherein the evaluation result of neuron importance is obtained based on the activation value distribution, gradient information and weight size of the neurons in the deep neural network model;

[0012] The full-color image is output.

[0013] Furthermore, the network pruning process includes: sorting the importance and connections of neurons based on the neuron importance evaluation results, marking neurons and connections whose importance is lower than a preset threshold as pruning candidates, and identifying and retaining the key layers for processing the original image and neurons whose importance is higher than another preset threshold through visualization tools or analysis of the intermediate output of the deep neural network model.

[0014] Further, the visualization tool includes: a feature map visualization tool or a gradient visualization tool;

[0015] The visualization tool is used to display the response of neurons to features in the original image and the gradient changes of neurons in back propagation to assist in identifying key layers and important neurons.

[0016] Further, the key layers of the original image processing and neurons whose importance is higher than another preset threshold are retained, including:

[0017] Set retention thresholds for the key layers and neurons with high importance in the deep neural network model that process the original image;

[0018] After each cropping, the validation set is used to evaluate the performance of the cropped deep neural network model;

[0019] Fine-tune the parameters of the deep neural network model based on the performance evaluation results.

[0020] Furthermore, fine-tuning the parameters of the deep neural network model according to the performance evaluation results includes:

[0021] For the key layers of the original processing or the key layers that have a greater impact on performance in the deep neural network model, add regularization terms and adjust hyperparameters, and retrain.

[0022] Furthermore, the quantization process includes: quantizing weights and activation values;

[0023] The weight is quantified as follows:

[0024] Determine the number of quantization bits and use linear quantization to map the weight parameters to a smaller value range;

[0025] Or use the distribution characteristics of weights by nonlinear quantization to retain important weight parameters through mapping functions;

[0026] The activation value is quantified, specifically:

[0027] The quantization parameter of the activation value is dynamically adjusted based on the activation value distribution or the input raw data.

[0028] Furthermore, quantizing the weights and activation values ​​further includes:

[0029] The weights and activation values ​​are quantized first, and then forward propagation and backpropagation calculations are performed.

[0030] Furthermore, quantizing the weights and activation values ​​further includes:

[0031] Add compensation values ​​to the quantized weights and activation values, specifically:

[0032] By analyzing the mean and variance of the quantization error, the size and direction of the compensation value are determined and added to the quantized weights and activation values.

[0033] Furthermore, the multiple images in the extremely weak light environment are enhanced and converted into full-color images, specifically:

[0034] The original image is input into a deep neural network model after network cropping and quantization processing to perform image enhancement, and the enhanced grayscale image is converted into a full-color image using a color conversion algorithm.

[0035] Furthermore, the calculation method of the mapping function is specifically as follows:

[0036] Using the quantization method of the logarithmic function, the quantized result Q(ω) of the weight value ω satisfies Q(ω)=αlog(β|ω|+1); wherein α and β are constants determined according to the weight distribution.

[0037] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0038] 1. The present invention significantly reduces the number of network parameters and the amount of calculation through network pruning and quantization processing. After pruning and quantization, the network parameters can be reduced by 30%-50%, and the calculation time can be reduced by an average of 40%, which improves the calculation efficiency and meets the real-time requirements. It is suitable for resource-constrained devices such as embedded systems and mobile devices, and improves the user experience.

[0039] 2. Greatly reduce storage requirements. After processing, network storage requirements are reduced by an average of 50%, saving storage space and facilitating the storage and processing of more image data, which is crucial for devices with limited storage resources.

[0040] 3. Clipping and quantization can remove network noise and redundant information, making the neural network model more concise and efficient, enhancing the robustness to noise and interference in extremely weak light environments, better processing weak light signals, and improving image quality. Experiments show that the image signal-to-noise ratio is improved by an average of 8dB and the clarity is improved by an average of 30% after processing, making the generated image clearer and more realistic.

[0041] 4. Smaller-scale networks are easier to deploy to different devices, whether cloud servers or edge devices. They can be widely used in various extremely weak light scenarios such as deep-sea exploration, field surveys, and night monitoring to achieve high-quality full-color imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] The extremely weak light imaging method based on deep neural network clipping and quantization technology of the present invention is further described below in conjunction with the accompanying drawings;

[0044] Figure 1 It is a schematic diagram of the overall process of the extremely weak light imaging method based on deep neural network clipping and quantization technology provided by the present invention. DETAILED DESCRIPTION

[0045] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0046] In order to better understand the purpose, structure and function of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0047] like Figure 1 As shown, the present invention provides an extremely weak light imaging method based on deep neural network clipping and quantization technology, comprising the following steps:

[0048] Collect multiple images in extremely weak light environments to obtain original images;

[0049] Using the deep neural network model after network clipping and quantization processing of the evaluation result of neuron importance, the original image is enhanced and converted into full-color imaging; wherein the neuron importance evaluation result is obtained based on the activation value distribution, gradient information and weight size of the neurons in the deep neural network model;

[0050] The full-color image is output.

[0051] It should be noted that the deep neural network model is used to process extremely weak light images. The network model includes multiple convolutional layers, pooling layers and fully connected layers, which can effectively extract the feature information of the image.

[0052] The deep neural network model after network pruning and quantization processing using the evaluation results of neuron importance includes:

[0053] Evaluate the distribution of activation values ​​of neurons when processing different input images;

[0054] During the back propagation process, the gradient information of each neuron is calculated;

[0055] The size of the weight parameter that measures the connection between two neurons;

[0056] The network is pruned based on the activation value distribution, the gradient information of each neuron, and the neuron importance evaluation results obtained based on the weight size.

[0057] The deep neural network model after network pruning and quantization processing using the evaluation results of neuron importance, wherein before network pruning, it also includes:

[0058] sorting the importance of neurons and connections based on the neuron importance evaluation result, and marking neurons and connections with lower importance as pruning candidates;

[0059] Through visualization tools or analysis of the intermediate outputs of the deep neural network model, key layers and neurons with high importance for extremely weak light image processing can be identified and retained;

[0060] Preprocess multiple images in extremely weak light environments and divide them into training sets and validation sets. Input the training set into the deep neural network model without network cropping and quantization to obtain the predicted value.

[0061] The loss function is used to calculate the difference between the predicted value and the true value, and the weights and parameters are adjusted so that the deep neural network model that has not undergone network cropping and quantization processing can learn the characteristics and rules of extremely weak light images.

[0062] The network is pruned based on the neuron importance evaluation result obtained based on the activation value distribution, the gradient information of each neuron and the weight size, wherein the network pruned process includes:

[0063] Set retention thresholds for the key layers and their important neurons in the deep neural network model that process extremely low-light images;

[0064] After each cropping, the validation set is used to evaluate the performance of the cropped deep neural network model;

[0065] Fine-tune the parameters of the deep neural network model based on the performance evaluation results.

[0066] The method of fine-tuning the parameters of the deep neural network model according to the performance evaluation results includes:

[0067] For the key layers of the deep neural network model that process extremely weak light images or have a greater impact on performance, add regularization terms and adjust hyperparameters, and retrain.

[0068] The quantization includes: quantizing weights and activation values;

[0069] The weight is quantified as follows:

[0070] Determine the number of quantization bits and use linear quantization to map the weight parameters to a smaller value range;

[0071] Or use the distribution characteristics of weights by nonlinear quantization to retain important weight parameters through mapping functions;

[0072] The activation value is quantified as follows:

[0073] The quantization parameter of the activation value is dynamically adjusted based on the activation value distribution or the input raw data.

[0074] The weight and activation value quantization further includes:

[0075] The weights and activation values ​​are quantized first, and then forward propagation and backpropagation calculations are performed.

[0076] The weight and activation value quantization further includes:

[0077] Add compensation values ​​to the quantized weights and activation values, specifically:

[0078] By analyzing the mean and variance of the quantization error, the size and direction of the compensation value are determined and added to the quantized weights and activation values.

[0079] The multiple images in the extremely weak light environment are enhanced and converted into full-color images, specifically:

[0080] The original image is input into a deep neural network model after network cropping and quantization processing to perform image enhancement, and the enhanced grayscale image is converted into a full-color image using a color conversion algorithm.

[0081] Embodiment 1

[0082] 1. Before network cropping, use a low-light camera to shoot in extremely weak light scenes such as at night, deep sea, and dim caves. The image format can be common JPEG, PNG and other formats. These original images are used as the basic data for subsequent processing, and they are preprocessed, including normalization, data enhancement and other operations to improve the quality and diversity of the data. Then, select appropriate optimization algorithms and loss functions, such as stochastic gradient descent algorithm and cross entropy loss function, train the deep neural network model network, and continuously adjust the network weights and parameters so that the network can learn the characteristics and laws of extremely weak light images and improve the processing ability of extremely weak light images; through multiple iterative training, until the deep neural network model network converges or reaches the predetermined performance indicators. The fully trained network model has better performance and stability, providing a solid foundation for subsequent network cropping.

[0083] 2. The process of network cutting includes the following steps:

[0084] Step 1: Evaluate importance. Perform a comprehensive analysis of the constructed deep neural network, and use evaluation indicators such as neuron activation value distribution, gradient information, weight size, etc. to determine the importance of each neuron and connection. The details are as follows:

[0085] Evaluation using activation value distribution: observe the distribution of activation values ​​of neurons when processing different input images. If the activation value of a neuron is always very close to zero or distributed in a very narrow range, it means that the neuron is insensitive to changes in the input and may contribute less to the processing of the image. For example, in a convolutional layer, when processing various extremely weak light images, the activation values ​​of some neurons fluctuate almost in a very small range, such as [0.01, 0.02], while the activation value range of other neurons may be between [-1, 1]. Then these neurons with a smaller activation value range can be used as candidates for pruning, judging based on gradient information: During the back propagation process, the gradient of each neuron is calculated. The gradient reflects the degree of influence of the neuron on the final output. If the gradient of a neuron is very small, it means that it does not play a big role in the parameter update process and contributes relatively little to the processing of the image. For example, when training the network, the gradient of some neurons is always below 0.001, while the gradient of most neurons is between 0.01 and 0.1. Then these neurons with smaller gradients may not be very important and can be considered for pruning.

[0086] Measure by weight: Analyze the weights of the connections between neurons. Connections with larger weights usually mean that the relationship between the corresponding two neurons is more important. Conversely, connections with very small weights may have less impact on the output of the network. For example, in a fully connected layer, some connections have a weight of only 0.0001, while other connections have weights between 0.1 and 1. Then these connections with extremely small weights can be used as targets for pruning, specifically:

[0087] Evaluate the distribution of activation values ​​of neurons when processing different input images. Observe the neuron activation values. If they are always close to zero or have a narrow distribution, such as when some neurons in the convolutional layer process extremely low-light images, their activation values ​​range from [0.01, 0.02], while others are between [-1, 1], then the former can be used as a candidate for cropping.

[0088] During the back propagation process, the gradient information of each neuron is calculated. If the gradient of a neuron is small, such as always below 0.001 (most neuron gradients are between 0.01 and 0.1), its contribution to image processing may be low, and you can consider pruning it.

[0089] A measure of the size of the weight parameter of the connection between two neurons. Connections with very small weights, such as some weights of only 0.0001 in the fully connected layer (others are between 0.1 and 1), can be used as pruning targets.

[0090] Based on the above evaluation results, neurons and connections are sorted, and neurons and connections whose importance is lower than a preset threshold (which can be determined based on experiments or experience) are marked as candidates for pruning. At the same time, by displaying the distribution of neuron activation values, feature maps, etc. through visualization tools (such as TensorBoard, etc.), or analyzing the intermediate output of the deep neural network model (such as calculating the information entropy of each layer output), the key layers for extremely weak light image processing and neurons whose importance is higher than another preset threshold are identified and retained, and the retention ratio of key layers is not less than 80%, and the retention ratio of key neurons is not less than 70%.

[0091] Step 2: Develop a pruning strategy. First, sort the importance of neurons and connections based on the evaluation results of step 3.1. Mark neurons and connections with lower importance as pruning candidates. Then, determine the initial pruning ratio. This ratio can be determined based on experience or through some small-scale experiments. For example, you can start with a smaller ratio, such as 10%. Then, gradually prune neurons or connections according to the pruning ratio. After each pruning, the performance of the network needs to be evaluated. The following methods can be used to evaluate network performance:

[0092] Use the validation set for evaluation: Use a portion of extremely weak light image data that was not used in the training as the validation set. Apply the cropped network to the validation set images and calculate some commonly used image quality evaluation indicators, such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). If the decrease in these indicators compared to before cropping is within an acceptable range, for example, the PSNR decrease does not exceed 1dB, and the SSIM decrease does not exceed 0.05, then the network performance is considered to be within an acceptable range and the next round of cropping can be continued. If the indicator decreases significantly, you need to adjust the cropping ratio or stop cropping, specifically:

[0093] After each cropping, the validation set (including extremely weak light images that did not participate in training) was used to evaluate the performance of the cropped deep neural network model using indicators such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). If the PSNR drops by no more than 1 dB, the SSIM drops by no more than 0.05.

[0094] Conduct subjective evaluation: Invite professionals to conduct subjective evaluation of the cropped images. Score the image quality based on image clarity, color reproduction, detail expression, etc. If the average score does not drop significantly compared to before cropping, it means that the network performance is acceptable and cropping can continue. If the score drops significantly, you need to reconsider the cropping strategy, specifically:

[0095] Invite professionals to subjectively evaluate the image quality score. If the decrease is not obvious, the cropping can continue; otherwise, adjust the cropping ratio or stop cropping. After cropping, add regularization terms (such as L1 or L2 regularization) and adjust hyperparameters (such as setting the learning rate to one-tenth of the original training) to the key layers for extremely weak light image processing or key layers that have a greater impact on performance, and then retrain.

[0096] Step 3: Ensure structural stability and functional integrity. During the trimming process, the structural stability and functional integrity of the network need to be considered comprehensively to retain key layers and neurons and ensure that the basic functions of the network in extremely weak light image processing are not affected. The details are as follows:

[0097] Identify key layers and neurons: By analyzing the structure and function of the network, determine which layers and neurons are critical to extremely weak light image processing. For example, in a convolutional neural network, the convolutional layers close to the input layer are usually responsible for extracting low-level features of the image, such as edge and texture information, while the layers close to the output layer are responsible for extracting and classifying high-level features. For extremely weak light images, the extraction of low-level features is often more critical, so these convolutional layers close to the input layer can be regarded as key layers. At the same time, neurons that show higher activation values ​​or larger gradients during training may also be key neurons. These key layers and neurons can be identified through visualization tools or by analyzing the intermediate outputs of the network.

[0098] Retain key parts: During the pruning process, priority is given to retaining the identified key layers and neurons. For those non-critical parts, they can be gradually pruned according to the pruning strategy. For example, if a convolutional layer is determined to be a key layer, most of the neurons and connections of this layer can be retained as much as possible during pruning. At the same time, a threshold for the retention ratio can be set to ensure that the retention of the key parts is high enough. For example, it can be stipulated that the retention ratio of key layers should not be less than 80%, and the retention ratio of key neurons should not be less than 70%.

[0099] Fine-tune the pruned network: Fine-tune the pruned network appropriately to further optimize network performance. The following methods can be used for fine-tuning:

[0100] Retrain some parameters: Select some key layers or layers that have a greater impact on performance and retrain their parameters. A smaller learning rate can be used to avoid causing excessive disturbance to the network. For example, the learning rate can be set to one-tenth of the original training. By retraining these parameters, the network can be better adapted to the pruned structure and improve performance.

[0101] Add regularization terms: During fine-tuning, you can add some regularization terms, such as L1 regularization or L2 regularization, to prevent overfitting. Regularization terms can constrain the parameters of the network to make it smoother and more stable. For example, you can add an L2 regularization term to the loss function, and its weight coefficient can be adjusted according to experimental results.

[0102] Adjust hyperparameters: According to the performance of the pruned network, adjust some hyperparameters, such as learning rate, batch size, etc. If the network performance decreases, you can appropriately reduce the learning rate or batch size to improve the stability of training. If the network performance is good, you can try to increase the learning rate or batch size to speed up the training.

[0103] 3. Carry out quantification processing, including the following steps:

[0104] (1) Determine the number of quantization bits: First, determine the appropriate number of quantization bits based on actual needs and hardware resource limitations. Generally speaking, the lower the number of quantization bits, the less storage space and computing resources are required, but this may result in a certain degree of precision loss. For example, 8-bit or 16-bit quantization can be selected, and for some devices with extremely limited resources, 4-bit or 2-bit quantization can even be considered. When determining the number of quantization bits, it is necessary to comprehensively consider the balance between network performance and resource requirements.

[0105] (2) Quantized weights: For the weights in the network, linear quantization or nonlinear quantization can be used. Linear quantization is to linearly map the weight value to a smaller range of values, for example, mapping the weight value originally between [-1,1] to the integer range of [0,255]. Nonlinear quantization can use more complex mapping functions according to the distribution characteristics of the weights to better retain important weight information. For example, a quantization method based on a logarithmic function can be used to give more quantization levels to larger weight values ​​and perform a coarser quantization for smaller weight values; specifically:

[0106] Linear quantization is used to reduce the weight parameters from the original range [ω min ,ω max ] is mapped to the target range [q min ,q max ], the formula is Or use nonlinear quantization, such as the quantization method based on logarithmic function Q(ω)=αlog(β|ω|+1) (α and β are determined according to weight distribution) to retain important weight parameters.

[0107] Quantize activation values: Quantization of activation values ​​can be done in a similar way to weight quantization. During training, the distribution of activation values ​​can be recorded, and then the appropriate quantization method can be selected based on the distribution characteristics. For example, if the distribution of activation values ​​is relatively concentrated, fewer quantization levels can be used; if the distribution is relatively dispersed, more quantization levels are required to ensure accuracy. At the same time, dynamic quantization methods can be used to dynamically adjust quantization parameters according to different input data to improve the adaptability of quantization; specifically:

[0108] Dynamically adjust the quantization parameters based on the distribution of activation values ​​or the input raw data. For example, according to the mean μ and standard deviation σ of the activation value distribution, the quantization interval is determined by μ±κσ (κ is an empirical constant). When the distribution is concentrated, fewer quantization levels are used, and when it is dispersed, more levels are used. The distribution of activation values ​​can also be recorded during the training process to select a suitable quantization method. At the same time, in order to ensure the performance of the network after quantization, quantization-aware training can be used. Each iteration first quantizes the weights and activation values ​​before forward propagation and back propagation calculations. Compensation values ​​can also be added to the quantized weights and activation values. The size and direction of the compensation values ​​are determined by analyzing the mean and variance of the quantization error. In addition, mixed precision quantization can be used for different layers or modules. Higher quantization precision is used for layers that have a great impact on performance, and lower precision is used for unimportant layers.

[0109] Output full-color imaging. The original image is input into the deep neural network model after network cropping and quantization for image enhancement. The network automatically adjusts parameters by learning a large amount of extremely weak light image data, improves image brightness, contrast and color saturation, and restores detail information. Then, a color conversion algorithm (such as a color conversion algorithm based on deep learning or a traditional RGB and grayscale conversion algorithm, etc.) is used to convert the enhanced grayscale image into a full-color image, ensuring that the color conversion is accurate and natural.

[0110] In order to ensure that the performance of the network does not drop significantly after quantization, the following quantization methods can be used: Quantization-aware training: During the network training process, quantization operations are introduced to allow the network to learn and adapt to the errors caused by quantization. Specifically, in each iteration, the weights and activation values ​​can be quantized first, and then forward propagation and back propagation calculations can be performed. In this way, the network can gradually adjust parameters to reduce the impact of quantization errors on performance. For example, in the first few epochs of training, the intensity of quantization can be gradually increased to allow the network to gradually adapt to quantization.

[0111] (3) Error compensation: After quantization, the quantization error can be compensated. A common method is to add a small compensation value to the quantized weights and activation values ​​to reduce the quantization error. This compensation value can be determined by analyzing the statistical characteristics of the quantization error. For example, the mean and variance of the quantization error can be calculated, and then the size and direction of the compensation value can be determined based on these statistical information.

[0112] (4) Mixed precision quantization: Different quantization precisions can be used for different layers or modules. For example, for layers that have a greater impact on performance, a higher quantization precision can be used, while for some relatively unimportant layers, a lower quantization precision can be used. In this way, the consumption of storage and computing resources can be minimized while ensuring performance.

[0113] 4. Image enhancement: The collected extremely weak light original image is input into the deep neural network after cropping and quantization for image enhancement. By learning a large amount of extremely weak light image data, the network can automatically adjust parameters to improve the brightness, contrast and color saturation of the image and restore the image details.

[0114] 5. Full-color imaging: Perform full-color processing on the enhanced image and use a suitable color conversion algorithm to convert the enhanced grayscale image into a full-color image. Ensure the accuracy and naturalness of the color conversion, making the generated full-color image more realistic.

[0115] Embodiment 2:

[0116] In order to verify the effectiveness of the method of the present invention, a low-light camera was used to collect extremely weak light original images in a night environment. The collected images were input into the constructed deep neural network, and after network cropping and quantization processing, image enhancement and full-color processing were performed. Through comparative experiments with traditional imaging methods, the advantages of the method of the present invention in improving image quality, reducing computational complexity and achieving full-color imaging were verified.

[0117] In the deep sea environment, an underwater camera was used to collect extremely weak light images. A large number of experiments were conducted to process the collected images using the method of the present invention, observe the image effects after processing, and evaluate the applicability and reliability of the method of the present invention in the deep sea environment. These image data cover a variety of different lighting conditions and object types and are highly representative.

[0118] The method of the present invention was compared with the traditional imaging method. The experimental results show that in extremely weak light environments, the method of the present invention can significantly improve the image quality and color reproduction while reducing the computational complexity. The specific data are as follows:

[0119] In terms of image quality, the image signal-to-noise ratio after processing by the method of the present invention is improved by an average of 8dB, and the image clarity is improved by an average of 30%; in terms of computational complexity, the computational time of the method of the present invention is reduced by an average of 40%, and the storage requirement is reduced by an average of 50%.

[0120] In addition, the network performance under different cropping ratios was tested. The experimental results show that under appropriate cropping ratios, the network performance will not be significantly reduced, and the computational complexity can be effectively reduced. The specific data are as follows:

[0121] When the cropping ratio is 30%, the network performance degradation is within an acceptable range, and a good balance is achieved between image quality and computational complexity.

[0122] As the cropping ratio further increases, the network performance gradually decreases, but the reduction in computational complexity is also more obvious. When the cropping ratio is 50%, the computational complexity is reduced by 60%, but the image quality decreases beyond the acceptable range.

[0123] In summary, the experimental data fully proves the effectiveness and feasibility of the method of the present invention, and has the following technical effects:

[0124] S1. Deep neural networks usually have a large number of parameters, which leads to high computational complexity and consumes a lot of computing resources and time. By pruning redundant neurons and connections, and quantizing weights and activation values, the number of network parameters and the amount of computation can be significantly reduced, the computational efficiency can be improved, and real-time requirements can be met. For example, on some resource-constrained devices, such as embedded systems and mobile devices, this optimization can enable extremely weak light full-color imaging methods to run faster and improve user experience.

[0125] S2. Large-scale deep neural networks require a lot of storage space to store model parameters. After pruning and quantization, the number of network parameters is reduced and the storage requirements are reduced accordingly. This is very important for devices with limited storage resources, which can save storage space and allow more image data to be stored and processed.

[0126] S3, the cropping and quantization process can remove some noise and redundant information in the network, making the model more concise and efficient. At the same time, the optimized network is more robust to noise and interference in extremely weak light environments, and can better process weak light signals and improve image quality. For example, in extremely weak light environments, the cropped and quantized network can more accurately extract image features and reduce the impact of noise on the image, thereby generating clearer and more realistic full-color images.

[0127] S4. Smaller-scale networks are easier to deploy on various devices, whether cloud servers or edge devices. Deep neural network cropping and quantization technology makes the extremely weak light full-color imaging method more widely applicable to different scenarios to meet the needs of different users. For example, in some special application scenarios, such as deep-sea exploration and field surveys, imaging equipment needs to be deployed in resource-constrained environments. At this time, the cropped and quantized network can better adapt to these environments and achieve high-quality full-color imaging.

[0128] The present invention also provides a device for implementing an extremely weak light imaging method based on deep neural network clipping and quantization technology, comprising:

[0129] Memory for storing computer programs;

[0130] A processor is used to implement the steps of an extremely weak light imaging method based on deep neural network clipping and quantization technology when executing the computer program.

[0131] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of an extremely weak light imaging method based on deep neural network clipping and quantization technology are implemented.

[0132] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A very weak light imaging method based on deep neural network clipping and quantization technology, characterized in that: The following steps are involved: Collect multiple images in extremely weak light environments to obtain original images; Using the deep neural network model after network clipping and quantization processing based on the evaluation result of neuron importance, the original image is enhanced and converted into full-color imaging; wherein the evaluation result of neuron importance is obtained based on the activation value distribution, gradient information and weight size of the neurons in the deep neural network model; The full-color image is output.

2. The extremely weak light imaging method based on deep neural network clipping and quantization technology according to claim 1 is characterized in that: The network pruning process includes: sorting the importance and connections of neurons based on the neuron importance evaluation results, marking neurons and connections whose importance is lower than a preset threshold as pruning candidates, and identifying and retaining the key layers for processing the original image and neurons whose importance is higher than another preset threshold through visualization tools or analysis of the intermediate output of the deep neural network model.

3. The extremely weak light imaging method based on deep neural network clipping and quantization technology according to claim 2 is characterized in that: The visualization tool includes: a feature map visualization tool or a gradient visualization tool; The visualization tool is used to display the response of neurons to features in the original image and the gradient changes of neurons in back propagation to assist in identifying key layers and important neurons.

4. The extremely weak light imaging method based on deep neural network clipping and quantization technology according to claim 2 is characterized in that: The key layers and neurons whose importance is higher than another preset threshold for processing the original image are retained, including: Set retention thresholds for the key layers and neurons with high importance in the deep neural network model that process the original image; After each cropping, the validation set is used to evaluate the performance of the cropped deep neural network model; Fine-tune the parameters of the deep neural network model based on the performance evaluation results.

5. The extremely weak light imaging method based on deep neural network clipping and quantization technology according to claim 4 is characterized in that: The method of fine-tuning the parameters of the deep neural network model according to the performance evaluation results includes: For the key layers of the original processing or the key layers that have a greater impact on performance in the deep neural network model, add regularization terms and adjust hyperparameters, and retrain.

6. The extremely weak light imaging method based on deep neural network clipping and quantization technology according to claim 1, characterized in that: The quantization process includes: quantizing weights and activation values; The weight is quantified as follows: Determine the number of quantization bits and use linear quantization to map the weight parameters to a smaller value range; Or use the distribution characteristics of weights by nonlinear quantization to retain important weight parameters through mapping functions; The activation value is quantified, specifically: The quantization parameter of the activation value is dynamically adjusted based on the activation value distribution or the input raw data.

7. The extremely weak light imaging method based on deep neural network clipping and quantization technology according to claim 6, characterized in that: The weight and activation value quantization further includes: The weights and activation values ​​are quantized first, and then forward propagation and backpropagation calculations are performed.

8. The extremely weak light imaging method based on deep neural network clipping and quantization technology according to claim 6, characterized in that: The weight and activation value quantization further includes: Add compensation values ​​to the quantized weights and activation values, specifically: By analyzing the mean and variance of the quantization error, the size and direction of the compensation value are determined and added to the quantized weights and activation values.

9. The extremely weak light imaging method based on deep neural network clipping and quantization technology according to claim 1, characterized in that: The multiple images in the extremely weak light environment are enhanced and converted into full-color images, specifically: The original image is input into a deep neural network model after network cropping and quantization processing to perform image enhancement, and the enhanced grayscale image is converted into a full-color image using a color conversion algorithm.

10. The extremely weak light imaging method based on deep neural network clipping and quantization technology according to claim 6, characterized in that: The calculation method of the mapping function is specifically as follows: Using the quantization method of the logarithmic function, the quantized result Q(ω) of the weight value ω satisfies Q(ω)=αlog(β|ω|+1); wherein α and β are constants determined according to the weight distribution.

Images