End-side lightweight infrared image denoising method and device, electronic equipment

Abstract

The present application relates to the technical field of infrared image processing, and provides an end-side lightweight infrared image denoising method and device and electronic equipment.The method comprises the following steps: acquiring an infrared image with noise to be processed; inputting the infrared image with noise into a target lightweight network trained in advance to output a predicted denoising image; the target lightweight network is obtained by optimization in a training stage based on a joint loss function; the joint loss function at least comprises a pixel-level reconstruction loss, a gradient domain sparse regularization loss and a vertical direction smoothing constraint loss; the gradient domain sparse regularization loss constrains a residual image between a reference image and the predicted denoising image, so that the residual image remains sparse in the gradient domain; the vertical direction smoothing constraint loss is used to calculate a difference value of adjacent pixels in the vertical direction of the predicted denoising image, and the difference value is punished to suppress horizontal stripe noise in the infrared image.The present application solves the contradiction between limited computing power and high-fidelity denoising.

Description

Technical Field

[0001] This invention relates to the field of infrared image processing technology, and in particular to an end-side lightweight infrared image denoising method, apparatus, and electronic device. Background Technology

[0002] As infrared imaging devices migrate to portable devices, drones, and other edge devices, the computing power and memory bottlenecks of on-chip systems equipped with AI accelerators on the edge are becoming increasingly prominent. The stripe noise unique to infrared images originates from the non-uniformity of the detector focal plane array and the thermal noise of the readout circuit, which exhibits strong global directionality and local periodicity.

[0003] Existing lightweight convolutional neural networks face insurmountable technical barriers when dealing with this type of noise. On the one hand, limited by edge computing power, lightweight networks have small receptive fields, making it difficult to capture the global periodic distribution of stripe noise, resulting in incomplete denoising. Although large-scale pre-trained visual foundation models possess powerful natural image structure priors and generalization capabilities, their massive parameter scale (typically in the hundreds of millions to tens of billions) and high computational cost prevent them from being directly deployed on resource-constrained edge devices. On the other hand, forcibly increasing network depth or introducing complex attention mechanisms can lead to inference latency that cannot meet the demands of real-time video streaming. More critically, the true target edges in infrared images, such as building outlines and human thermal radiation boundaries, often overlap with high-frequency stripe noise in their gradient features. Without large-scale, high-quality labeled data and strong structural prior guidance, lightweight networks are prone to erasing weak true edges when smoothing noise, resulting in over-smoothing of the image and severely impacting subsequent target detection accuracy. Summary of the Invention

[0004] This invention provides a lightweight edge-side infrared image denoising method, apparatus, and electronic device. It effectively injects the structural prior of a large-scale pre-trained model into the edge-side lightweight network, which can selectively remove directional stripe noise and protect the true edges, thus resolving the contradiction between limited computing power and high-fidelity denoising.

[0005] In a first aspect, embodiments of the present invention provide an end-side lightweight infrared image denoising method, comprising:

[0006] Acquire the noisy infrared image to be processed;

[0007] The noisy infrared image is input into a pre-trained lightweight target network, and a predicted denoised image is output.

[0008] The target lightweight network is optimized during the training phase based on a joint loss function, which includes at least pixel-level reconstruction loss, gradient domain sparse regularization loss, and vertical direction smoothing constraint loss. The gradient domain sparse regularization loss is used to constrain the residual image between the reference image and the predicted denoised image, so that the residual image remains sparse in the gradient domain to preserve the true edge details of the image. The vertical direction smoothing constraint loss is used to calculate the difference between adjacent pixels in the vertical direction of the predicted denoised image and to penalize the difference to suppress horizontal stripe noise in the infrared image.

[0009] In some embodiments, the calculation process of the gradient domain sparse regularization loss includes:

[0010] Calculate the residual image obtained by subtracting the reference image from the predicted denoised image;

[0011] The horizontal and vertical gradients of the residual image are extracted using edge detection operators, respectively.

[0012] The absolute values ​​of the horizontal gradient and the vertical gradient are summed.

[0013] In some embodiments, during the training phase, the target lightweight network is trained using a cross-model feature alignment architecture;

[0014] The cross-model feature alignment architecture includes a pre-trained prior network with frozen parameters and the target lightweight network to be optimized. The joint loss function also includes distillation loss, which is used to guide the target lightweight network to mimic the noise prior and feature distribution of the pre-trained prior network.

[0015] In some embodiments, the distillation loss includes feature layer distillation loss and response layer distillation loss. The feature layer distillation loss is used to match the feature map differences between the target lightweight network and the pre-trained prior network at the corresponding feature layer. The response layer distillation loss is used to calculate the relative entropy between the output distribution of the pre-trained prior network and the output distribution of the target lightweight network.

[0016] The output distribution undergoes a softening process with a preset temperature coefficient before the relative entropy is calculated.

[0017] In some embodiments, the calculation of the feature layer distillation loss further includes:

[0018] When the feature map dimensions of the target lightweight network and the pre-trained prior network do not match, the feature map of the pre-trained prior network is mapped to the same feature space as the target lightweight network through a projection layer with preset weight parameters. Then, the mean square error between the mapped feature map of the pre-trained prior network and the feature map of the target lightweight network is calculated and used as the feature map difference of the feature layer.

[0019] In some embodiments, the target lightweight network integrates a multi-scale feature perception module;

[0020] The multi-scale feature perception module includes multiple parallel convolutional branches, each of which uses a convolutional kernel with a different dilation rate to capture noise features of different granularities. The feature maps output by each convolutional branch are spliced ​​and fused along the channel dimension.

[0021] In some embodiments, the multi-scale feature perception module includes at least three parallel convolutional branches;

[0022] The first convolutional branch uses a baseline convolutional kernel with a first dilation rate to extract local details, the second convolutional branch uses a dilated convolutional kernel with a second dilation rate, and the third convolutional branch uses a dilated convolutional kernel with a third dilation rate, wherein the third dilation rate is greater than the second dilation rate.

[0023] In some embodiments, each convolutional layer of the target lightweight network employs depthwise separable convolution, and the target lightweight network integrates a lightweight attention module that combines spatial and channel-based approaches.

[0024] Secondly, embodiments of the present invention also provide an end-side lightweight infrared image denoising device, comprising:

[0025] The image acquisition module is used to acquire a noisy infrared image to be processed.

[0026] A lightweight training module is used to input the noisy infrared image into a pre-trained target lightweight network and output a predicted denoised image.

[0027] The target lightweight network is optimized during the training phase based on a joint loss function, which includes at least pixel-level reconstruction loss, gradient domain sparse regularization loss, and vertical direction smoothing constraint loss. The gradient domain sparse regularization loss is used to constrain the residual image between the reference image and the predicted denoised image, so that the residual image remains sparse in the gradient domain to preserve the true edge details of the image. The vertical direction smoothing constraint loss is used to calculate the difference between adjacent pixels in the vertical direction of the predicted denoised image and to penalize the difference to suppress horizontal stripe noise in the infrared image.

[0028] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the end-side lightweight infrared image denoising method as described in the first aspect.

[0029] This invention achieves a mathematical balance between fidelity preservation and denoising by guiding a joint loss function. Pixel-level reconstruction loss ensures that the macroscopic image structure remains undistorted and serves as a fundamental fidelity preservation term. Gradient domain sparse regularization loss, by strictly constraining the residual image to contain no edge information, mathematically eliminates the possibility of the network smoothing the contours of real objects, thus achieving edge preservation. Since the stripe noise unique to infrared images is mostly horizontally distributed, and these horizontal stripes cause high-frequency abrupt changes in the vertical direction, the vertical smoothing constraint loss specifically penalizes these vertical abrupt changes, accurately eliminating horizontal stripes while barely affecting the real edges extending horizontally. Therefore, the synergy of these three factors ultimately achieves a high-fidelity denoising effect that removes only stripes without compromising the real contours. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0031] Figure 1 This is a schematic flowchart of a lightweight infrared image denoising method provided in an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram illustrating the cross-model feature alignment architecture principle of a lightweight infrared image denoising method for the edge side provided in an embodiment of the present invention.

[0033] Figure 3 This is a schematic diagram of the internal structure of a multi-scale feature perception module provided in an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the data flow for calculating gradient domain sparse regularization loss according to an embodiment of the present invention;

[0035] Figure 5 This is a hardware timing diagram of an electronic device performing inference operations according to an embodiment of the present invention;

[0036] Figure 6This is a schematic diagram of the delay performance test results of an edge-side lightweight infrared image denoising method provided in an embodiment of the present invention during edge-side inference;

[0037] Figure 7 This is a schematic diagram showing the comparison test results of the peak signal-to-noise ratio denoising effect between a target lightweight network and a baseline network provided in an embodiment of the present invention;

[0038] Figure 8 This is a schematic diagram illustrating the edge gradient response enhancement effect of a lightweight attention module combining space and channel in a target lightweight network provided by an embodiment of the present invention;

[0039] Figure 9 This is a schematic diagram of the structure of a lightweight infrared image denoising device provided in an embodiment of the present invention;

[0040] Figure 10 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0042] As infrared imaging devices migrate to portable devices, drones, and other edge computing environments, the computational and memory bottlenecks of edge processors, such as on-chip systems with AI computing capabilities, are becoming increasingly prominent. The unique stripe noise in infrared images originates from the non-uniformity of the detector's focal plane array and the thermal noise of the readout circuitry, exhibiting strong global directionality, such as horizontal lateral and local periodicity. Existing lightweight convolutional neural networks face insurmountable technical challenges when processing such infrared images. On the one hand, limited by edge computing power, the receptive field of lightweight networks is typically small, making it difficult to capture the global periodic distribution characteristics of stripe noise, resulting in incomplete removal of stripe noise at specific frequencies. On the other hand, the edges of real targets in infrared images, such as building outlines and human thermal radiation boundaries, often heavily overlap with high-frequency stripe noise in terms of gradient features. Traditional denoising methods, lacking strong structural prior guidance and effective physical constraints, are prone to mistakenly erasing weak real edges as noise during noise smoothing, leading to over-smoothing of the image and severely impacting the accuracy of subsequent target detection and recognition. In recent years, although large-scale pre-trained image models with tens of billions of parameters have emerged, possessing powerful natural image structure priors and generalization capabilities, their massive parameter scale and high computational overhead make them unsuitable for direct deployment on resource-constrained edge electronic devices.

[0043] In summary, existing technologies suffer from the following technical problems and shortcomings: First, the large number of network parameters leads to high computational costs. Existing deep learning denoising models have deep layers and numerous channels, typically with tens of millions of parameters, making them unsuitable for real-time operation on edge AI accelerator systems with limited computing power, resulting in low frame rates and high power consumption. Second, single-scale feature extraction leads to incomplete noise removal. Existing methods often use convolutional kernels with fixed receptive fields, making it difficult to simultaneously capture high-frequency noise and low-frequency background information, resulting in residual noise at specific frequencies. Third, feature redundancy leads to poor detail preservation. Existing network feature maps contain a large number of redundant responses and lack constraints on sparsity, making it easy to misjudge weak edges as noise during denoising and smooth them, causing image blurring. Fourth, large-scale model deployment on edge is difficult due to significant differences in parameter scale. Existing high-performance image denoising models have up to ten billion parameters and occupy tens of gigabytes of storage, making them unsuitable for operation on edge AI accelerator systems with limited computing power. There is a lack of effective knowledge distillation mechanisms to compress the capabilities of ten billion-level models to tens of millions of parameters while maintaining performance. Therefore, existing technologies have consistently failed to resolve the contradiction between limited computing power and high-fidelity noise reduction. How to establish a cross-model feature transfer mechanism, effectively inject the structural priors of large-scale pre-trained models into the lightweight edge network, and at the same time, selectively remove directional stripe noise and strongly protect the real edges at the physical and mathematical levels, is a technical problem that urgently needs to be solved in this field.

[0044] To address the aforementioned technical challenges, this invention provides a lightweight edge-side infrared image denoising method for resource-constrained embedded devices. Belonging to the field of edge computing applications utilizing large-scale pre-trained models for knowledge transfer, this method is primarily applied to resource-constrained embedded devices such as portable infrared thermal imagers, UAV infrared payloads, and handheld night vision devices. It can remove stripe noise and Gaussian noise from infrared thermal images in real time on the device side, while accurately preserving image edge details, effectively improving the accuracy of subsequent target detection and recognition. Specifically, this invention deploys a lightweight neural network, after large-model knowledge distillation and compression, on an edge-side AI accelerator on-chip system, replacing traditional manual algorithms or heavy-duty deep learning networks to achieve low-power, high-fidelity denoising processing of edge-side infrared images. Furthermore, the algorithm is provided as a dynamic link library or static library, supporting integration with the device-side image processing pipeline via C or C++ application programming interfaces or Python. This algorithm supports seamless integration with the raw image to luminance, chrominance, and RGB color space pipelines of mainstream image signal processing chips and can be configured as a pre-denoising module or a post-enhancement module. Furthermore, the memory usage of this algorithm can be dynamically adjusted when deployed on the device side, with the minimum frame buffer required to process one frame being 12 megabytes.

[0045] It should be noted that the embodiments of the present invention have constructed a rigorous collaborative mechanism for the characteristics of infrared physical noise. The technical features are interdependent, making up for the new defects introduced by each other, and jointly achieving high-fidelity infrared noise reduction under resource-constrained conditions. Its core advantages and beneficial effects are reflected in:

[0046] First, infrared edge devices have extremely low computing power and memory, making it impossible to accommodate large models. Traditional small models also produce poor denoising results. This invention uses a target lightweight network as the carrier to meet hardware limitations. However, ordinary lightweight networks are too weak. Therefore, a distillation loss under a cross-model feature alignment architecture is introduced, allowing the pre-trained prior network to transfer advanced denoising experience to the target lightweight network, resolving the contradiction between model size and denoising intelligence. Second, although the target lightweight network undergoes distillation training, the stripe noise in infrared images varies in length and thickness. Traditional convolutional layers can only identify noise of a specific width, resulting in incomplete denoising. This invention embeds a multi-scale feature perception module within the target lightweight network, providing richer and more comprehensive underlying raw materials. The distillation loss requires the multi-scale feature perception module to detect noise of all sizes before it can be effective, jointly solving the problem of identifying noise with varying shapes.

[0047] Third, when the network strives to smooth out noise of various sizes, misprocessing is inevitable. This means that the original, realistic building outlines and human body edges are also treated as noise and smoothed out, resulting in a blurry image. In the training phase, this embodiment introduces gradient domain sparse regularization loss. For the residual image generated by the network, it limits the network from extracting various information at multiple scales in the early stages, but ensures that high-frequency edges are preserved when generating the final image. This ensures that the image fidelity is not compromised during intensive denoising. Fourth, although a very strong image denoising system has been constructed, considering the physical limitations of infrared detectors, namely the tendency to generate horizontal stripe noise, this embodiment adds a vertical smoothing constraint loss. This loss works in conjunction with the multi-scale feature perception module, which is responsible for finding stripes, while the vertical smoothing constraint loss handles horizontal stripes. Simultaneously, it works in conjunction with the gradient domain sparse regularization loss. The gradient domain sparse regularization loss protects contour edges in all directions, while the vertical smoothing constraint loss, because it only constrains smoothness in the vertical direction, avoids destroying true horizontal edges.

[0048] Figure 1 This is a schematic flowchart of an end-side lightweight infrared image denoising method provided in an embodiment of the present invention. The end-side lightweight infrared image denoising method can be executed by the end-side lightweight infrared image denoising device provided in this embodiment of the present invention, which can be implemented in software and / or hardware. For example... Figure 1 As shown, the end-side lightweight infrared image denoising method includes the following steps:

[0049] S101. Acquire the noisy infrared image to be processed.

[0050] S102. Input the noisy infrared image into the pre-trained target lightweight network and output the predicted denoised image.

[0051] The target lightweight network is optimized during the training phase based on a joint loss function, which includes at least pixel-level reconstruction loss, gradient domain sparse regularization loss, and vertical direction smoothing constraint loss. The gradient domain sparse regularization loss is used to constrain the residual image between the reference image and the predicted denoised image, so that the residual image remains sparse in the gradient domain to preserve the true edge details of the image. The vertical direction smoothing constraint loss is used to calculate the difference between adjacent pixels in the vertical direction of the predicted denoised image and to penalize the difference to suppress horizontal stripe noise in the infrared image.

[0052] Specifically, edge-based lightweight infrared image denoising methods refer to a combination of processing steps designed to eliminate various noise signals in infrared images, operating on edge embedded devices with limited computing power and storage space. A noisy infrared image refers to the raw input data acquired by an infrared thermal imaging sensor, containing thermal noise, shot noise, and stripe noise caused by sensor non-uniformity. A target lightweight network refers to a deep neural network, designed with simplified parameters specifically adapted to the computing power of edge chips, responsible for performing the actual denoising inference. The predicted denoised image refers to the image output by the target lightweight network after forward computation, after which the noise components have been removed. The joint loss function refers to the comprehensive optimization objective, composed of a weighted combination of multiple mathematical penalty terms, used during the training phase to evaluate the quality of the network output. Pixel-level reconstruction loss refers to the fundamental mathematical error term that calculates the difference between the predicted denoised image and the real clear image pixel by pixel. Gradient domain sparse regularization loss refers to the penalty term that constrains the model behavior by limiting the gradient of the image difference matrix to approach zero, i.e., sparsification. Vertical smoothing constraint loss refers to a penalty term that specifically evaluates and suppresses the degree of abrupt changes in adjacent pixel values ​​along a column direction of the image. The reference image refers to the real, noise-free, high-resolution infrared image used as the standard answer during the training phase. The residual image is the difference matrix obtained by subtracting the reference image from the predicted denoised image at corresponding pixel positions.

[0053] During the deployment phase, the first step is to acquire the noisy infrared image to be processed, reading the raw data from the infrared detector or storage medium. Then, the noisy infrared image is input into a pre-trained target lightweight network, and a predicted denoised image is output. The acquired image data is then fed into the target lightweight network, which is already loaded into memory, for forward inference computation. The target lightweight network possesses this capability because its model parameters are rigorously optimized using a joint loss function during the training phase. Specifically, after each forward propagation during training, the system calculates multiple loss terms. First, a pixel-level reconstruction loss is calculated, which ensures the basic energy conservation and restoration capability of the predicted denoised image and the reference image at the pixel brightness level, preventing overall image distortion. Second, a gradient domain sparsity regularization loss is calculated. This loss works by subtracting the reference image from the predicted denoised image to obtain a residual image. If the network incorrectly erases the true edges of the image during denoising, the residual image will retain the contour information of these edges, indicating drastic gradient changes. By forcing the residual image to remain sparse in the gradient domain—that is, forcing the gradient value to be zero in most regions of the residual image—the system effectively forces the network to preserve the true edges intact in the predicted denoised image. Simultaneously, the system calculates a vertical smoothing constraint loss. This loss term works by calculating the difference between two adjacent pixels in the same column of the predicted denoised image and penalizing this difference, i.e., the vertical gradient. This loss term forcibly suppresses drastic pixel fluctuations in the vertical direction of the image.

[0054] Therefore, this embodiment of the invention achieves a mathematical balance between fidelity preservation and denoising through the guidance of a joint loss function. Pixel-level reconstruction loss ensures that the macroscopic image structure remains undistorted and serves as a fundamental fidelity preservation term. Gradient domain sparse regularization loss, by strictly constraining the residual image to contain no edge information, mathematically eliminates the possibility of the network smoothing the contours of real objects, thus achieving edge preservation. Since the stripe noise unique to infrared images is mostly horizontally distributed, and these horizontal stripes cause high-frequency abnormal jumps in the vertical direction, the vertical smoothing constraint loss specifically penalizes these vertical jumps, accurately eliminating horizontal stripes while barely affecting the real edges extending horizontally. Thus, the synergy of these three factors ultimately achieves a high-fidelity denoising effect that removes only stripes without compromising the real contours.

[0055] For example, the derivation of the joint loss function and the optimization of hyperparameters can be as follows: Let the reference image be... That is, a true, noise-free, high-definition infrared image, used as the standard answer during training, to predict the denoised image. This refers to the denoising result output by the target lightweight network. The overall expression for the joint loss function is: Among them, mission losses The pixel-level reconstruction loss uses L1 loss, and the calculation formula is as follows: This is used to ensure basic energy conservation and noise reduction capabilities. The formula for calculating the gradient domain sparse regularization loss is: ,in This represents the gradient operator, specifically using the Sobel operator to calculate the horizontal gradient. and vertical gradient This loss term is used to extract edge contour information from the residual image, and it forces the residual image to remain sparse in the gradient domain. The formula for calculating the vertical smoothing constraint loss is as follows: Where the subscript i represents the row coordinate of the image, and j represents the column coordinate of the image. This represents the predicted pixel value located in the i-th row and j-th column of the denoised image. This represents the pixel value of the pixel directly above it. Subtracting the two values ​​yields the pixel difference in the vertical direction. This loss term forces the vertical difference to approximate the true value, thereby suppressing horizontal stripe noise.

[0056] Distillation loss includes feature layer distillation loss and response layer distillation loss. Specifically, distillation loss is... ,in and These are the weighting coefficients for the feature layer distillation loss and the response layer distillation loss, respectively. These represent the feature maps output by the pre-trained prior network and the target lightweight network at the i-th layer, respectively. This represents the sum of squares, i.e., the pre-sigma of the mean square error. Relative entropy is used to measure the difference between two probability distributions. Here, represents the original output tensors of the teacher network and the student network, respectively; softmax represents the normalized exponential function; and T is the preset temperature coefficient. The recommended hyperparameter range for the ablation experiment is: weight of distillation loss. The weights of the gradient domain sparse regularization loss Weights of vertical smoothing constraint loss preset temperature coefficient .

[0057] In some embodiments, the calculation process of gradient domain sparse regularization loss includes: calculating the residual image obtained by subtracting the reference image from the predicted denoised image; extracting the horizontal and vertical gradients of the residual image using the edge detection operator; and summing the absolute values ​​of the horizontal and vertical gradients.

[0058] Specifically, edge detection operators refer to mathematical convolution matrices used to extract regions of abrupt change in pixel values ​​in an image, i.e., edges, such as the Sobel operator. Horizontal gradient refers to the rate of change of grayscale values ​​between adjacent pixels in the horizontal direction (X-axis). Vertical gradient refers to the rate of change of grayscale values ​​between adjacent pixels in the vertical direction (Y-axis). The sum of absolute values ​​is a mathematical operation that involves taking non-negative values ​​from all extracted gradient values ​​and accumulating them; it is equivalent to the L1 norm.

[0059] During the training phase, a residual image is calculated by subtracting the predicted denoised image from the reference image. The reference image, serving as the ground truth, is subtracted pixel-by-pixel from the currently generated predicted denoised image, resulting in a residual image containing residual noise and model errors. Next, the horizontal and vertical gradients of the residual image are extracted using edge detection operators. Sliding convolution operations are then performed on the residual image along the horizontal and vertical directions using edge detection convolution kernels such as the Sobel operator, calculating the horizontal and vertical gradient matrices. Finally, the absolute values ​​of the horizontal and vertical gradients are summed. The absolute values ​​of all elements in the two gradient matrices are calculated and accumulated to obtain a scalar value, which is the current gradient domain sparse regularization loss value.

[0060] Therefore, this invention provides a precise and differentiable edge protection supervision mechanism that can be used for backpropagation training. The residual image obtained through subtraction effectively removes the static background of the image, highlighting the location of errors. Using the edge detection operator to calculate the gradient amplifies the potential high-frequency edge information in the residual image, making it easier to capture small errors. Finally, the sum of absolute values, i.e., the L1 norm, is calculated. Leveraging the inherent mathematical property of the L1 norm in optimization theory—that it promotes sparsity of the solution vector, i.e., approximates zero—this operation imposes a severe linear penalty on any non-zero gradient value in the residual image, thereby efficiently preventing the network parameters from updating towards smooth, true edges during error backpropagation.

[0061] In some embodiments, during the training phase, the target lightweight network is trained using a cross-model feature alignment architecture; wherein the cross-model feature alignment architecture includes a pre-trained prior network with frozen parameters and the target lightweight network to be optimized, and the joint loss function also includes distillation loss, which is used to guide the target lightweight network to mimic the noisy prior and feature distribution of the pre-trained prior network.

[0062] Specifically, cross-model feature alignment architecture refers to a teacher-student structured machine learning framework used to transfer complex feature representations extracted from large models to smaller models. A pre-trained prior network with frozen parameters refers to a massive deep neural network model that has been pre-trained on a large dataset and whose internal weight parameters are not updated during training. A target lightweight network to be optimized refers to a small receiving model whose weight parameters need to be updated via backpropagation according to the loss function during the current training process to continuously learn. Distillation loss is a penalty term used to measure the knowledge gap in output or features between the target lightweight network and the pre-trained prior network. Noise prior and feature distribution refer to the high-order mathematical statistical laws regarding what constitutes noise and what constitutes clean image features, formed by large-scale networks after long-term learning from massive amounts of data.

[0063] During system training, the same noisy infrared image is simultaneously and independently input into both a pre-trained prior network with frozen parameters and a target lightweight network to be optimized. The pre-trained prior network, acting as a fixed and high-performance expert system, stably outputs its high-order understanding of the noise features and semantic information in the input image, namely its internal feature maps and final output. The target lightweight network to be optimized outputs its preliminary processing results under its current parameters. The system calculates the mathematical difference between the two, which could be intermediate layer feature maps or the final output, to generate a distillation loss, which is then added to the joint loss function. Therefore, when optimizing its own parameters, the target lightweight network's optimization objective is no longer merely to reduce its own denoising error, such as pixel-level reconstruction loss, but also to simultaneously strive to make its internal state and output results mimic and approximate the high-performance pre-trained prior network.

[0064] Therefore, this invention overcomes the physical bottleneck of limited expressive power caused by the small number of parameters in lightweight models. Since the pre-trained prior network with frozen parameters possesses hundreds of millions of parameters, the extracted noise priors and feature distributions far exceed the limits achievable by lightweight networks with their limited parameters. By establishing a cross-model feature alignment architecture and applying distillation loss, the target lightweight network does not need to explore complex denoising rules from scratch. Instead, it directly uses the feature distribution of the pre-trained prior network as the learning target and shortcut. This allows the final trained target lightweight network to possess high-quality denoising capabilities similar to its teacher network while retaining only a very low number of parameters, thus breaking the deadlock between edge computing power and model performance.

[0065] In some embodiments, the distillation loss includes feature layer distillation loss and response layer distillation loss. The feature layer distillation loss is used to match the feature map differences between the target lightweight network and the pre-trained prior network at the corresponding feature layer. The response layer distillation loss is used to calculate the relative entropy between the output distribution of the pre-trained prior network and the output distribution of the target lightweight network. The output distribution is softened by a preset temperature coefficient before calculating the relative entropy.

[0066] Specifically, the feature layer distillation loss refers to the error term that constrains the similarity of the multidimensional matrix output by the hidden layers of the network, i.e., the feature maps. The response layer distillation loss refers to the error term that constrains the similarity of the probability distribution results at the final output of the network. Feature map difference refers to the mathematical distance between the multidimensional feature matrices extracted by two networks at structurally similar intermediate layers. Relative entropy, also known as KL divergence, is a statistic used to measure the degree of asymmetric difference between two probability distribution functions. Output distribution refers to the set of probabilities that an image belongs to various patterns, given by the last layer of the network through a nonlinear mapping, such as a normalized exponential function. Preset temperature coefficient refers to a scalar constant greater than 1 introduced into the normalized exponential function to smooth the probability distribution curve. Softening processing refers to the operation of smoothing out the originally sharp probability distribution with large numerical differences by dividing by the preset temperature coefficient.

[0067] During training, the system performs feature layer distillation, which involves extracting feature maps from multiple intermediate layers of the target lightweight network and the pre-trained prior network, calculating the feature layer distillation loss between them, and forcing the intermediate inference process of the smaller network—that is, the feature extraction method—to be consistent with that of the larger network. Simultaneously, the system performs response layer distillation, which extracts the outputs of both networks at the final output. Before calculating the difference, the system performs a softening process, dividing both output values ​​by a preset temperature coefficient greater than 1, such as 5. This flattens and smooths out the sharp output probabilities that might otherwise approach 1 or 0, thereby exposing previously masked, secondary probabilistic information. Subsequently, using relative entropy, i.e., the KL divergence formula, the divergence value between the two softened output distributions is calculated, forming the response layer distillation loss.

[0068] Therefore, this invention achieves deep knowledge transfer from a large model to a small model and significantly improves the generalization robustness of the lightweight model. Feature layer distillation ensures that the student network is closely aligned with the teacher network in extracting basic features such as image texture and edges. Furthermore, the softening process of the response layer is crucial, as it ensures that the teacher network no longer transmits absolute right or wrong statements, but rather complex similarity relationships between different pixels or categories. By forcing the lightweight network to learn this flexible probability distribution rich in relational information through relative entropy, the student network can mimic the judgment logic of the teacher network when faced with complex striped noise it has never seen before, possessing the flexibility and strong generalization ability of a large model.

[0069] For example, during the training phase, when the feature map dimensions of the target lightweight network and the pre-trained prior network do not match, a projection layer with preset weight parameters is used to map the feature map of the pre-trained prior network to the same feature space as the target lightweight network. Specifically, a projection layer with a size of [missing information] is used. Learnable projection layer Feature maps of pre-trained prior networks Map the feature map to the feature space of the target lightweight network, and compute the mapped feature map and the feature map of the target lightweight network. The mean square error between them, as the characteristic layer distillation loss, is calculated using the following formula: This avoids high-frequency prior ambiguity caused by using bilinear interpolation for feature map size alignment. During the training phase, a course learning strategy is employed to gradually improve the model's generalization robustness. Initially, only Gaussian noise data with a signal-to-noise ratio (SNR) of 25 dB is used to stabilize the basic denoising capability; in the middle stage of training, mixed stripe noise and Gaussian noise are injected, with an SNR of 15 to 20 dB; in the later stage of training, low-quality infrared sequences collected by real equipment are introduced, with an SNR below 15 dB. The optimizer is configured as a variant of the adaptive moment estimation optimizer, and the hyperparameters are set to... Initial learning rate Cosine annealing learning rate scheduling is used, with weight decay at a certain value. The batch size is 16, and the training takes 150 iterations.

[0070] In some embodiments, when calculating the feature layer distillation loss, the method further includes: when the feature map dimensions of the target lightweight network and the pre-trained prior network do not match, mapping the feature map of the pre-trained prior network to the same feature space as the target lightweight network through a projection layer with preset weight parameters, calculating the mean square error between the mapped feature map of the pre-trained prior network and the feature map of the target lightweight network, and using it as the feature map difference at the feature layer level.

[0071] Specifically, feature map dimensionality mismatch refers to the inconsistency in shape and size (e.g., length, width, number of channels) of the extracted feature matrices at the desired distillation alignment layer due to differences in the number of channels or resolution design between the pre-trained prior network and the target lightweight network. A projection layer with preset weight parameters refers to a linear or non-linear transformation module containing learnable convolutional parameters used to change the dimension of the input feature tensor. Identical feature spaces refer to mathematical dimensional spaces with identical height, width, and number of channels. Mean squared error refers to the average of the squared differences between corresponding elements of two matrices.

[0072] When the system prepares to calculate the feature layer distillation loss, it first checks the feature map dimensions of the target lightweight network and the pre-trained prior network at the selected layer. If a dimensional mismatch is found, for example, the pre-trained prior network's feature map has 256 channels while the target lightweight network's has 64, the system will not perform the calculation directly but will instead perform an alignment operation. Specifically, the feature map of the pre-trained prior network is input into a projection layer with preset weight parameters. This projection layer, for example, a 64-channel 1x1 convolutional layer, compresses the number of channels of the input feature map from 256 to 64 through matrix multiplication, thus mapping it to the same feature space as the target lightweight network. After the mapping is completed, the two feature maps are completely identical in height, width, and number of channels. Subsequently, the system calculates the mean squared error of these two dimensionally aligned feature maps and uses this error value as the feature map difference for the current layer.

[0073] Therefore, this invention addresses the dimensionality difference problem during knowledge transfer between heterogeneous networks, avoiding the loss of structural prior information. Directly calculating the difference in dimensions-mismatched matrices is mathematically ineffective. If traditional bilinear interpolation or hard pooling is used to forcibly compress feature maps to the same size, the high-frequency, detailed noise prior information contained in the pre-trained prior network will inevitably be blurred or lost. This invention introduces an independent, learnable projection layer with preset weight parameters, utilizing the linear combination capability of 1x1 convolutions to achieve efficient dimensionality reduction and recombination of feature channel information. This not only ensures absolute alignment of the two feature maps in mathematical dimensions when calculating the mean squared error, but more importantly, the projection layer, through training, learns how to retain the important information in the original teacher features to the maximum extent, ensuring that teacher knowledge is losslessly measured and transferred to the student network.

[0074] In some embodiments, the target lightweight network integrates a multi-scale feature perception module; wherein, the multi-scale feature perception module includes multiple convolutional branches set in parallel, each convolutional branch adopts a convolutional kernel with a different dilation rate to capture noise features of different granularities, and the feature maps output by each convolutional branch are spliced ​​and fused along the channel dimension.

[0075] Specifically, the multi-scale feature perception module refers to a composite component specially designed within the target lightweight network, used to acquire image information with different receptive fields (i.e., the breadth of the input spatial range) in the same processing stage. Parallel convolutional branches refer to multiple independent convolutional processing channels that are copied and fed into each other simultaneously as the data stream passes through a specific node. Convolutional kernels with different dilation rates refer to filters that sample elements of the convolutional kernel matrix at different intervals (i.e., holes) during convolution operations; the larger the dilation rate, the wider the effective receptive field of the filter. Noise features of different granularities refer to noise patterns of different scales (from extremely small random snowflake-like noise to coarse stripe noise spanning the entire image). Channel-dimensional stitching and fusion refers to merging multiple feature matrices with the same height and width calculated by multiple parallel convolutional branches, like stacking cards, along their channel depth direction to form a thicker, more information-rich comprehensive feature matrix.

[0076] During the forward inference process of the target lightweight network, when image features flow to the multi-scale feature perception module, the data is simultaneously distributed to multiple parallel convolutional branches within it. Each branch is configured with convolutional kernels with different dilation rates. Due to the different dilation rates, these convolutional kernels examine the same regions on the input feature map at the same time, but their effective receptive fields are different. Specifically, branches with small dilation rates focus on local details, while branches with large dilation rates span a wider pixel region to obtain a larger range of contextual information. After each branch completes its parallel computation, it outputs a feature map. Finally, the system performs a concatenation and fusion operation along the channel dimension, using a tensor concatenation function to merge these feature maps into a single, thick feature map along its channel dimension, and then passes this fused feature map rich in multi-scale information to the next processing stage of the network.

[0077] Therefore, this invention overcomes the shortcomings of lightweight networks in terms of incomplete noise removal due to a single receptive field. Infrared stripe noise, due to its complex physical causes, manifests in images in various forms with varying lengths and thicknesses. Traditional single convolutional branches can only extract features of a fixed width, leading to the omission of certain frequencies, especially coarse stripe noise. The multi-scale feature perception module of this invention greatly enriches the network's receptive field range without significantly increasing the number of model parameters by using dilated convolutions with different dilation rates in parallel. The splicing and fusion operation ensures that the micro, meso, and macro-level noise features captured by different branches are uniformly integrated in the channel dimension, enabling the network to obtain omnidirectional noise detection capabilities like compound eyes in a single forward computation, thereby completely eliminating noise residue of various forms.

[0078] In some embodiments, the multi-scale feature perception module includes at least three parallel convolutional branches; wherein, the first convolutional branch uses a baseline convolutional kernel with a first dilation rate to extract local details, the second convolutional branch uses a dilated convolutional kernel with a second dilation rate, and the third convolutional branch uses a dilated convolutional kernel with a third dilation rate, wherein the third dilation rate is greater than the second dilation rate.

[0079] Specifically, the three parallel convolutional branches refer to three independent, simultaneously computed convolutional processing paths instantiated within the multi-scale feature perception module. The first dilation rate baseline convolutional kernel refers to a standard convolutional filter with a dilation rate of the minimum baseline value, such as 1, meaning there are no holes between kernel elements. Extracting local details refers to using convolutional kernels with small receptive fields to capture the most subtle high-frequency abrupt changes in the image, such as random noise. The second and third dilation rate dilated convolutional kernels refer to dilated filters with skip intervals between kernel elements, and the skip interval of the third dilation rate is wider than that of the second dilation rate, thus having a larger effective receptive field.

[0080] Within the module, input data is simultaneously fed into these three parallel convolutional branches. In the first convolutional branch, the data undergoes a dense, gapless scan using a baseline convolutional kernel with a first dilation rate, such as a 3x3 kernel with a dilation rate of 1, focusing on capturing closely connected local details. In the second convolutional branch, the data is sampled across pixels with a stride of 2 using a dilated convolutional kernel with a second dilation rate, such as a 3x3 kernel with a dilation rate of 2, thus covering a medium-sized context region. In the third convolutional branch, the data undergoes a larger-span sampling scan using a dilated convolutional kernel with a third dilation rate, such as a 3x3 kernel with a dilation rate of 3, with the largest receptive field, used to capture wide stripes or global background information. Within the same computation cycle, the three branches perform feature extraction at different granularities, from microscopic to mesoscopic to macroscopic.

[0081] Therefore, this embodiment of the invention accurately adapts to the span characteristics of infrared physical noise, maximizes feature extraction efficiency, and maintains hardware friendliness. By setting three incremental dilation rates in a stepwise manner, the system constructs a continuous receptive field. The first convolutional branch is responsible for capturing fine local details such as Gaussian noise, the second convolutional branch is responsible for capturing medium-width stripes, and the effective receptive field of the third convolutional branch is multiplied, for example, from 3x3 to 7x7, which can cover coarse stripes and large areas of non-uniform background gradients. Thus, this parallel design with clear division of labor allows the network to capture a wide range of noise patterns in a single forward propagation, while the use of dilated convolutions avoids the problem of severe loss of spatial location information caused by downsampling pooling operations in traditional multi-scale designs.

[0082] For example, the receptive field calculation of the multi-scale feature perception module can be: let the size of the dilated convolution kernel used in the multi-scale feature perception module be... expansion rate ,in =1 corresponds to the first expansion rate, d=2 corresponds to the second expansion rate, and d=3 corresponds to the third expansion rate, with the third expansion rate being greater than the second expansion rate. Effective receptive field This corresponds to the set of receptive fields. That is, the three parallel convolutional branches have effective receptive fields of 3×3, 5×5, and 7×7, respectively. The feature maps output by the three parallel convolutional branches of the multi-scale feature perception module are concatenated and fused along the channel dimension, and then... Convolution performs channel reduction, and the fusion formula is as follows: Where F is the input feature map, and [;;] represents the concatenation operation along the channel dimension. This represents a 1×1 convolution, used for dimensionality reduction to remove redundant information. This represents the final feature map after multi-scale fusion. This design allows the network to capture features in a single forward pass. to The span noise pattern avoids the loss of spatial location information caused by multi-scale pooling.

[0083] In some embodiments, each convolutional layer of the target lightweight network employs depthwise separable convolution, and the target lightweight network integrates a lightweight attention module that combines spatial and channel-based approaches.

[0084] Specifically, each convolutional layer refers to all the main computational layers responsible for feature mapping that form the basic framework of the target lightweight network. Depthwise separable convolution refers to an algorithmic structure that decomposes the traditional standard 3D convolution operation into two independent steps: the first step is depthwise convolution, which independently performs 2D spatial convolution on each channel of the input; the second step is pointwise convolution, which uses a 1x1 convolution kernel to linearly combine the outputs of the depthwise convolution along the channel direction to generate the final output. The lightweight attention module combining spatial and channel dimensions refers to a plug-in module that uses very few additional parameters. This module can sequentially calculate the importance weights of the feature map at its 2D spatial location and its importance weights in the depth channel dimension, and apply these two weights to the original feature map for feature calibration.

[0085] In the skeleton computation of the target lightweight network, when the feature tensor enters each convolutional layer, the system first uses depthwise separable convolution for computation. Specifically, the first step performs depthwise convolution, which is a two-dimensional convolutional filtering performed only on the independent channel plane of the input feature map. This step is responsible for extracting spatial features within each channel. The second step performs pointwise convolution, which uses a 1x1 convolutional kernel to perform a weighted linear combination of all single-channel results from the first step in the depth direction to generate the final feature map. This step is responsible for fusing cross-channel information. In addition, at key locations in the network, such as the bottleneck layer, a lightweight attention module that combines spatial and channel attention is integrated. This module first calculates channel attention weights to evaluate which feature channels are rich in effective information, such as edges, and applies larger weight multipliers to them. Subsequently, it calculates spatial attention weights to evaluate which spatial locations in the feature map are more likely to belong to real targets or important structures, and applies spatial weight multipliers again for fine-tuning.

[0086] For example, the integration strategy for lightweight attention modules could be: combining spatial and channel-based lightweight attention modules using a convolutional block attention module structure. For the input feature map... Channel attention The calculation formula is: ,in represents the normalized exponential function, MLP represents the multilayer perceptron, AvgPool represents average pooling, and MaxPool represents max pooling. Spatial attention. The calculation formula is: ,in This represents a 7×7 convolution operation, and [;] indicates concatenation along the channel dimension. The module output is... ,in This indicates element-wise multiplication. This module adds approximately 0.15% of the parameters, which can improve the response gain of edge regions in the predicted denoised image by 1.8 times.

[0087] Therefore, this invention enables efficient support for high-performance network architectures under extremely limited computing power on edge devices. The number of parameters in standard convolution increases quadratically with the number of channels, while depthwise separable convolution decouples it into two steps: depthwise convolution and pointwise convolution. Taking an output channel count of 64 and a kernel size of 3 as an example, the calculated parameter compression ratio is close to 89%, which is the foundation for the successful deployment of the model on edge AI on-chip systems. Building upon this, a lightweight attention module combining spatial and channel-based approaches requires only a minimal increase in parameters—approximately 0.15%—yet provides the network with powerful feature calibration capabilities, dynamically enhancing the response to edge contours and suppressing the activation of useless noise regions.

[0088] For example, the mathematical expression of a lightweight convolutional neural network architecture can be: Let the noisy infrared image to be processed be... ,in Let H represent the set of real numbers, H be the image height, and W be the image width. The encoder of the target lightweight network... Layer output feature map ,in This represents the input feature map of the l-th layer. and These represent the height and width of the feature map, respectively. Let be the number of input channels in layer l. Each convolutional layer uses depthwise separable convolution, which is decomposed into two steps: depthwise convolution and pointwise convolution. The formula for calculating depthwise convolution is: ,in This represents the intermediate transition feature map output after depthwise convolution. This indicates a depthwise convolution operation, which performs spatial convolution only within a single channel plane. The kernel is a depthwise convolution, and k is the kernel size. For example, k=3 represents a 3×3 kernel. The number of parameters in this step is... The formula for calculating pointwise convolution is: ,in This represents pointwise convolution, i.e., 1×1 convolution. For pointwise convolution kernels, Let be the number of output channels for the (l+1)th layer. The number of parameters in this step is... Compared to standard convolution, which has 1000 parameters... The parameter compression ratio of depthwise separable convolution is When the number of output channels At that time, the parameter compression rate reached 89%.

[0089] For example, the edge deployment process of the target lightweight network includes model export, integer quantization, memory reuse, operator fusion, and pipelined inference. The model export path is as follows: the deep learning framework model is converted to an open neural network exchange format, and then converted to a tensor inference engine. Integer quantization employs post-training quantization and dynamic range calibration, using 2000 noisy infrared images as a calibration set. A histogram of activation values ​​for each layer is calculated, and the truncation threshold is determined by minimizing relative entropy. Regarding memory reuse, a strict type switch for the tensor inference engine is enabled, forcing the use of 8-bit integer operations throughout, avoiding switching back to 32-bit floating-point numbers, thus ensuring that memory usage remains at a minimum. Simultaneously, adjacent convolutional layers, batch normalization layers, and linear rectified layers are merged, eliminating the need for repeated data reads and writes in memory, achieving operator fusion, and significantly saving computation time and temporary memory. Pipeline inference employs a double-buffering mechanism: the central processing unit (CPU) handles image preprocessing and post-processing, while the neural network processing unit (NN) handles the forward inference of the target lightweight network, and data transmission and computation steps overlap through a direct memory access channel. The neural network processing unit refers to the acceleration core specifically designed for matrix multiplication operations in the edge AI acceleration on-chip system, while the direct memory access channel (DMI) refers to the hardware channel that can move data between memory and peripherals without the intervention of the central processing unit. The measured end-to-end latency is less than 32 milliseconds, the image resolution is 512x512, and the bit width is 8 bits.

[0090] In one specific embodiment, an edge-side AI acceleration system-on-a-chip is used. The noisy infrared image to be processed has a resolution of 512x512. The model is constructed and trained according to the aforementioned method, where the number of parameters of the target lightweight network is approximately 5.2M. After 8-bit integer quantization by the tensor inference engine, the model storage size is 7.4MB. The following data were obtained from actual tests on the above hardware platform: First, in terms of inference performance, the end-to-end average latency is 31 milliseconds, corresponding to a frame rate of approximately 32 frames per second, which meets the processing requirement of 30 frames per second for real-time video streaming. Second, in terms of denoising effect, compared with the undistilled baseline network, on a test set containing 200 real infrared images, the average peak signal-to-noise ratio is improved by 1.7dB, and the structural similarity index is improved by 0.05. In particular, after introducing a lightweight attention module that combines spatial and channel aspects, the gradient response intensity of the selected edge region is approximately 1.8 times that without using this module. The above data are test results under this specific embodiment. Different network parameters, hardware platforms, and test conditions may yield different values.

[0091] In some embodiments, considering the physical characteristics of the infrared detector itself, the intensity of its generated stripe noise and thermal noise is highly positively correlated with the real-time operating temperature of the device's core; that is, the higher the temperature, the more complex and intense the noise. To further improve the robustness of the model under extreme physical environments, this embodiment of the invention introduces a temperature-sensing-based dynamic weight adjustment mechanism in the cross-model feature alignment stage and the edge-side inference stage. Specifically, the real-time physical temperature value of the core is read through the hardware interface of the infrared imaging device; after normalizing the temperature value, a dynamic temperature coefficient is generated. In the target lightweight network, the weights of the activation output or distillation loss of each layer feature are modulated by the dynamic temperature coefficient. When the read physical temperature increases, the weights of the deep multi-scale feature sensing module are dynamically amplified, prompting the network to focus on suppressing the global periodic stripe noise amplified by high temperature; when the physical temperature decreases, the weights of the shallow feature branches are dynamically amplified, prompting the network to focus on preserving the weak thermal radiation edges of the image with high fidelity. Through this mechanism, the limitations of traditional pure software algorithms are broken, and a deep closed-loop collaboration between the physical states of software and hardware is achieved.

[0092] In some embodiments, for applications where portable devices typically output continuous infrared video streams, denoising only a single frame can easily lead to visual flickering between adjacent frames. To address this, this invention introduces a simplified temporal residual alignment strategy in the target lightweight network. Specifically, during network deployment, a very small static buffer is allocated in memory to store only the low-dimensional feature vector or residual data of the previous denoised image. When processing the current frame, the target lightweight network performs lightweight channel concatenation fusion of its extracted spatial features with the temporal features in the buffer. During the knowledge distillation training phase, in addition to single-frame feature matching, the temporal residual change rate of the student network at each time step is further constrained to approximate the temporal residual change rate of the large model teacher network at the same time step. This strategy avoids complex 3D convolution or optical flow calculations, preventing overloading of edge computing power, and significantly improves the smoothness and visual coherence of the infrared video stream in the temporal dimension simply by aligning the inter-frame residual change rates.

[0093] Figure 2 This is a schematic diagram illustrating the cross-model feature alignment architecture principle of a lightweight edge-side infrared image denoising method provided in an embodiment of the present invention. Figure 2 As shown, during the training phase, the noisy infrared image is simultaneously input into the pre-trained prior network on the left, corresponding to... Figure 2 The leftmost layer consists of stacked rectangular layers, forming a ring-shaped symmetrical structure with the target lightweight network in the middle. The encoder of the target lightweight network progressively extracts deep features during downsampling, while the decoder reconstructs the image during upsampling, and this is achieved through skip connections. Figure 2The curved arrows shown convey edge details. Each layer nests a multi-scale feature-aware module, denoted as MSFE. The final output is a predicted denoised image, or DenoisedImage, which is then subjected to overall optimization constraints through a joint loss function at the bottom.

[0094] Figure 3 This is a schematic diagram of the internal structure of a multi-scale feature perception module provided in an embodiment of the present invention. Figure 3 As shown, the multi-scale feature perception module receives the input feature map and splits it into three parallel convolutional branches. Branch 1 uses a baseline convolutional kernel to extract fine-grained local details, while branches 2 and 3 use dilated convolutional kernels to capture medium and macroscopic stripes. After each branch passes through the ReLU activation function, they are concatenated along the channel dimension to output a feature map rich in multi-scale information, known as the Fused Feature Map.

[0095] Figure 4 This is a schematic diagram illustrating the data flow of the calculation logic for gradient domain sparse regularization loss provided in an embodiment of the present invention. For example... Figure 4 As shown, when calculating the gradient domain sparse regularization loss, the reference image Ground Truth and the predicted denoised image Prediction are subtracted to generate a residual image. Subsequently, the horizontal gradient Gx and vertical gradient Gy are extracted using the Sobel edge detection operator. The aforementioned embodiment describes the horizontal gradient as... and vertical gradient The corresponding parameters have the same meaning. The sum of the absolute values ​​of the extracted gradients is calculated using L1 Norm and then multiplied by the weight coefficient. The final penalty item will be obtained. It is used to prevent the edges from being worn away.

[0096] Figure 5 This is a hardware timing diagram illustrating the inference operation performed by an electronic device according to an embodiment of the present invention. Figure 5 As shown, on the horizontal timeline, image acquisition, pre-processing of the central processing unit, model inference of the neural network processing unit, and post-processing and display are presented as overlapping black bars. This indicates that through direct memory access (DMA) technology, computation and data transfer are parallelized in time, thereby significantly reducing the total latency of processing a single frame image to 30ms.

[0097] Figure 6This is a schematic diagram illustrating the delay performance test results of an edge-side lightweight infrared image denoising method provided in an embodiment of the present invention during edge-side inference. Figure 6 As shown, the performance of the target lightweight network during forward inference operations on an edge AI accelerator on-chip system is illustrated. The horizontal axis represents the frame index of the continuously processed infrared image video stream, and the vertical axis represents the end-to-end latency. When processing infrared images with a resolution of 512×512, the target lightweight network, after 8-bit integer quantization and with a model storage size of only 7.4MB, maintains a highly stable processing latency curve, with an average end-to-end latency of approximately 31 milliseconds, corresponding to a processing frame rate of approximately 32 frames per second. Therefore, thanks to the compression of the underlying architecture using depthwise separable convolutions and the computational overlap of the pipelined double-buffering mechanism, the network completely eliminates redundant floating-point operation load. Mathematical and engineering tests fully demonstrate that this embodiment of the invention, while maintaining extremely low memory usage, meets the stringent latency requirements of resource-constrained edge devices for processing real-time infrared video streams of ≥30 frames per second.

[0098] Figure 7 This is a schematic diagram illustrating the comparison test results of the peak signal-to-noise ratio denoising effect between a target lightweight network and a baseline network provided in an embodiment of the present invention. Figure 7 As shown, this paper presents a comparison of objective denoising quality metrics between the network trained by the method of this invention and the baseline network without cross-model feature alignment architecture distillation on a test set containing multiple real noisy infrared images to be processed. The horizontal axis represents the image index in the test set, and the vertical axis represents the peak signal-to-noise ratio (PSNR). The PNR curve of the target lightweight network with approximately 5.2M parameters is significantly higher than that of the baseline network overall and consistently. Specifically, the method of this invention achieves an average PNR improvement of 1.7 dB, while simultaneously improving the structural similarity index by 0.05. Thus, through the distillation loss introduced by the cross-model feature alignment architecture, the noise prior and feature distribution of the large pre-trained prior network are losslessly transferred to the target lightweight network. Objectively, this demonstrates that the target lightweight network achieves significantly higher denoising fidelity than traditional small networks while achieving extreme lightweight design, effectively overcoming the performance degradation trap caused by a sharp reduction in parameters.

[0099] Figure 8 This is a schematic diagram illustrating the edge gradient response enhancement effect of a lightweight attention module combining spatial and channel elements in a target lightweight network provided by an embodiment of the present invention. Figure 8As shown, this paper reveals the mechanism by which a lightweight attention module combining spatial and channel elements is integrated into the bottleneck layer of the target lightweight network to protect and enhance the true edge details of infrared images. The horizontal axis represents the spatial location of the selected true edge in the predicted denoised image, and the vertical axis represents the gradient magnitude at that location. A comparison of the two curves clearly shows that the gradient response magnitude curve of the network integrating this lightweight attention module in the edge region is significantly higher than the baseline curve without this module. Calculations show that this module increases the gradient response intensity in the edge region by approximately 1.8 times. Therefore, this module sequentially calculates the importance weights of the feature map in its two-dimensional spatial location and its importance weights in the depth channel direction, and applies a multiplier calibration to high-frequency pixels rich in edge information. At the physical computation level, this demonstrates that the module can accurately identify and significantly and dynamically enhance weak true edge information features, which corresponds to the gradient domain sparse regularization loss in the joint loss function.

[0100] The end-side lightweight infrared image denoising device provided by the present invention will be described below. The end-side lightweight infrared image denoising device described below can be referred to in correspondence with the end-side lightweight infrared image denoising method described above.

[0101] Figure 9 This is a schematic diagram of the structure of a lightweight end-side infrared image denoising device provided in an embodiment of the present invention. Figure 9 As shown, the edge-side lightweight infrared image denoising device includes an image acquisition module 201 for acquiring a noisy infrared image to be processed; and a lightweight training module 202 for inputting the noisy infrared image into a pre-trained target lightweight network and outputting a predicted denoised image. The target lightweight network is optimized during the training phase based on a joint loss function, which includes at least pixel-level reconstruction loss, gradient domain sparse regularization loss, and vertical direction smoothing constraint loss. The gradient domain sparse regularization loss constrains the residual image between the reference image and the predicted denoised image, ensuring the residual image remains sparse in the gradient domain to preserve real edge details. The vertical direction smoothing constraint loss calculates the difference between adjacent pixels in the vertical direction of the predicted denoised image and penalizes the difference to suppress horizontal stripe noise in the infrared image.

[0102] Figure 10 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Figure 10As shown, the electronic device may include: a processor 401, a communications interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communications interface 402, and the memory 403 communicate with each other via the communication bus 404. The processor 401 can call logical instructions in the memory 403 to execute an edge-side lightweight infrared image denoising method, including:

[0103] Acquire the noisy infrared image to be processed;

[0104] The noisy infrared image is input into a pre-trained lightweight target network, and the predicted denoised image is output.

[0105] The target lightweight network is optimized during the training phase based on a joint loss function, which includes at least pixel-level reconstruction loss, gradient domain sparse regularization loss, and vertical direction smoothing constraint loss. The gradient domain sparse regularization loss is used to constrain the residual image between the reference image and the predicted denoised image, so that the residual image remains sparse in the gradient domain to preserve the true edge details of the image. The vertical direction smoothing constraint loss is used to calculate the difference between adjacent pixels in the vertical direction of the predicted denoised image and to penalize the difference to suppress horizontal stripe noise in the infrared image.

[0106] Furthermore, the logical instructions in the aforementioned memory 403 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] Therefore, in actual industrial deployments, the parameters of the pre-trained lightweight target network are highly quantized and compressed before being burned into the memory of a portable infrared device. When the processor calls this program to process noisy infrared image streams, the system consumes very few computational resources and has minimal effective memory usage by abandoning the heavy traditional network architecture and instead utilizing a lightweight skeleton rigorously reshaped by the joint loss function. Combined with pipelined scheduling such as direct memory access, high-fidelity real-time denoising is achieved on resource-constrained endpoints.

[0108] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to perform the above-described edge-side lightweight infrared image denoising method, including:

[0109] Acquire the noisy infrared image to be processed;

[0110] The noisy infrared image is input into a pre-trained lightweight target network, and the predicted denoised image is output.

[0111] The target lightweight network is optimized during the training phase based on a joint loss function, which includes at least pixel-level reconstruction loss, gradient domain sparse regularization loss, and vertical direction smoothing constraint loss. The gradient domain sparse regularization loss is used to constrain the residual image between the reference image and the predicted denoised image, so that the residual image remains sparse in the gradient domain to preserve the true edge details of the image. The vertical direction smoothing constraint loss is used to calculate the difference between adjacent pixels in the vertical direction of the predicted denoised image and to penalize the difference to suppress horizontal stripe noise in the infrared image.

[0112] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-described end-side lightweight infrared image denoising method, comprising:

[0113] Acquire the noisy infrared image to be processed;

[0114] The noisy infrared image is input into a pre-trained lightweight target network, and the predicted denoised image is output.

[0115] The target lightweight network is optimized during the training phase based on a joint loss function, which includes at least pixel-level reconstruction loss, gradient domain sparse regularization loss, and vertical direction smoothing constraint loss. The gradient domain sparse regularization loss is used to constrain the residual image between the reference image and the predicted denoised image, so that the residual image remains sparse in the gradient domain to preserve the true edge details of the image. The vertical direction smoothing constraint loss is used to calculate the difference between adjacent pixels in the vertical direction of the predicted denoised image and to penalize the difference to suppress horizontal stripe noise in the infrared image.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight end-to-end infrared image denoising method, characterized in that, include: Acquire the noisy infrared image to be processed; The noisy infrared image is input into a pre-trained lightweight target network, and a predicted denoised image is output. The target lightweight network is optimized during the training phase based on a joint loss function, which includes at least pixel-level reconstruction loss, gradient domain sparse regularization loss, and vertical direction smoothing constraint loss. The gradient domain sparse regularization loss is used to constrain the residual image between the reference image and the predicted denoised image, so that the residual image remains sparse in the gradient domain to preserve the true edge details of the image. The vertical direction smoothing constraint loss is used to calculate the difference between adjacent pixels in the vertical direction of the predicted denoised image and to penalize the difference to suppress horizontal stripe noise in the infrared image.

2. The end-side lightweight infrared image denoising method according to claim 1, characterized in that, The calculation process of the gradient domain sparse regularization loss includes: Calculate the residual image obtained by subtracting the reference image from the predicted denoised image; The horizontal and vertical gradients of the residual image are extracted using edge detection operators, respectively. The absolute values ​​of the horizontal gradient and the vertical gradient are summed.

3. The end-side lightweight infrared image denoising method according to claim 1, characterized in that, During the training phase, the target lightweight network is trained using a cross-model feature alignment architecture; The cross-model feature alignment architecture includes a pre-trained prior network with frozen parameters and the target lightweight network to be optimized. The joint loss function also includes distillation loss, which is used to guide the target lightweight network to mimic the noise prior and feature distribution of the pre-trained prior network.

4. The end-side lightweight infrared image denoising method according to claim 3, characterized in that, The distillation loss includes feature layer distillation loss and response layer distillation loss. The feature layer distillation loss is used to match the feature map differences between the target lightweight network and the pre-trained prior network at the corresponding feature layer. The response layer distillation loss is used to calculate the relative entropy between the output distribution of the pre-trained prior network and the output distribution of the target lightweight network. The output distribution undergoes a softening process with a preset temperature coefficient before the relative entropy is calculated.

5. The end-side lightweight infrared image denoising method according to claim 4, characterized in that, The calculation of the distillation loss of the feature layer also includes: When the feature map dimensions of the target lightweight network and the pre-trained prior network do not match, the feature map of the pre-trained prior network is mapped to the same feature space as the target lightweight network through a projection layer with preset weight parameters. Then, the mean square error between the mapped feature map of the pre-trained prior network and the feature map of the target lightweight network is calculated and used as the feature map difference of the feature layer.

6. The end-side lightweight infrared image denoising method according to any one of claims 1-5, characterized in that, The target lightweight network integrates a multi-scale feature perception module; The multi-scale feature perception module includes multiple parallel convolutional branches, each of which uses a convolutional kernel with a different dilation rate to capture noise features of different granularities. The feature maps output by each convolutional branch are spliced ​​and fused along the channel dimension.

7. The end-side lightweight infrared image denoising method according to claim 6, characterized in that, The multi-scale feature perception module includes at least three parallel convolutional branches; The first convolutional branch uses a baseline convolutional kernel with a first dilation rate to extract local details, the second convolutional branch uses a dilated convolutional kernel with a second dilation rate, and the third convolutional branch uses a dilated convolutional kernel with a third dilation rate, wherein the third dilation rate is greater than the second dilation rate.

8. The end-side lightweight infrared image denoising method according to any one of claims 1-5, characterized in that, Each convolutional layer of the target lightweight network employs depthwise separable convolution, and the target lightweight network integrates a lightweight attention module that combines spatial and channel-based approaches.

9. A lightweight end-side infrared image denoising device, characterized in that, include: The image acquisition module is used to acquire the noisy infrared image to be processed; A lightweight training module is used to input the noisy infrared image into a pre-trained target lightweight network and output a predicted denoised image. The target lightweight network is optimized during the training phase based on a joint loss function, which includes at least pixel-level reconstruction loss, gradient domain sparse regularization loss, and vertical direction smoothing constraint loss. The gradient domain sparse regularization loss is used to constrain the residual image between the reference image and the predicted denoised image, so that the residual image remains sparse in the gradient domain to preserve the true edge details of the image. The vertical direction smoothing constraint loss is used to calculate the difference between adjacent pixels in the vertical direction of the predicted denoised image and to penalize the difference to suppress horizontal stripe noise in the infrared image.

10. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the end-side lightweight infrared image denoising method as described in any one of claims 1-8.

Images