An embedded NPU video super-resolution reconstruction method, system, and device

Abstract

This invention provides an embedded NPU video super-resolution reconstruction method, system, and device, comprising: converting the original high-resolution image into a low-resolution image through a combination of degradation operations to simulate the image degradation process in a real-world scene for model training; reconstructing the low-resolution image into a super-resolution image through a lightweight image super-resolution generator and discriminator, wherein the generator employs depthwise separable convolution and an effective channel attention mechanism, combined with residual modules and feature fusion techniques, to significantly reduce model parameters and computational load while maintaining high image reconstruction quality; training the model using an image training dataset; quantizing and compressing the model; when deployed on an embedded device, inputting a low-resolution video, reading video frames through the OpenCV library, preprocessing them, inputting them into the quantized model for inference, outputting a super-resolution image, and recombining them into a high-resolution video; and finally using hardware acceleration methods to accelerate the inference process for the deployed model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an embedded NPU video super-resolution reconstruction method, system, and device. Background Technology

[0002] Image super-resolution reconstruction is a processing method aimed at improving image resolution and enhancing image detail and quality. This technique generates high-resolution images from single or multiple low-resolution images by analyzing the information and structure contained within the image and learning prior knowledge. It has wide applications in various fields such as surveillance video, medical imaging, satellite remote sensing, and video enhancement. Traditional image super-resolution reconstruction methods are mainly based on interpolation techniques, such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. While these methods are simple to operate, enlarging the image simply by increasing pixel size, they often fail to effectively recover high-frequency details in the image, thus limiting their reconstruction results.

[0003] With the rapid development of deep learning technology, significant progress has been made in the field of computer vision, especially in deep learning-based image super-resolution reconstruction techniques. For example, the SRCNN model proposed by Dong et al. uses a simple three-layer convolutional neural network (CNN) to learn the mapping relationship between low-resolution images and high-resolution images. Subsequently, various efficient super-resolution models such as RCAN, EDSR, and SRGAN have emerged, which further improve the learning ability and image reconstruction performance of the network by increasing the network depth and the number of parameters.

[0004] However, deploying these deep network models on embedded devices with limited computing resources and memory presents challenges. The limitations of embedded devices restrict the size and complexity of the deployed networks, and in practical applications, issues such as slow inference speed and reduced reconstruction accuracy may arise. Therefore, a comprehensive balance must be struck between model structure, inference speed, and reconstruction accuracy, employing lightweight and optimization strategies to adapt to the resource constraints of embedded devices. Summary of the Invention

[0005] To address the aforementioned issues, this invention aims to propose an embedded NPU video super-resolution reconstruction method, system, and device for reconstructing low-resolution videos into high-resolution videos. This method reduces computational load and parameter count while maintaining high image quality, and utilizes hardware acceleration techniques to improve model inference speed.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] An embedded NPU video super-resolution reconstruction method includes:

[0008] S1. Data Processing: Convert the original high-resolution image into a low-resolution image through a combination of degradation operations to simulate the image degradation process in real-world scenarios for use in model training.

[0009] S2. Model Construction: Low-resolution images are reconstructed into super-resolution images through a lightweight image super-resolution generator and discriminator. The generator uses depthwise separable convolution and an effective channel attention mechanism, combined with residual modules and feature fusion techniques, to significantly reduce model parameters and computational cost while maintaining high image reconstruction quality.

[0010] S3. Model Training: The model is trained using an image training dataset. Low-resolution images are used as input to the generator network, and the corresponding original high-resolution image samples are used as the expected output of the generator network. The generator network is trained using the backpropagation algorithm. Low-resolution images and their corresponding high-resolution images are used as positive image pairs, and low-resolution images in the training sample dataset and the corresponding output images of the generator network are used as negative image pairs. The discriminator network is trained using the backpropagation algorithm.

[0011] S4. Model Quantization and Compression: Convert the trained super-resolution model file into the intermediate ONNX format, and further convert the ONNX format model file into the embedded device compatible RKNN format.

[0012] S5. Deployment: Input a low-resolution video, read each frame of the video using the OpenCV library, perform necessary preprocessing on each frame, and input it into the quantized model. The model will infer the input low-resolution image and load the model weight file obtained during training. After processing by the quantized model, the super-resolution image is output. Finally, the processed super-resolution image is reassembled into a video using the OpenCV library, thus completing the reconstruction of the super-resolution video.

[0013] S6. Acceleration: Use hardware acceleration methods to speed up the inference process for the model deployed in step S5.

[0014] Furthermore, the data processing in step S1 includes: performing cropping, Gaussian blur, defocusing and motion blur simulation, and adding Gaussian noise on the original high-resolution image to generate a first reference image;

[0015] JPEG compression is applied to the first reference image to introduce image quality compression, resulting in a second reference image;

[0016] Bicubic interpolation downsampling is performed on the second reference image, and normalization is applied to obtain the degraded low-resolution image.

[0017] Furthermore, the image reconstruction in step S2 includes: feeding the low-resolution image into the shallow feature extraction module of the image super-resolution generation network to obtain a shallow feature map of the image;

[0018] The shallow feature map of the image is fed into the deep feature extraction module of the image super-resolution generation network to obtain the deep feature map of the image.

[0019] Furthermore, the deep feature extraction module includes a stack of a preset number of residual modules. Each residual module is composed of a cascade of 1x1 convolution, 3x3 convolution, sigmoid and an effective channel attention mechanism, and a depthwise separable convolution operation is introduced.

[0020] The shallow and deep feature maps of the image are fed into the feature fusion module to obtain a fused feature map;

[0021] The fused feature map is fed into the upsampling module of the image super-resolution generation network to obtain the super-resolution image of the low-resolution image;

[0022] By inputting the super-resolution image and the high-resolution image into the image super-resolution discrimination network, the probability value of judging the super-resolution image as a real image can be obtained.

[0023] The discriminant network is an image classification convolutional neural network that uses the LeakReLU activation function to prevent negative outputs from dying.

[0024] The input image is processed through a classification convolutional neural network and then through two linear layers to obtain the output result.

[0025] Based on the high-resolution image, the super-resolution image, and the probability values ​​output by the discriminator network, the loss functions of the generator network and the discriminator network are determined.

[0026] Furthermore, the loss function of the discriminant network is:

[0027]

[0028] The loss function of the generator network consists of three parts:

[0029]

[0030] Where L percep Using VGG loss:

[0031]

[0032] W and H represent the dimensions of the feature map in the VGG network, respectively;

[0033] i and j refer to the j-th convolutional layer before max pooling in the i-th layer;

[0034] The adversarial loss function of the generator network

[0035]

[0036] in Used to evaluate the generated super-resolution image G(x) i The content loss is the 1-norm distance between the original high-resolution image and the original high-resolution image, where λ and η are weighting coefficients.

[0037] Furthermore, the model training in step S3 includes: optimizing and fine-tuning the model using the created video training data samples;

[0038] First, obtain a batch of low-resolution image groups from the generated training data. Group The batch size is set to 32, meaning 32 low-resolution images are acquired at a time.

[0039] Secondly, a corresponding super-resolution image set (SR) is generated using a generative network. Group ;

[0040] Next, the generated SR Group Compared with the original high-resolution image group in the training data HR Group The corresponding input is sent to the discrimination network;

[0041] Meanwhile, the model is tuned using the loss functions of the generator and the discriminator to complete the model training process.

[0042] Furthermore, the model quantization compression in step S4 also includes: during this conversion process, the input size, mean, normalization parameters, and whether to perform quantization configuration options of the model need to be preset;

[0043] Perform model quantization on the RKNN file, that is, convert floating-point numbers to fixed-point numbers for efficient operation on embedded devices.

[0044] Furthermore, the acceleration in step S6 includes: using the hardware image processing accelerator (RGA) of the embedded device to speed up image preprocessing.

[0045] Concurrency is achieved through thread pools, thereby accelerating model inference and improving the real-time performance of the system.

[0046] Before embedded inference, RGA is used to segment large images into sub-blocks for super-resolution reconstruction, the sub-block stitching process is optimized, and overlapping region fusion technology is used to solve the impact of stitching lines.

[0047] To achieve the above objectives, the present invention also discloses an embedded NPU video super-resolution reconstruction system, comprising:

[0048] Data processing module: used to convert the original high-resolution image into a low-resolution image through a variety of combined degradation operations for use in model training;

[0049] Model building module: Reconstructs low-resolution images into super-resolution images using a lightweight image super-resolution generator and discriminator;

[0050] Model training module: The generator network is trained using an image training dataset. Low-resolution images are used as input to the generator network, and the corresponding original high-resolution image samples are used as the expected output of the generator network. The generator network is trained using the backpropagation algorithm. Low-resolution images and their corresponding high-resolution images are used as positive image pairs, and low-resolution images in the training sample dataset and the corresponding output images of the generator network are used as negative image pairs. The discriminator network is trained using the backpropagation algorithm.

[0051] Model quantization and compression module: used to convert trained super-resolution model files into intermediate ONNX format, and further convert ONNX format model files into RKNN format compatible with embedded devices;

[0052] Deployment module: Input low-resolution video, read each frame of the video using the OpenCV library, perform necessary preprocessing on each frame, and input it into the quantized model. The model will infer the input low-resolution image and load the model weight file obtained during training. After processing by the quantized model, the super-resolution image is output. Finally, the processed super-resolution image is recombined into the video using the OpenCV library, thus completing the reconstruction of the super-resolution video.

[0053] Acceleration Module: Uses hardware acceleration methods to speed up the inference process for the model deployed in the module.

[0054] To achieve the above objectives, the present invention also discloses an embedded NPU video super-resolution reconstruction device, including a processor and a storage medium;

[0055] The storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the above method.

[0056] Beneficial effects: Based on existing image super-resolution reconstruction algorithms, this invention significantly reduces the number of model parameters and computational complexity by using a lightweight residual structure to replace the original residual network. The model conversion and quantization process, while ensuring image reconstruction quality, further compresses the model size, enhances its portability, and makes it more suitable for deployment on resource-constrained embedded devices.

[0057] Furthermore, this invention integrates multiple hardware acceleration technologies to fully exploit hardware potential, significantly improving the inference speed of the model on embedded devices, thereby enhancing its practicality. Validation on different test datasets demonstrates that the super-resolution reconstruction method of this invention not only possesses fast inference capabilities but also high portability, accelerating the development of deep learning super-resolution reconstruction in the embedded field. Attached Figure Description

[0058] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0059] Figure 1 This is a flowchart of the embedded NPU video super-resolution reconstruction method according to an embodiment of the present invention;

[0060] Figure 2 This is a structural diagram of the image super-resolution generator in the embedded NPU video super-resolution reconstruction method described in this embodiment of the invention;

[0061] Figure 3 This is a flowchart of model quantization compression in the embedded NPU video super-resolution reconstruction method described in this embodiment of the invention;

[0062] Figure 4 This is a structural diagram of the embedded NPU video super-resolution reconstruction method described in this embodiment of the invention, which uses a thread pool for parallel model operation.

[0063] Figure 5 This is a diagram of the cropping and splicing scheme in the embedded inference of the embedded NPU video super-resolution reconstruction method described in this embodiment of the invention;

[0064] Figure 6 This is a schematic diagram of the embedded NPU video super-resolution reconstruction system according to an embodiment of the present invention. Detailed Implementation

[0065] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0066] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0067] Example 1

[0068] See Figure 1-5 An embedded NPU video super-resolution reconstruction method, comprising:

[0069] S1. Data Processing: Convert the original high-resolution image into a low-resolution image through a combination of degradation operations to simulate the image degradation process in real-world scenarios for use in model training.

[0070] S2. Model Construction: Low-resolution images are reconstructed into super-resolution images through a lightweight image super-resolution generator and discriminator. The generator uses depthwise separable convolution and an effective channel attention mechanism, combined with residual modules and feature fusion techniques, to significantly reduce model parameters and computational cost while maintaining high image reconstruction quality.

[0071] S3. Model Training: The model is trained using an image training dataset. Low-resolution images are used as input to the generator network, and the corresponding original high-resolution image samples are used as the expected output of the generator network. The generator network is trained using the backpropagation algorithm. Low-resolution images and their corresponding high-resolution images are used as positive image pairs, and low-resolution images in the training sample dataset and the corresponding output images of the generator network are used as negative image pairs. The discriminator network is trained using the backpropagation algorithm.

[0072] S4. Model Quantization and Compression: Convert the trained super-resolution model file into the intermediate ONNX format, and further convert the ONNX format model file into the embedded device compatible RKNN format.

[0073] S5. Deployment: Input a low-resolution video, read each frame of the video using the OpenCV library, perform necessary preprocessing on each frame, and input it into the quantized model. The model will infer the input low-resolution image and load the model weight file obtained during training. After processing by the quantized model, the super-resolution image is output. Finally, the processed super-resolution image is reassembled into a video using the OpenCV library, thus completing the reconstruction of the super-resolution video.

[0074] S6. Acceleration: Use hardware acceleration methods to speed up the inference process for the model deployed in step S5.

[0075] The combined degradation operation in this embodiment can effectively simulate motion-induced blur; the lightweight image super-resolution generator and discriminator can effectively reduce model parameters and computational load, enhance the network's expressive power, and maintain high image reconstruction quality; by utilizing image processing accelerators and thread pool technology, the inference speed on embedded devices is improved. This embodiment realizes the efficient implementation of the super-resolution algorithm and is suitable for real-time video processing in embedded environments.

[0076] In a specific example, the data processing in step S1 includes: performing cropping, Gaussian blur, defocusing and motion blur simulation, and Gaussian noise addition on the original high-resolution image to generate a first reference image;

[0077] JPEG compression is applied to the first reference image to introduce image quality compression, resulting in a second reference image;

[0078] Bicubic interpolation downsampling is performed on the second reference image, and normalization is applied to obtain the degraded low-resolution image.

[0079] In the specific implementation, each original high-resolution image can be randomly cropped into 88x88 sub-images; then, Gaussian blur is applied to these sub-images, where the size of the blur kernel is uniformly and randomly selected between 7x7, 9x9, ..., 21x21 to simulate the blur effect in actual shooting; a reflection filling method is used to ensure the consistency of the output space of the blur operation; then, a blur effect simulating defocus and motion is added to the blurred sub-images, and Gaussian noise is introduced to generate the first reference image;

[0080] Next, the first reference image is compressed using JPEG image quality to obtain the second reference image; the JPEG quality factor is uniformly selected from [30, 95].

[0081] The second reference image is downsampled by bicubic 2x to obtain the degraded low-resolution image.

[0082] Common interpolation functions are:

[0083] Then, the downsampled low-resolution image and the corresponding original high-resolution image are normalized to scale the pixel values ​​to the range of [0,1] in preparation for subsequent model training.

[0084] Through the above operations, the original high-resolution image is converted into a low-resolution image that is similar to the noise and blur caused by motion and other factors in real life, thus providing a dataset that is close to real-world scenarios for model training.

[0085] In a specific example, the image reconstruction in step S2 includes: feeding the low-resolution image into the shallow feature extraction module of the image super-resolution generation network to obtain a shallow feature map of the image;

[0086] The shallow feature map of the image is fed into the deep feature extraction module of the image super-resolution generation network to obtain the deep feature map of the image.

[0087] In a specific implementation, an image super-resolution generator such as Figure 2 The diagram consists of the following parts:

[0088] The input to the network is a 3-channel RGB low-resolution image. First, a 9x9 convolutional layer and the PReLU activation function are used to extract shallow feature maps. Next, these feature maps are fed into 10 EC-PA modules to progressively refine the image's deeper features. Then, the deep feature maps are processed by a 3x3 convolutional layer, and the resulting feature map is concatenated with the initial shallow feature maps to generate a fused feature map. This fused feature map then enters the PixelShuffle module for a 2x upsampling operation. Finally, the upsampled feature map is processed by a 9x9 deconvolutional layer and the Tanh activation function, ultimately outputting a super-resolution reconstructed image.

[0089] The EC-PA module first separates the input shallow feature map xn-1 into two feature maps, x'n-1 and x”n-1, through two 1x1 convolutional layers. x'n-1 then passes through a 3x3 DSConv and a PAConv module, which includes a 1x1 convolution and a sigmoid activation function. The two feature maps are then element-wise multiplied, and the result is passed through a 3x3 DSConv to obtain the feature map x'n. x”n-1 then passes through a 3x3 DSConv and an ECAConv module. To further enhance the correlation between channels, the two resulting feature maps are concatenated along the channel dimension and then fed into a 3x3 DSConv module to obtain feature map x”n; x’n and x”n are concatenated along the channel dimension and then fed into a 1x1 convolution to obtain the enhanced feature map, restoring the number of channels to the input number of channels; the enhanced feature map is then added element-wise to the input feature map to form the final output feature map xn; the ECAConv module implements attention calculation for each channel through global average pooling and 1D convolution, and the output feature map is multiplied element-wise with the input feature map;

[0090] The PixelShuffle upsampling module utilizes convolution and pixel rearrangement techniques to upsample the image. It receives input with `in_channels` channels and generates an output feature map with `in_channels*up_scale^2` channels. The 3x3 convolutional kernels and 1-padded layers ensure consistent input and output dimensions, pre-increasing the number of channels in preparation for pixel rearrangement. Pixel rearrangement transforms the multi-channel feature map into a higher-resolution image. The `up_scale` parameter determines the upsampling ratio; for example, when `up_scale = 2`, the image size doubles. After upsampling, the feature map is activated by the PReLU function to obtain the final output.

[0091] Table 1 Comparison of parameters, computational cost, and central data in the Set test dataset.

[0092]

[0093] As shown in Table 1, the method of this invention has lower parameter and computational costs compared to other current image super-resolution reconstruction methods at magnifications of 2x and 4x. Furthermore, it maintains high computational performance on Set5 and Set14 data sets.

[0094] In a specific example, the deep feature extraction module includes a stack of a preset number of residual modules. Each residual module is composed of a cascade of 1x1 convolution, 3x3 convolution, sigmoid and an effective channel attention mechanism, and a depthwise separable convolution operation is introduced.

[0095] The shallow and deep feature maps of the image are fed into the feature fusion module to obtain a fused feature map;

[0096] The fused feature map is fed into the upsampling module of the image super-resolution generation network to obtain the super-resolution image of the low-resolution image;

[0097] By inputting the super-resolution image and the high-resolution image into the image super-resolution discrimination network, the probability value of judging the super-resolution image as a real image can be obtained.

[0098] The discriminant network is an image classification convolutional neural network that uses the LeakReLU activation function to prevent negative outputs from dying.

[0099] The input image is processed through a classification convolutional neural network and then through two linear layers to obtain the output result.

[0100] Based on the high-resolution image, the super-resolution image, and the probability values ​​output by the discriminator network, the loss functions of the generator network and the discriminator network are determined.

[0101] In its implementation, the image super-resolution discriminator consists of the following parts:

[0102] The discriminator network architecture is a convolutional neural network-based structure used to distinguish between real and generated images. The network input is a three-channel image, processed through a series of convolutional layers, batch normalization layers, and activation functions. The network contains multiple convolutional blocks, each consisting of two convolutional layers: a 3x3 convolution followed by a 4x4 convolution, then batch normalization and a LeakyReLU activation function. Spatial downsampling is achieved between convolutional blocks using convolutional layers with a stride of 2, progressively reducing the feature map size and increasing the number of channels. Finally, fully connected layers flatten the feature map into a one-dimensional vector, and two fully connected layers output a single real value as the result of image real / fake discrimination. The entire network design emphasizes maintaining a doubling of the number of channels between convolutional blocks to progressively extract higher-level features, while batch normalization and the LeakyReLU activation function stabilize the training process.

[0103] The discriminator is fed with real high-resolution images (IHR) and super-resolution reconstructed images (ISR) and scores them. The discriminator network finally outputs a probability value, which represents the probability that the input image is a real image.

[0104] In a specific example, the loss function of the discriminant network is:

[0105]

[0106] The loss function of the generator network consists of three parts:

[0107]

[0108] Where L percep Using VGG loss:

[0109]

[0110] W and H represent the dimensions of the feature map in the VGG network, respectively;

[0111] i and j refer to the j-th convolutional layer before max pooling in the i-th layer;

[0112] The adversarial loss function of the generator network

[0113]

[0114] in Used to evaluate the generated super-resolution image G(x) i The content loss is the 1-norm distance between the original high-resolution image and the original high-resolution image, where λ and η are weighting coefficients.

[0115] The above description of this embodiment illustrates the principle of the network loss function. It can be seen that the discriminant network loss function directly calculates the loss between the original high-resolution image and the super-resolution image, with the aim of making the super-resolution image closer to the high-resolution image. The purpose of the generator network loss function is to make the result of the generator network able to deceive the discriminant network well.

[0116] In a specific example, the model training in step S3 includes: optimizing and fine-tuning the model using the created video training data samples;

[0117] First, obtain a batch of low-resolution image groups from the generated training data. Group The batch size is set to 32, meaning 32 low-resolution images are acquired at a time.

[0118] Secondly, a corresponding super-resolution image set (SR) is generated using a generative network. Group;

[0119] Next, the generated SR Group Compared with the original high-resolution image group in the training data HR Group The corresponding input is sent to the discrimination network;

[0120] Meanwhile, the model is tuned using the loss functions of the generator and the discriminator to complete the model training process.

[0121] This process in this embodiment aims to allow the model to gradually learn how to generate high-quality reconstructed images from low-resolution images, thereby improving its performance and adaptability.

[0122] In the specific implementation, training can be performed using an Ubuntu 16.04 system, a Tesla K80 GPU, PyTorch 1.11.0, Python 3.6.8, and CUDA version 11.3.1. The training process uses the Adam optimizer with a learning rate of 10e-4 and a batch size of 32.

[0123] This experiment uses the DIV2K dataset as the image training dataset and the REDS dataset as the video training dataset. The training datasets are processed by the data processing module and then fed into the generator and discriminator models for training.

[0124] By performing 100 iterations, the generator eventually learns the sample distribution of high-resolution images during the adversarial training process with the discriminator, resulting in a trained generator model file.

[0125] The model files obtained from the above training were fine-tuned for 50 rounds using the REDS video training dataset to learn how to better handle noise, thereby improving robustness to low-quality images and temporal consistency features.

[0126] In a specific example, the model quantization compression in step S4 further includes: during this conversion process, the input size, mean, normalization parameters, and whether to perform quantization configuration options need to be preset.

[0127] Perform model quantization on the RKNN file, that is, convert floating-point numbers to fixed-point numbers for efficient operation on embedded devices.

[0128] In the actual implementation, the trained super-resolution model file is converted into a model file in the RKNN format supported by the device;

[0129] The weights and activation values ​​of the trained model are quantized and compressed. The quantization process involves replacing the 32-bit floating-point weights and activation values ​​with 8-bit fixed-point values, compressing them from the original signed continuous values ​​to a discrete value range consisting of only 28 fixed-point numbers. The quantization process is as follows:

[0130]

[0131] Where x is a floating-point number and xint is a quantized fixed-point number. For rounding operations, s is the quantization scale factor, z is the quantization zero point, and b is the quantization bit width, such as b being 8 in the INT8 data type; clamp is the truncation operation, specifically defined as follows:

[0132]

[0133] The quantized network model is then tested on the corresponding test dataset using an embedded board to evaluate its accuracy, performance, and memory usage. If the inference results in the embedded system differ significantly in accuracy from those on the PC, the KL-Divergence quantization algorithm can be used to recalibrate the quantized model.

[0134] The algorithm in this embodiment updates the distribution of floating-point and fixed-point numbers by adjusting different thresholds, and determines the maximum and minimum values ​​of the quantization range by minimizing the similarity between the two distributions based on KL divergence. The KL-Divergence quantization algorithm, by minimizing the distribution difference between floating-point and fixed-point numbers, can better adapt to non-uniform data distributions and mitigate the impact of a few outliers. The amount of quantized data is approximately 20-100 images.

[0135] In a specific example, the acceleration of step S6 includes: using the hardware image processing accelerator (RGA) of the embedded device to speed up image preprocessing;

[0136] Concurrency is achieved through a thread pool, thereby accelerating model inference and improving system real-time performance. Thread A is responsible for video reading, thread C is responsible for storing inference results, and multiple threads in thread pool B are responsible for image preprocessing, inference, and post-processing. The number of threads in the thread pool can be dynamically adjusted to adapt to different computational loads.

[0137] Before embedded inference, RGA is used to segment large-sized images into sub-blocks for super-resolution reconstruction. The sub-block stitching process is optimized, and overlapping region fusion technology is used to solve the influence of stitching lines. Specifically, before embedded inference, RGA is used to segment large-sized images into sub-blocks for super-resolution reconstruction; the sub-block stitching process is optimized, overlapping region fusion technology is used to solve the influence of stitching lines, and the computational cost is reduced by optimizing the sub-block processing order and stitching algorithm.

[0138] This embodiment integrates multiple hardware acceleration technologies to fully exploit hardware potential, significantly improving the inference speed of the model on embedded devices and thus enhancing its practicality. Validation on different test datasets demonstrates that the super-resolution reconstruction method of this invention not only possesses fast inference capabilities but also high portability, accelerating the development of deep learning super-resolution reconstruction in the embedded field.

[0139] In its implementation, the deployment module includes:

[0140] Extract image frames from the video file, load the quantized RKNN model file, and complete the model initialization;

[0141] The required memory resources are pre-allocated within the NPU, and the input and output memory information of the model is configured. In this embodiment, the input dimension is set to (1, 3, 1920, 1080), and the output dimension is set to (1, 3, 3840, 2160), corresponding to the resolution sizes of the input and output images, respectively.

[0142] Each frame of the image is normalized, and the channel order is converted to NHWC format. Then, the processed input data is written to the memory allocated by the NPU, and the super-resolution reconstruction model is run for inference computation.

[0143] After inference is complete, the generated data is copied from NPU memory back to the CPU for denormalization. Finally, the reconstructed super-resolution images are composited into a video and exported as an output file.

[0144] The acceleration module includes:

[0145] Hardware Image Processing Accelerator (RGA) is used to speed up image preprocessing for normalization and channel order conversion in the deployment module;

[0146] To improve inference efficiency, a parallel inference architecture based on multithreading (thread pool) was designed.

[0147] A schematic diagram of a multi-threaded inference structure is shown below. Figure 4 As shown, where:

[0148] Thread A is responsible for reading image frames from the video;

[0149] Thread C is responsible for saving the inference output results;

[0150] The number of threads in thread pool B is determined based on the model computation intensity, hardware memory, and computing power. In this embodiment, four threads are set to achieve parallel inference. Multiple threads in thread pool B are used for image preprocessing, model inference, and result post-processing.

[0151] Before inference on embedded devices, high-resolution images are cropped into multiple sub-blocks, and super-resolution reconstruction is performed on each sub-block separately. This strategy allows high-resolution images to be reconstructed using models with lower input resolution, reducing computational intensity and memory usage while ensuring the stability of super-resolution results.

[0152] In this embodiment, as Figure 5 The input image resolution is 1920×1080. This is for 2x super-resolution testing.

[0153] Resolution reconstruction employs the following cropping and stitching process:

[0154] The 1920×1080 input image is cropped into 4 sub-images, each with a resolution of 960×540;

[0155] Each sub-image is reconstructed using a 2x super-resolution method to generate four high-resolution sub-images with a resolution of 1920×1080.

[0156] Finally, the four reconstructed sub-images were stitched together into a high-resolution image with a resolution of 3840×2160;

[0157] Directly cropping a sub-image to 960×540 and then reconstructing and stitching it together may result in obvious stitching marks at the edges of the sub-images, affecting the visual effect.

[0158] This embodiment proposes an optimization scheme that introduces overlapping areas of sub-images by adjusting the cropping method to eliminate splicing marks;

[0159] The top-left corner of the input image is set as the origin. Four sub-images with a resolution of 970×550 are cropped. Each sub-image is cropped an additional 10 pixels in both width and height to create overlapping areas. The specific cropping range is as follows:

[0160] First sub-image: Width 0-970, Height 0-550;

[0161] Second sub-image: Width 950-1920, Height 0-550;

[0162] The third sub-image: width 0-970, height 530-1080;

[0163] Sub-image #4: Width 950-1920, Height 530-1080.

[0164] After super-resolution reconstruction of each sub-image, they are stitched together in the original cropping order, and overlapping areas are processed using a weighted average or seamless fusion algorithm.

[0165] This cropping and splicing method in this embodiment effectively avoids splicing marks while ensuring the integrity and quality of the final output image.

[0166] Example 2

[0167] To achieve the above objectives, see Figure 6 This embodiment also discloses an embedded NPU video super-resolution reconstruction system, including:

[0168] Data processing module: used to convert the original high-resolution image into a low-resolution image through a variety of combined degradation operations for use in model training;

[0169] Model building module: Reconstructs low-resolution images into super-resolution images using a lightweight image super-resolution generator and discriminator;

[0170] Model training module: The generator network is trained using an image training dataset. Low-resolution images are used as input to the generator network, and the corresponding original high-resolution image samples are used as the expected output of the generator network. The generator network is trained using the backpropagation algorithm. Low-resolution images and their corresponding high-resolution images are used as positive image pairs, and low-resolution images in the training sample dataset and the corresponding output images of the generator network are used as negative image pairs. The discriminator network is trained using the backpropagation algorithm.

[0171] Model quantization and compression module: used to convert trained super-resolution model files into intermediate ONNX format, and further convert ONNX format model files into RKNN format compatible with embedded devices;

[0172] Deployment module: Input low-resolution video, read each frame of the video using the OpenCV library, perform necessary preprocessing on each frame, and input it into the quantized model. The model will infer the input low-resolution image and load the model weight file obtained during training. After processing by the quantized model, the super-resolution image is output. Finally, the processed super-resolution image is recombined into the video using the OpenCV library, thus completing the reconstruction of the super-resolution video.

[0173] Acceleration module: Uses hardware acceleration methods to speed up the inference process for the deployed model.

[0174] Example 3

[0175] To achieve the above objectives, this embodiment also discloses an embedded NPU video super-resolution reconstruction device, including a processor and a storage medium;

[0176] The storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the above method.

[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An embedded NPU video super-resolution reconstruction method, characterized in that, Includes the following steps: S1. Data Processing: Convert the original high-resolution image into a low-resolution image through a combination of degradation operations to simulate the image degradation process in real-world scenarios for use in model training. S2. Model Construction: Low-resolution images are reconstructed into super-resolution images through a lightweight image super-resolution generator and discriminator. The generator uses depthwise separable convolution and an effective channel attention mechanism, combined with residual modules and feature fusion techniques, to significantly reduce model parameters and computational cost while maintaining high image reconstruction quality. The image reconstruction in step S2 includes: feeding the low-resolution image into the shallow feature extraction module of the image super-resolution generation network to obtain the shallow feature map of the image; The shallow feature map of the image is fed into the deep feature extraction module of the image super-resolution generation network to obtain the deep feature map of the image; The deep feature extraction module includes a stack of a preset number of residual modules. Each residual module is composed of a cascade of 1x1 convolution, 3x3 convolution, sigmoid and an effective channel attention mechanism, and introduces a depthwise separable convolution operation. The shallow and deep feature maps of the image are fed into the feature fusion module to obtain a fused feature map; The fused feature map is fed into the upsampling module of the image super-resolution generation network to obtain the super-resolution image of the low-resolution image; By inputting the super-resolution image and the high-resolution image into the image super-resolution discrimination network, the probability value of judging the super-resolution image as a real image can be obtained. The discriminant network is an image classification convolutional neural network that uses the LeakReLU activation function to prevent negative outputs from dying. The input image is processed through a classification convolutional neural network and then through two linear layers to obtain the output result. Based on the high-resolution image, the super-resolution image, and the probability values ​​output by the discriminator network, the loss functions of the generator network and the discriminator network are determined. S3. Model Training: The model is trained using an image training dataset. Low-resolution images are used as input to the generator network, and the corresponding original high-resolution image samples are used as the expected output of the generator network. The generator network is trained using the backpropagation algorithm. Low-resolution images and their corresponding high-resolution images are used as positive image pairs, and low-resolution images in the training sample dataset and the corresponding output images of the generator network are used as negative image pairs. The discriminator network is trained using the backpropagation algorithm. S4. Model Quantization and Compression: Convert the trained super-resolution model file into the intermediate ONNX format, and further convert the ONNX format model file into the embedded device compatible RKNN format. S5. Deployment: Input a low-resolution video, read each frame of the video using the OpenCV library, perform necessary preprocessing on each frame, and input it into the quantized model. The model will infer the input low-resolution image and load the model weight file obtained during training. After processing by the quantized model, the super-resolution image is output. Finally, the processed super-resolution image is reassembled into a video using the OpenCV library, thus completing the reconstruction of the super-resolution video. S6. Acceleration: Use hardware acceleration methods to speed up the inference process for the model deployed in step S5.

2. The embedded NPU video super-resolution reconstruction method according to claim 1, characterized in that, The data processing in step S1 includes: performing cropping, Gaussian blur, defocusing and motion blur simulation, and Gaussian noise addition on the original high-resolution image to generate a first reference image; JPEG compression is applied to the first reference image to introduce image quality compression, resulting in a second reference image; Bicubic interpolation downsampling is performed on the second reference image, and normalization is applied to obtain the degraded low-resolution image.

3. The embedded NPU video super-resolution reconstruction method according to claim 1, characterized in that, The loss function of the discriminant network is: The loss function of the generator network consists of three parts: in Using VGG loss: W and H represent the dimensions of the feature map in the VGG network, respectively; i and j refer to the j-th convolutional layer before max pooling in the i-th layer; The adversarial loss function of the generator network : in Used to evaluate the generated super-resolution image Content loss between the 1-norm distance and the original high-resolution image. , It is the weighting coefficient.

4. The embedded NPU video super-resolution reconstruction method according to claim 3, characterized in that, The model training in step S3 includes: optimizing and fine-tuning the model using the created video training data samples; First, obtain a batch of low-resolution image groups from the generated training data. The batch size is set to 32, meaning 32 low-resolution images are acquired at a time. Secondly, a corresponding super-resolution image set is generated using a generative network. ; Next, the generated Compared with the original high-resolution image group in the training data The corresponding input is sent to the discrimination network; Meanwhile, the model is tuned using the loss functions of the generator and the discriminator to complete the model training process.

5. The embedded NPU video super-resolution reconstruction method according to claim 1, characterized in that, The model quantization compression in step S4 also includes: during this conversion process, the input size, mean, normalization parameters and whether to perform quantization configuration options of the model need to be preset. Perform model quantization on the RKNN file, that is, convert floating-point numbers to fixed-point numbers for efficient operation on embedded devices.

6. The embedded NPU video super-resolution reconstruction method according to claim 1, characterized in that, The acceleration in step S6 includes: using the hardware image processing accelerator (RGA) of the embedded device to speed up image preprocessing. Concurrency is achieved through thread pools, thereby accelerating model inference and improving the real-time performance of the system. Before embedded inference, RGA is used to segment large images into sub-blocks for super-resolution reconstruction, the sub-block stitching process is optimized, and overlapping region fusion technology is used to solve the impact of stitching lines.

7. An embedded NPU video super-resolution reconstruction system, characterized in that, include: Data processing module: used to convert the original high-resolution image into a low-resolution image through a variety of combined degradation operations for use in model training; Model building module: Reconstructs low-resolution images into super-resolution images through a lightweight image super-resolution generator and discriminator; the image reconstruction includes: feeding the low-resolution image into the shallow feature extraction module of the image super-resolution generation network to obtain a shallow feature map of the image; The shallow feature map of the image is fed into the deep feature extraction module of the image super-resolution generation network to obtain the deep feature map of the image; The deep feature extraction module includes a stack of a preset number of residual modules. Each residual module is composed of a cascade of 1x1 convolution, 3x3 convolution, sigmoid and an effective channel attention mechanism, and introduces a depthwise separable convolution operation. The shallow and deep feature maps of the image are fed into the feature fusion module to obtain a fused feature map; The fused feature map is fed into the upsampling module of the image super-resolution generation network to obtain the super-resolution image of the low-resolution image; By inputting the super-resolution image and the high-resolution image into the image super-resolution discrimination network, the probability value of judging the super-resolution image as a real image can be obtained. The discriminant network is an image classification convolutional neural network that uses the LeakReLU activation function to prevent negative outputs from dying. The input image is processed through a classification convolutional neural network and then through two linear layers to obtain the output result. Based on the high-resolution image, the super-resolution image, and the probability values ​​output by the discriminator network, the loss functions of the generator network and the discriminator network are determined. Model training module: The generator network is trained using an image training dataset. Low-resolution images are used as input to the generator network, and the corresponding original high-resolution image samples are used as the expected output of the generator network. The generator network is trained using the backpropagation algorithm. Low-resolution images and their corresponding high-resolution images are used as positive image pairs, and low-resolution images in the training sample dataset and the corresponding output images of the generator network are used as negative image pairs. The discriminator network is trained using the backpropagation algorithm. Model quantization and compression module: used to convert trained super-resolution model files into intermediate ONNX format, and further convert ONNX format model files into RKNN format compatible with embedded devices; Deployment module: Input low-resolution video, read each frame of the video using the OpenCV library, perform necessary preprocessing on each frame, and input it into the quantized model. The model will infer the input low-resolution image and load the model weight file obtained during training. After processing by the quantized model, the super-resolution image is output. Finally, the processed super-resolution image is recombined into the video using the OpenCV library, thus completing the reconstruction of the super-resolution video. Acceleration Module: Uses hardware acceleration methods to speed up the inference process for the model deployed in the module.

8. An embedded NPU video super-resolution reconstruction device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; the processor is used to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 6.

Images