Lightweight video super-resolution method and system based on hypernetwork dynamic pruning

Abstract

The present application provides a lightweight video super-resolution method and system based on hypernetwork dynamic pruning, the method comprising: obtaining a video sequence of power inspection, the video sequence comprising multiple frames of images and timing information between the images; dividing each frame of the video sequence into multiple image blocks of fixed size; obtaining timing information between the multiple image blocks; constructing a layered patch diffusion model based on the timing information between the multiple image blocks; performing multi-scale dilated convolution processing on the multiple image blocks in the layered patch diffusion model through a dilated convolution layer to extract the spatiotemporal features of each image block to generate a global response feature; inputting the global response feature and the image block into a hypernetwork structure; deconvolving the feature map processed by the subnetwork into a high-resolution image block, and splicing and outputting a complete super-resolution video frame according to the original image position. By introducing the hypernetwork-driven dynamic pruning and spatiotemporal collaborative diffusion model, the model computational complexity and parameter scale are significantly reduced.

Description

Technical Field

[0001] The present application relates to the technical field of power equipment monitoring, and in particular to a lightweight video super-resolution method and system based on super-network dynamic pruning. Background Art

[0002] With the rapid development of applications such as intelligent power inspection, remote monitoring, and edge computing, high-definition video is becoming increasingly important in power equipment status perception and fault identification. However, due to the limited computing resources, storage bandwidth, and energy consumption of front-end devices (such as drones and edge nodes), the actual video captured and transmitted is often in low-resolution formats, which cannot meet the requirements for detail restoration and recognition accuracy in key areas. Therefore, achieving efficient and high-quality video super-resolution (VSR) in environments with limited computing resources has become an urgent problem.

[0003] Related video super-resolution methods primarily rely on large-scale convolutional neural networks (CNNs) or spatiotemporal transformation models. These methods suffer from common problems such as large parameter redundancy, high computational complexity, and unsuitability for mobile deployment, limiting their application in practical scenarios like power inspections. Furthermore, traditional static model structures cannot dynamically allocate computing resources based on the texture complexity and motion intensity of the input content, resulting in wasted computing power or reconstruction failures. Summary of the Invention

[0004] In view of this, it is necessary to provide a lightweight video super-resolution method and system based on hypernetwork dynamic pruning, which can overcome at least one of the above defects.

[0005] In a first aspect, an embodiment of the present application provides a lightweight video super-resolution method based on super-network dynamic pruning, which is applied to power inspection video processing. The method includes:

[0006] Acquire a video sequence of power inspection, wherein the video sequence includes multiple frames of images and timing information between the images;

[0007] Dividing each frame of image in the video sequence into a plurality of image blocks of fixed size;

[0008] Acquiring timing information between the plurality of image blocks;

[0009] constructing a hierarchical patch diffusion model based on temporal information between the plurality of image blocks;

[0010] Performing multi-scale dilated convolution processing on the plurality of image blocks in the layered patch diffusion model through a dilated convolution layer to extract spatiotemporal features of each image block;

[0011] The spatiotemporal features are weightedly fused through a Softmax gating mechanism to generate a global response feature;

[0012] Inputting the global response feature and the image block into a super-network structure, the super-network including a plurality of candidate sub-network structures with pruning modules and a channel mask generator, the super-network being used to dynamically predict a pruning strategy based on the feature complexity of the image block and generate a corresponding sub-network structure to perform adaptive inference;

[0013] The feature map processed by the sub-network is deconvolved and reconstructed into a high-resolution image block, and then spliced ​​according to the original image position to output a complete super-resolution video frame.

[0014] In one embodiment of the present application, the method further includes:

[0015] Constructing a multi-objective joint optimization function, wherein the multi-objective joint optimization function includes a reconstruction loss term, a sparsity regularization term, and a computational efficiency constraint term;

[0016] The formula of the multi-objective joint optimization function is:

[0017]

[0018] in, is the reconstruction loss term, is the sparse regularization term, is the computational efficiency constraint.

[0019] In one embodiment of the present application, the reconstruction loss term is used to constrain the pixel-level difference between the super-resolution result and the true high-resolution frame;

[0020] The expression of the reconstruction loss term is:

[0021]

[0022] in, represents a real high-resolution video frame, represents the super-resolution video frame reconstructed by the model, It represents the sum of squares of all pixel differences.

[0023] In one embodiment of the present application, the sparse regularization term is used to constrain the sparsity of the channel mask and the weight sparse matrix;

[0024] The expression of the sparse regularization term is:

[0025]

[0026] in, , are the weight coefficients of the channel pruning binary mask and the weight retention probability matrix, represents the L1 norm, is the channel pruning binary mask, The L1 norm of corresponds to the total number of retained channels, Keep the probability matrix for the weights, The L1 norm of corresponds to the sum of the cumulative probabilities of the retained weights.

[0027] In one embodiment of the present application, the computational efficiency constraint item is used to limit the total computational amount to be lower than a preset upper limit;

[0028] The expression of the computational efficiency constraint term is:

[0029]

[0030] in, Indicates the The floating point operations consumed by the block in the super-resolution inference stage, is a hyperparameter, and .

[0031] In one embodiment of the present application, the multi-scale dilated convolution processing includes:

[0032] Extracting multi-scale features through the cascaded dilated convolutional layers, wherein the dilation rate of the dilated convolutional layers increases exponentially;

[0033] Perform linear mapping on each scale feature to generate an importance score;

[0034] Weighting the importance scores by a weight matrix and a bias term;

[0035] The importance score is normalized into a dynamic weight coefficient through the Softmax function, and the features under different void rates are weighted and fused. The formula for normalizing the importance score is:

[0036]

[0037] in, Indicates the The weight coefficient corresponding to the void rate is is the importance score, is the learnable weight matrix, is the global average pooling operation, is the bias term.

[0038] In one embodiment of the present application, the structured channel pruning adjustment includes:

[0039] The number of input and output channels of each layer of the backbone network is dynamically trimmed according to the channel mask, and the channel retention ratio is:

[0040]

[0041] in, Reserve ratio for the channel, Channel mask , is the original number of channels in a certain layer of the backbone network, Indicates the number of channels retained for statistics;

[0042] The number of channels after cropping is:

[0043]

[0044] in, is the original number of channels, This is a rounding operation to ensure that the number of channels after clipping is an integer;

[0045] The parameter reduction ratio is:

[0046]

[0047] represents the product of the retention ratios of all layers, is the ratio of reduction in the number of model parameters.

[0048] In one embodiment of the present application, the generation of the channel mask includes:

[0049] Perform global average pooling on the global response features to generate a channel feature vector;

[0050] The channel feature vector is mapped to the corresponding probability value through the Sigmoid function, which serves as the binarization threshold of the channel mask;

[0051] The number of input and output channels of each layer of the backbone network is dynamically trimmed according to the channel mask.

[0052] In one embodiment of the present application, the dynamic prediction pruning strategy based on the feature complexity of the image block includes:

[0053] The intra-frame gradient intensity of the image block is calculated by the Sobel operator to quantify the texture complexity. The formula is:

[0054]

[0055] in, For the image block at position The gradient, is the L2 norm;

[0056] The motion amplitude is quantified by the difference norm of adjacent frames, and the formula is:

[0057]

[0058] in, is the Frobenius norm;

[0059] The block-level channel retention rate is generated according to the weighted sum of the intra-frame gradient strength and the motion amplitude, and the formula is:

[0060]

[0061] in, is the benchmark calculation amount, is the intra-frame gradient strength, is the range of motion, Used to smooth intensity changes.

[0062] In a second aspect, an embodiment of the present application provides a lightweight video super-resolution system based on hypernetwork dynamic pruning, the system comprising:

[0063] An image acquisition module is used to acquire a video sequence of power inspection, wherein the video sequence includes multiple frames of images and timing information between the images;

[0064] An image processing module is configured to divide each frame of the video sequence into a plurality of image blocks of a fixed size; obtain timing information between the plurality of image blocks; and construct a layered patch diffusion model based on the timing information between the plurality of image blocks;

[0065] a dilated convolution layer, configured to perform multi-scale dilated convolution processing on the plurality of image blocks within the hierarchical patch diffusion model to extract the spatiotemporal features of each image block; and to perform weighted fusion of the spatiotemporal features through a Softmax gating mechanism to generate a global response feature;

[0066] A hypernetwork structure for obtaining the global response features, the hypernetwork comprising a plurality of candidate subnetwork structures with prunable modules and a channel mask generator, the hypernetwork being configured to dynamically predict pruning strategies based on the feature complexity of the image blocks, generate corresponding subnetwork structures, and perform adaptive inference;

[0067] The image restoration module is used to perform deconvolution on the feature map processed by the sub-network and reconstruct it into a high-resolution image block, and splice it according to the original image position to output a complete super-resolution video frame.

[0068] The embodiments of this application provide a lightweight video super-resolution method and system based on hypernetwork dynamic pruning. By introducing hypernetwork-driven dynamic pruning and a spatiotemporal co-diffusion model, this method significantly reduces the model computational complexity and parameter size while ensuring super-resolution results. This method is suitable for edge computing scenarios. The use of a content-aware mechanism dynamically allocates computing resources, improving the reconstruction quality of key areas. The system offers advantages such as high efficiency, lightweightness, and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flowchart of a lightweight video super-resolution method based on hypernetwork dynamic pruning provided in one embodiment of the present application.

[0070] Figure 2 A schematic diagram of a lightweight video super-resolution system module based on hypernetwork dynamic pruning provided in one embodiment of the present application.

[0071] Figure 3 A schematic diagram of an electronic device provided in one embodiment of the present application.

[0072] Description of main component symbols

[0073] Lightweight video super-resolution system based on hypernetwork dynamic pruning10

[0074] Image acquisition module 100

[0075] Image processing module 200

[0076] Atrous convolution layer 300

[0077] Super Network Structure 400

[0078] Image restoration module 500

[0079] Electronic devices 20

[0080] Processor 21

[0081] Memory 22

[0082] Method steps S100-800 DETAILED DESCRIPTION

[0083] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0084] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art to which this application relates. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0085] It should be noted that, in the embodiments of the present application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order. Features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0086] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0087] With the rapid development of applications such as intelligent power inspection, remote monitoring, and edge computing, high-definition video is becoming increasingly important in power equipment status perception and fault identification. However, due to the limited computing resources, storage bandwidth, and energy consumption of front-end devices (such as drones and edge nodes), the actual video captured and transmitted is often in low-resolution formats, which cannot meet the requirements for detail restoration and recognition accuracy in key areas. Therefore, achieving efficient and high-quality video super-resolution (VSR) in environments with limited computing resources has become an urgent problem.

[0088] Related technologies for video super-resolution primarily rely on large-scale convolutional neural networks (CNNs) or spatiotemporal transformation models. While these methods have achieved significant progress in improving image quality, they suffer from widespread issues such as excessive model parameter redundancy, high computational complexity, and unsuitability for mobile deployment, limiting their application in practical scenarios like power inspections. Furthermore, traditional static model structures are unable to dynamically allocate computing resources based on the texture complexity and motion intensity of the input content, resulting in wasted computing power or reconstruction failures.

[0089] The embodiments of this application provide a lightweight video super-resolution system based on hypernetwork dynamic pruning. By introducing hypernetwork-driven dynamic pruning and a spatiotemporal co-diffusion model, this system significantly reduces the model computational complexity and parameter size while ensuring effective super-resolution. This system is suitable for edge computing scenarios. The use of a content-aware mechanism dynamically allocates computing resources, improving the reconstruction quality of key areas. The system offers advantages such as high efficiency, lightweightness, and adaptability.

[0090] HyperNetwork and dynamic pruning technologies are gaining increasing attention. They can dynamically generate model structures or adjust computational paths based on input data characteristics, improving model adaptability and efficiency. Furthermore, content-aware computing is being used to adaptively allocate computing resources based on the importance of image regions, ensuring reconstruction quality in key areas while reducing the overall computational burden.

[0091] Figure 1 FIG. 1 is a flow chart of a lightweight video super-resolution method based on dynamic pruning of a super-network according to an embodiment of the present application. Figure 1 The lightweight video super-resolution method based on dynamic pruning of a hypernetwork shown includes at least the following steps: S100: obtaining a video sequence of power inspection, the video sequence including multiple frames of images and timing information between images; S200: dividing each frame of image in the video sequence into multiple image blocks of fixed size; S300: obtaining timing information between multiple image blocks; S400: constructing a layered patch diffusion model based on the timing information between multiple image blocks; S500: performing multi-scale hole convolution processing on multiple image blocks in the layered patch diffusion model through a hole convolution layer to extract the spatiotemporal features of each image block; S600: and weightedly fusing the spatiotemporal features through a Softmax gating mechanism to generate a global response feature; S700: inputting the global response feature and the image block into the hypernetwork structure; S800: deconvolving the feature map processed by the subnetwork into a high-resolution image block, and splicing it according to the original image position to output a complete super-resolution video frame.

[0092] S100: Acquire a video sequence of power inspection, where the video sequence includes multiple frames of images and timing information between the images.

[0093] In an embodiment of the present application, a video stream of power equipment is collected by a drone or a fixed camera as input data for subsequent processing. The system receives the video stream in real time from the acquisition device, and caches consecutive frames through a frame buffer module, while adding a timestamp to each frame to form an image sequence with time-series association. The video sequence undergoes preprocessing, including decoding, color space conversion (such as RGB to YUV), and pixel value normalization (mapping the pixel value range to [0,1]), and is finally stored in the form of a tensor for subsequent analysis. In addition, the system extracts pixel-level motion vectors of adjacent frames through optical flow estimation (such as the PWC-Net algorithm), generates a motion intensity matrix, and quantifies dynamic interference such as transmission line swing and cloud movement.

[0094] Specifically, the system transmits video streams in real time from a front-end acquisition device (such as a drone or a fixed camera), caches consecutive frames with timestamps through a frame buffer module, and forms an image sequence with sequential association.

[0095] Understandably, the core value of this step lies in providing the model with motion and texture information for consecutive frames. The frame buffer module's low-latency design ensures real-time processing capabilities for edge devices like drones, preventing inference stalls caused by cache overload. Furthermore, modeling temporal information (such as the motion intensity matrix) provides the foundation for spatiotemporal feature extraction in subsequent steps. This is particularly important for dynamic backgrounds such as rain and fog interference and sudden changes in illumination, which are common in power grid scenarios. This can enhance inter-frame consistency and reduce flicker or artifacts in super-resolution video.

[0096] S200: Divide each frame of an image in a video sequence into a plurality of image blocks of a fixed size.

[0097] In an embodiment of the present application, the lightweight video super-resolution method based on super-network dynamic pruning includes in step S200: dividing each frame image in the video sequence into multiple image blocks of fixed size.

[0098] Specifically, each frame of a video sequence is divided into multiple fixed-size image blocks, where the size is determined by both the scene requirements and the hardware's computing power. For example, on a drone edge device, a small 8×8 pixel block size is selected to reduce the computational effort per block; on the server side, a large 16×16 pixel block size is used to improve the reconstruction quality of large-scale structures such as transmission tower trusses. The block operation uses a non-overlapping segmentation strategy, dividing the image into several blocks and labeling each block with a position index. If the resolution is not divisible by the block size, the image edges are mirrored to avoid information loss.

[0099] As can be seen, the blocking strategy is designed to balance computational efficiency and reconstruction quality. Small block sizes (e.g., 8×8) significantly reduce the computational complexity of a single block, adapting to the computing power constraints of drones in power inspection scenarios. Large block sizes (e.g., 16×16) improve global consistency for highly repetitive structures (such as towers and conductor arrays) by reducing inter-block inconsistencies. Furthermore, non-overlapping blocks reduce redundant computations but may introduce blocking artifacts that must be compensated for through boundary smoothing and fusion in subsequent steps.

[0100] S300: Acquire timing information between multiple image blocks.

[0101] In an embodiment of the present application, the lightweight video super-resolution method based on super-network dynamic pruning includes in step S300: obtaining timing information between multiple image blocks.

[0102] Specifically, the method quantifies the motion intensity and texture complexity of image blocks by analyzing the differences between adjacent frames. For motion intensity, the system calculates the pixel-level differences of image blocks in the same position to generate an inter-frame difference matrix. This matrix is ​​then combined with the optical flow estimation results to produce a motion intensity heat map. For example, the differential values ​​in the area where the transmission line is swinging are large, corresponding to higher motion intensity. For texture complexity, the method uses the Sobel operator to extract the gradient information of the image block and calculates the mean gradient amplitude as a texture complexity indicator. High-texture areas such as insulator cracks have gradient values ​​significantly higher than the sky background, and are therefore marked as high-complexity blocks.

[0103] It's understandable that modeling timing information provides a basis for dynamic resource allocation. Regions with high motion intensity (such as swaying conductors) require enhanced timing consistency to avoid jitter between frames after super-resolution. Regions with high texture complexity (such as insulator cracks) require more computing resources to ensure clear reconstruction of millimeter-level defects. Furthermore, the combined perception of motion intensity and texture complexity (e.g., block-level computational intensity coefficients) lays the foundation for adaptive computational resource allocation in subsequent steps, enabling precise deployment of computing power.

[0104] S400: Constructing a hierarchical patch diffusion model based on temporal information between multiple image blocks.

[0105] In an embodiment of the present application, the lightweight video super-resolution method based on super-network dynamic pruning includes in step S400: constructing a hierarchical patch diffusion model according to the temporal information between multiple image blocks.

[0106] Specifically, the Hierarchical Patch Diffusion Model (HPDM) uses image patches as basic units and enhances spatiotemporal features through cross-frame information diffusion. The model adopts a U-Net architecture, consisting of an encoder, a diffusion layer, and a decoder. The encoder extracts shallow features through depthwise separable convolutions. The diffusion layer uses cascaded dilated convolutions to capture multi-scale context (with exponentially increasing dilation rates, such as 1, 2, 4, and 8). The decoder then progressively upsamples to generate preliminary high-resolution patches. During the diffusion process, the model iteratively passes messages between local patches, integrating spatial dependencies and temporal associations across frames. For example, the wire joint area in the current frame references the corresponding block features in the adjacent frame to correct artifacts caused by sudden changes in illumination.

[0107] The HPDM design addresses the conflict between highly repetitive structures and dynamic interference in power transmission scenarios. Cascaded dilated convolutions capture both local details and global structure of transmission lines through multi-scale receptive fields, while the cross-frame diffusion mechanism leverages spatiotemporal consistency to optimize reconstruction quality under dynamic interference. For example, in frames affected by rain and fog, the model enhances the robustness of texture details through diffusion, preventing reconstruction failures caused by local blur.

[0108] S500: Perform multi-scale dilated convolution processing on multiple image blocks in the hierarchical patch diffusion model through a dilated convolution layer to extract the spatiotemporal features of each image block.

[0109] In an embodiment of the present application, the lightweight video super-resolution method based on hypernetwork dynamic pruning includes in step S500: performing multi-scale dilated convolution processing on multiple image blocks in the hierarchical patch diffusion model through a dilated convolution layer to extract the spatiotemporal features of each image block.

[0110] Specifically, multi-scale dilated convolutions are used to extract features from image patches in the HPDM. The kernel size of the dilated convolutions increases exponentially (e.g., 1, 2, 4, 8), gradually expanding the receptive field while maintaining feature map resolution. For example, a dilated convolution ratio of 1 in the first layer focuses on local textures (e.g., cracks on the insulator surface); while a dilated convolution ratio of 8 in the final layer captures global structure (e.g., the alignment of a transmission line). Features at different dilated ratios are linearly mapped to generate importance scores, which are then normalized to dynamic weights using the Softmax function for weighted fusion of multi-scale features.

[0111] Specifically, for low-resolution video frames , multi-granularity block processing is performed to adapt to the complex background requirements of highly repetitive structures (such as tower trusses, wire arrays) and dynamic interference (such as rain, fog, and light fluctuations) in power inspection videos. The video frame is divided into N × N Non-overlapping blocks (in the present invention N =8), each block is recorded as ,in Represents the spatial index of the block in the original image, is the time step. The block formula is:

[0112]

[0113] For example, a video frame with a resolution of 256×256 is divided into 32×32 8×8 blocks, covering all spatial locations.

[0114] A lightweight diffusion network, χ, was constructed, with only 30% of the parameters of traditional models. This network consists of three layers of depthwise separable convolutions and ReLU activation functions, extracting spatiotemporal features layer by layer.

[0115] Enter the current frame block:

[0116]

[0117] in, is the network parameter, including 3 layers of depth-wise separable convolution and ReLU activation function.

[0118] It is understandable that this application uses spatiotemporal consistency to optimize reconstruction quality under dynamic interference by inputting the current frame block and its adjacent frame blocks (previous and next frames). For example, in frames with swaying power lines, the network suppresses blur or jitter caused by motion by fusing the stable structural features of adjacent frames; in frames disturbed by rain and fog, the network enhances the robustness of texture details by using multi-frame gradient information. Compared to traditional convolution, depthwise separable convolution reduces the number of parameters by over 88% (for example, with 64 input channels, 128 output channels, and a kernel size of 3×3, traditional convolution requires 73,728 parameters, while depthwise separable convolution only requires 8,768), significantly reducing memory usage and power consumption on edge devices. At the same time, the ReLU activation function accelerates feature sparsification and reduces inefficient computation.

[0119] Understandably, the design of multi-scale dilated convolution aims to balance local details with global structure. In power scenarios, high-frequency textures (such as surface corrosion on conductors) and low-frequency structures (such as the overall line alignment) of transmission lines need to be modeled simultaneously. The cascaded dilated ratio design enables the model to simultaneously extract the micro-texture of insulator cracks and the macro-structure of tower trusses. The dynamic weight fusion mechanism automatically adjusts the contribution of multi-scale features based on content complexity, for example, enhancing the weight of local details in complex areas (such as insulator strings) and emphasizing global consistency in simple areas (such as the sky).

[0120] S600: The spatiotemporal features are weightedly fused through the Softmax gating mechanism to generate global response features.

[0121] In an embodiment of the present application, the lightweight video super-resolution method based on dynamic pruning of a super-network includes in step S600: performing weighted fusion of spatiotemporal features through a Softmax gating mechanism to generate a global response feature.

[0122] Specifically, a Softmax gating mechanism is used to perform weighted fusion of multi-scale features. Specifically, the model performs global average pooling on the features output by the dilated convolution to generate a channel feature vector. This is then transformed into an importance score through a linear transformation and normalized into a weight coefficient using the Softmax function. The weight coefficient reflects the contribution of features at each scale to the final reconstruction. For example, in areas of wire corrosion, local detail features are given a higher weight, while in areas with uniform backgrounds, global structural features dominate. The weighted fusion of global response features preserves key details while suppressing noise caused by dynamic interference.

[0123] It is understandable that the introduction of the gating mechanism improves the model's adaptability. In power scenarios, there are significant differences in the texture complexity and motion intensity of transmission lines, and traditional fixed-weight fusion methods have difficulty balancing detail enhancement and global consistency. This step uses dynamic weight allocation to enable the model to automatically adjust the fusion ratio of multi-scale features based on the input content. For example, in frames where the wires swing violently, the model prioritizes retaining temporal consistency features; in static background frames, the model focuses on enhancing local textures, thereby improving the visual quality of the reconstructed video and the accuracy of defect detection.

[0124] S700: Input the global response features and image blocks into the hypernetwork structure.

[0125] In the embodiment of the present application, the lightweight video super-resolution method based on dynamic pruning of a hypernetwork includes, in step S700: inputting the global response feature and the image block into the hypernetwork structure. The hypernetwork includes multiple candidate subnetwork structures with prunable modules and a channel mask generator. The hypernetwork is used to dynamically predict the pruning strategy based on the feature complexity of the image block and generate the corresponding subnetwork structure to perform adaptive inference.

[0126] Specifically, global response features and image patches are fed into a hypernetwork, which dynamically generates a pruning strategy. The hypernetwork, comprised of a lightweight encoder, extracts input features through shallow convolutions and generates a channel mask (a binary vector where 0 indicates pruning and 1 indicates retention) and a weight retention probability matrix (which controls the sparsification of convolution kernel weights). The channel mask dynamically adjusts the number of channels based on the texture complexity of the image patch, for example, retaining more channels for high-complexity patches. The weight retention probability matrix prunes redundant weights by entropy sorting, for example, forcibly removing low-probability weights in simple background patches, retaining only critical connections.

[0127] Understandably, dynamic pruning strategies address the conflict between model redundancy and performance degradation in traditional methods. In power scenarios, computational requirements vary significantly across different regions (such as insulator cracks and the sky background). By using hypernetwork prediction to prune parameters, the model retains more channels and weights in complex areas to maintain accuracy, while significantly reducing computational effort in simpler areas. For example, on drone edge devices, the model can reduce parameter count by over 40% while maintaining a stable PSNR, ensuring high-definition reconstruction within limited computing power.

[0128] S800: Deconvolve the feature map processed by the sub-network to reconstruct it into a high-resolution image block, and splice it according to the original image position to output a complete super-resolution video frame.

[0129] In an embodiment of the present application, the lightweight video super-resolution method based on dynamic pruning of a super-network includes in step S800: deconvolving the feature map processed by the sub-network into a high-resolution image block, and splicing and outputting a complete super-resolution video frame according to the original image position.

[0130] Specifically, the pruned subnetwork output is upsampled to generate high-resolution image blocks. Specifically, deconvolution or sub-pixel convolution is used to upsample the feature map by a factor of two or four to restore image detail. The system then stitches the high-resolution blocks together into a complete frame based on the positional indices of the original image blocks. Block boundaries are smoothed (e.g., using weighted averaging or fusion of overlapping regions) to eliminate artifacts of block reconstruction. The resulting super-resolution video frames are compressed using H.265 encoding and transmitted in real time to the monitoring center.

[0131] Understandably, this step achieves the unification of lightweight compression and high-quality reconstruction. In power inspection scenarios, drones have limited bandwidth and storage capacity, and video volume must be compressed while ensuring defect detection accuracy. Through block upsampling and boundary smoothing, the system reduces computational complexity while avoiding blocking effects, achieving a balance between visual quality and algorithm performance. For example, the detailed reconstruction of the crack area of ​​the insulator meets the requirements of millimeter-level defect detection, while the compression processing of the sky background significantly reduces storage and transmission overhead, adapting to the real-time monitoring and remote transmission scenarios of power inspections.

[0132] In an embodiment of the present application, the lightweight video super-resolution method based on super-network dynamic pruning also includes: constructing a multi-objective joint optimization function, which includes a reconstruction loss term, a sparse regularization term, and a computational efficiency constraint term.

[0133] The formula of the multi-objective joint optimization function is:

[0134]

[0135] in, is the reconstruction loss term, is the sparse regularization term, is the computational efficiency constraint.

[0136] Specifically, this method constructs a multi-objective joint optimization function to simultaneously optimize the reconstruction quality, model sparsity and computational efficiency in the power inspection video super-resolution task. The function consists of three parts: reconstruction loss term , the sparse regularization term , calculation efficiency constraint The three loss terms are collaboratively optimized through weighted summation to ensure lightweight compression while maintaining reconstruction accuracy in key areas and strictly meeting the computing power limitations of edge devices.

[0137] In the embodiment of the present application, the reconstruction loss term is used to constrain the pixel-level difference between the super-resolution result and the true high-resolution frame.

[0138] The expression of the reconstruction loss term is:

[0139]

[0140] in, represents a real high-resolution video frame, represents the super-resolution video frame reconstructed by the model, It represents the sum of squares of all pixel differences.

[0141] Specifically, the reconstruction loss term measures the pixel-level difference between the super-resolution video frame output by the model and the true high-resolution frame. The pixel-by-pixel difference between the two is constrained using the mean squared error (MSE). This loss term drives the model to learn the mapping from low-resolution input to high-resolution output, specifically targeting critical defects in power scenarios (such as insulator cracks and conductor corrosion), ensuring accurate reconstruction of detailed textures.

[0142] Understandably, power inspection videos require precise identification of millimeter-level defects, making pixel-level accuracy crucial. The MSE loss directly optimizes the visual quality and geometric consistency of the super-resolution results by minimizing pixel-by-pixel differences. Compared to perceptual losses (such as VGG feature matching), MSE focuses more on local details and is therefore suitable for tasks sensitive to geometric accuracy, such as defect detection.

[0143] In an embodiment of the present application, the sparse regularization term is used to constrain the sparsity of the channel mask and the weight sparse matrix.

[0144] The expression of the sparse regularization term is:

[0145]

[0146] in, , are the weight coefficients of the channel pruning binary mask and the weight retention probability matrix, represents the L1 norm, is the channel pruning binary mask, The L1 norm of corresponds to the total number of retained channels, Keep the probability matrix for the weights, The L1 norm of corresponds to the sum of the cumulative probabilities of the retained weights.

[0147] Specifically, this part of the loss constrains the sparsity of the channel mask and the weight retention probability matrix through L1 regularization. The channel mask is a binary vector (elements are 0 or 1) that indicates whether the input / output channels of each layer of the backbone network are retained. Its L1 norm is equal to the total number of retained channels. The weight retention probability matrix is ​​a real number matrix (element range [0,1]) that represents the retention probability of the convolution kernel weights. Its L1 norm is the cumulative value of all retention probabilities, reflecting the sparsity of the weights. Hyperparameters The penalty strengths for channel pruning and weight sparsification are controlled separately and need to be adjusted dynamically based on hardware resources and task requirements.

[0148] It's understandable that L1 regularization of the channel mask compresses model parameters by reducing the number of retained channels, while L1 regularization of the weight-preserving probability matrix further eliminates redundant connections, achieving unstructured pruning. The combined constraints of these two significantly reduce computational complexity and memory usage while ensuring reconstruction quality in key areas. For example, on drone edge devices, this strategy can reduce model parameters by over 40% while maintaining a stable PSNR.

[0149] In the embodiment of the present application, the computational efficiency constraint term is used to limit the total computational amount to be lower than a preset upper limit;

[0150] The expression for calculating the efficiency constraint is:

[0151]

[0152] in, Indicates the The floating point operations consumed by the block in the super-resolution inference stage, is a hyperparameter, and .

[0153] Specifically, the computational efficiency constraint, partial loss, limits the model's total computational cost to below the hardware limit through a squared penalty. This loss term rises sharply as the total computational cost approaches the hardware limit, forcing the model to prioritize reducing the computational cost of complex blocks. For example, a lower FLOPs budget is allocated to the sky background area, while reserving more computational resources for the insulator crack area.

[0154] It is understandable that in power inspection scenarios, the computing power of drones or edge devices is limited, and it is necessary to strictly avoid inference delays or task failures caused by excessive computational load in super-resolution models. The square penalty term has stronger dynamic adaptability than traditional linear constraints: when the computational load approaches the upper limit, the loss value increases exponentially, forcing the model to quickly adjust the pruning strategy instead of relying on manual parameter adjustment. For example, when hardware resources are tight, the system will actively reduce the channel retention rate in simple areas. β , giving priority to ensuring the reconstruction quality of complex areas.

[0155]

[0156] Table 1: Experimental results under different parameter combinations

[0157] Please refer to Table 1. This example uses power inspection videos as test subjects to verify the effectiveness of a lightweight video super-resolution method based on multi-objective joint optimization. The experiment utilizes a dataset of transmission line videos captured by a drone at high altitude, including typical defect scenarios such as insulator cracks and conductor corrosion. A grid search method is used to iterate over parameter combinations, calculating the PSNR (dB), FLOPs (G), and score for each parameter combination.

[0158] Table 1 shows the experimental results for different parameter combinations, with the comparison between three typical parameter combinations being particularly striking. The first parameter set generated a model with 14.8 GB of FLOPs, exceeding the hardware limit (10 GB). According to the scoring formula, the squared penalty caused the score to drop sharply from 32.5 to 23.28, indicating that the model was eliminated due to a severe overshoot. The second parameter set generated 10.5 GB of FLOPs, which, while close to the limit, still yielded a score of 31.80, lower than the third set (31.10). This indicates that even slight overshoots can trigger strong penalties, forcing the model to further reduce computational overhead.

[0159] The third set of parameters generates 9.8 GB of FLOPs, fully adapting to the computing power constraints of edge devices. The linear penalty is reduced by only 0.1 points (31.0 → 31.1), demonstrating that the model can maintain high reconstruction quality even with sufficient resources. Compared to other combinations, the third set of parameters reduces the computational load to 9.8 GB while maintaining a PSNR of 31.0 dB, a parameter reduction of 45%, significantly outperforming traditional pruning methods (≈30%).

[0160] In this embodiment, a multi-objective joint optimization strategy is adopted. On the edge device of the drone, the model compresses FLOPs to 9.8G through dynamic pruning, while maintaining PSNR = 31.0 dB, meeting the requirements of millimeter-level defect detection.

[0161] In low-complexity areas (such as the sky background), the model automatically reduces the channel retention rate β to 0.4 to reduce redundant computation. In high-complexity areas (such as tower trusses), β is increased to 0.8, ensuring high-definition reconstruction of critical structures. The model achieves inference speed of 28fps (9.8GB of FLOPs), meeting the 30fps real-time requirement of the drone. Compared to the unoptimized baseline model (14.8GB of FLOPs and 18fps inference speed), this solution improves computational efficiency by 55% and increases defect detection accuracy (mAP) from 85.7% to 91.7%.

[0162] In an embodiment of the present application, the multi-scale dilated convolution processing includes: extracting multi-scale features through cascaded dilated convolution layers, and the dilated rate of the dilated convolution layer increases exponentially. Each scale feature is linearly mapped to generate an importance score. The importance score is weighted by a weight matrix and a bias term. The importance score is normalized to a dynamic weight coefficient through a Softmax function, and features at different dilated rates are weighted and fused. The formula for importance score normalization is:

[0163]

[0164] in, Indicates the The weight coefficient corresponding to the void rate is is the importance score, is the learnable weight matrix, is the global average pooling operation, is the bias term.

[0165] Specifically, the system adopts multiple dilated convolution modules, and the dilation rate of each module increases exponentially (e.g. d =1,2,4,8), gradually expanding the receptive field. The first layer with a voiding rate of 1 focuses on the local texture of the image patch (such as the edge details of a crack on the surface of an insulator); the second layer with a voiding rate of 2 captures the distribution of textures over a slightly larger area (such as the local structure of a conductor joint); the third layer with a voiding rate of 4 focuses on the geometric features of a larger area (such as the direction of a transmission line); and the fourth layer with a voiding rate of 8 finally covers the global information of the entire image patch (such as the overall layout of the tower and conductor array). Through this exponentially increasing design, the model can gradually integrate multi-scale information from micro to macro without reducing the resolution of the feature map, avoiding the loss of detail caused by traditional downsampling.

[0166] It is understandable that after feature extraction is completed, the system performs global average pooling (GAP) on the feature maps at each scale to generate a channel feature vector. Subsequently, the feature vector is converted into an importance score through a linear mapping (weight matrix and bias term), quantifying the contribution of the features at that scale to the final reconstruction. The calculation of the importance score combines the statistical information of the channel features with learnable parameters, enabling the model to dynamically adjust the priority of features at each scale based on the input content. For example, in the crack area of ​​an insulator with high texture complexity, the importance score of local detail features (low void ratio) is significantly higher than that of global features (high void ratio), while in the sky background area with uniform illumination, the weight of global features is even higher. Finally, the system inputs the importance scores of all scales into the Softmax function for normalization, generating dynamic weight coefficients to guide the weighted fusion of multi-scale features.

[0167] It is understandable that the core goal of the dynamic weight fusion mechanism is to adaptively adjust the contribution ratio of multi-scale features according to the image content, so as to achieve efficient and accurate reconstruction in complex power scenes. In power inspection videos, there are significant differences in the texture complexity and motion intensity of transmission lines: in static background areas (such as the main body of the tower), the model needs to prioritize retaining the sharpness of local textures; while in areas of dynamic interference (such as wire swings, rain and fog occlusion), it is necessary to strengthen inter-frame consistency and global structural stability. Traditional fixed-weight multi-scale fusion methods are difficult to meet this dynamic requirement. This application generates dynamic weights through Softmax normalization, which enables the model to perceive changes in input content in real time and adjust feature fusion strategies.

[0168] For example, in insulator crack regions, local detail features (low void ratio) have a higher importance score, and the normalized weight coefficient significantly amplifies the contribution of features at this scale, ensuring clear reconstruction of the crack edge. In straight sections of transmission lines, global structural features (high void ratio) are weighted more heavily, enhancing the continuity and geometric consistency of the line's trajectory. This dynamic adjustment not only improves reconstruction quality but also optimizes the allocation of computing resources: for simple background areas, the model automatically reduces the weight of low void ratio features to reduce redundant computation; while for critical defect areas, computing power is concentrated to improve detail resolution. Experiments show that the dynamic weight fusion mechanism improves the model's PSNR by approximately 1.2dB in power scenarios, while reducing inter-frame jitter by 30%, significantly outperforming fixed weight fusion methods.

[0169] Furthermore, the design of dynamic weights enhances the model's robustness to complex interference. In frames subject to rain, fog, or sudden changes in illumination, local texture features may become ineffective due to blurring. In these cases, the weights of global structural features are automatically enhanced to avoid reconstruction failures caused by the failure of a single feature. For example, when the surface texture of an insulator is blurred due to rain or fog in a frame, the model prioritizes preserving global structural features to ensure that the reconstructed insulator outline remains consistent with adjacent frames, thus avoiding inter-frame flickering or artifacts. This dynamic adaptability enables the model to output stable, clear super-resolution video even in complex and dynamic backgrounds, meeting the high-precision defect detection requirements of power inspections.

[0170] The implementation of dynamic weight fusion relies not only on mathematical normalization design but also on ensuring engineering success through end-to-end training and hardware adaptation strategies. During training, the weight matrix and bias terms are jointly optimized with the backbone network, enabling the model to automatically learn the optimal weight distribution based on the task objective (such as defect detection accuracy). For example, in the early stages of training, the model may assign higher weights to local detail features to achieve rapid convergence. As training progresses, the weight of global structural features is gradually increased to enhance inter-frame consistency. This progressive optimization strategy allows the model to maintain local accuracy while avoiding global structural distortion or loss.

[0171] During the deployment phase, dynamic weight fusion is further optimized in conjunction with the hardware computing power of edge devices. For example, on drone edge devices, the system dynamically adjusts the initialization strategy of the weight matrix to prioritize low-voiding rate features (reducing computational effort) and enable high-voiding rate features only in complex areas. On the server side, the model fully utilizes the complete information of multi-scale features to achieve higher reconstruction quality. This hierarchical adaptation mechanism enables the same model to flexibly respond to tasks under varying computing power conditions, meeting the real-time requirements of drones while supporting high-precision analysis at ground stations.

[0172] It is understandable that the dynamic weight fusion mechanism has demonstrated significant technical advantages in power inspection scenarios. First, it solves the problem of the difficulty in balancing local details and global structures in traditional multi-scale fusion methods. By dynamically adjusting weights, the model enhances local features in complex texture areas (such as insulator cracks) and strengthens global consistency in uniform background areas (such as the sky), thereby achieving cross-scale detail enhancement and structural stability. Secondly, the dynamic weight mechanism and subsequent pruning strategy form a synergistic optimization: the channels and weights corresponding to high-weight features are retained first, and low-weight features are further compressed through pruning. For example, in simple background areas, the model may only retain 2-3 convolutional layers with a void rate, while enabling all scale features in complex areas, thereby achieving fine-grained computing power allocation.

[0173] In addition, dynamic weight fusion significantly improves the model's robustness to dynamic interference in power scenarios. In frames where transmission lines are swinging or illumination changes suddenly, the model dynamically adjusts weights to prioritize temporal consistency features (such as structural correlations between adjacent frames) and suppress local noise. For example, when a frame contains a conductor that is violently swinging due to strong winds, the model increases the weight of features with high void ratios and uses global structural information to correct local deformations, thus avoiding inter-frame jitter in the super-resolution results. Experiments show that this mechanism improves the model's reconstruction stability by 40% under dynamic interference, while maintaining millimeter-level accuracy for key defects. This capability is crucial for power inspection tasks, as videos collected by drones are often affected by weather and environmental interference, requiring the model to have dynamic adaptability to ensure the reliability of defect detection.

[0174] In the embodiment of the present application, the structured channel pruning adjustment includes: dynamically cutting the number of input and output channels of each layer of the backbone network according to the channel mask. The channel retention ratio is:

[0175]

[0176] in, Reserve ratio for the channel, Channel mask , is the original number of channels in a certain layer of the backbone network, Indicates the number of channels reserved for statistics.

[0177] Specifically, the core of structured channel pruning is to dynamically trim the number of input / output channels of each layer of the backbone network according to the feature complexity of the image block. Specifically, the system first generates a channel mask through the hypernetwork. Each element of the channel mask represents the first i The channels are retained (1) or pruned (0). Then, the channel retention ratio is calculated. For example, if the original number of channels of a layer is 64 and 50 channels are retained, then β =50 / 64=0.78125. This ratio reflects the demand for the number of channels for the texture complexity and motion intensity of the current image block - the high complexity area (such as insulator cracks) β Higher values ​​lead to lower complexity areas (such as the sky background). β The value is lower.

[0178] The number of channels after cropping is:

[0179]

[0180] in, is the original number of channels, This is a rounding operation to ensure that the number of channels after cropping is an integer. For example, if the original number of channels in a layer is 128 and β = 0.75, the number of channels after cropping is 128 × 0.75 = 96. The purpose of the rounding operation is to avoid decimals in the number of channels (such as 64.3 channels), thereby ensuring the feasibility of the model calculation.

[0181] The parameter reduction ratio is:

[0182]

[0183] represents the product of the retention ratios of all layers, is the ratio of model parameter reduction. For example, if a model has three layers, with β values ​​of 0.8, 0.75, and 0.7, the parameter reduction ratio is 1 − (0.8 × 0.75 × 0.7) = 1 − 0.42 = 0.58, representing a 58% parameter reduction. This formula intuitively reflects the cumulative effect of multi-layer pruning—small changes in single-layer pruning are multiplied and directly affect the overall model size.

[0184] Understandably, the key to structured channel pruning lies in precisely allocating computing power through dynamic β. In power inspection scenarios, the computational requirements of different areas (such as insulator cracks and sky backgrounds) vary significantly. For example, for areas with high texture complexity, such as insulator cracks, the hypernetwork generates a high β value (e.g., 0.9), retaining more channels to enhance local detail reconstruction. For simpler areas, such as the sky background, the hypernetwork uses a lower β value (e.g., 0.5), significantly reducing computational effort. This dynamic pruning mechanism avoids the rigidity of traditional fixed pruning ratios (e.g., uniformly retaining 50% of channels) while also addressing the issues of redundant computation and resource waste.

[0185] Furthermore, the design of the parameter reduction ratio highlights the synergistic effect of multi-layer pruning. For example, if a model contains four convolutional layers, with β values ​​of 0.8, 0.7, 0.6, and 0.5, the overall parameter reduction ratio is 1−(0.8×0.7×0.6×0.5)=1−0.168=0.832, a parameter reduction of 83.2%. This result demonstrates that by stacking the effects of pruning layer by layer, the model can be lightweight and compressed without significantly losing accuracy.

[0186] The advantages of structured pruning are particularly evident in the super-resolution task of power inspection videos. For example, on a drone edge device, the system can compress the model parameters from 12.4M to 2.1M, increasing the inference speed from 18fps to 28fps while maintaining a PSNR of 31.0dB, meeting the requirements for high-definition reconstruction of key defects such as insulator cracks. This achievement demonstrates the engineering value of dynamic pruning strategies in resource-constrained environments.

[0187] In the embodiment of the present application, unstructured pruning is used to sparsely process the convolution kernel weights of the backbone network to further compress the model size and improve computational efficiency. The specific steps are as follows.

[0188] (1) Weight sparsification operation:

[0189] Hypernetwork generation weight preservation probability matrix ,in, is the number of input / output channels, is the convolution kernel size.

[0190] Keep the weights as probability matrix With the original weight matrix , element-by-element multiplication (⊙) generates a sparse weight matrix , the formula is:

[0191]

[0192] in, represents the global sparse threshold, and γ is the global sparse threshold, ensuring that at least a fixed proportion of weights are retained.

[0193] (2) Forced pruning when sparsity is insufficient:

[0194] If the current sparsity <γ, then press The entropy ranking enforces pruning of low probability weights.

[0195] Calculating entropy , the formula is:

[0196]

[0197] in, Keep the elements in the probability matrix for the weights. After sorting the entropy values, keep the top high probability weights, and the rest are set to zero. For example, if the total number of weights in a layer is 100,000, γ =0.6, then the 60,000 highest probability weights are retained.

[0198] (3) Fine-tuning and optimization of the model after pruning:

[0199] The weight after sparsification , perform end-to-end fine-tuning, the formula is:

[0200]

[0201] in, is the learning rate, is a multi-objective loss function.

[0202] During fine-tuning, the learning rate is dynamically adjusted (e.g., using cosine annealing) to ensure that the remaining weights converge to the optimal solution. For example, on drone edge devices, the learning rate is gradually decayed from 1e−3 to 1e−5 during fine-tuning to stabilize the training process in low-computing environments.

[0203] The unstructured pruning and entropy ranking mechanism in this embodiment resolves the conflict between model compression and performance preservation in the super-resolution task of power inspection videos. The weight retention probability matrix is ​​dynamically generated by the super-network based on the texture complexity and motion intensity of the image blocks. For example, in the cracked insulator region, the probability distribution of Sw is concentrated (low entropy), preserving more key connections; while in the sky background region, the entropy of Sw is higher (dispersed distribution), forcing the pruning of redundant weights.

[0204] A global sparsity threshold constraint of γ = 0.6 ensures that the model retains sufficient expressive power after pruning. For example, if a layer originally has 100,000 parameters, pruning will retain 60,000 key weights, reducing the number of parameters by 40% while only decreasing the PSNR by 0.3dB.

[0205] When dynamic disturbances (such as rain, fog, or sudden changes in illumination) cause localized weight distribution anomalies, the entropy sorting mechanism automatically identifies and prunes low-probability weights to prevent local noise from disrupting overall performance. For example, if raindrops in a frame cause an abnormal increase in the probability of some weights in Sw, entropy sorting will retain only the truly important weights, suppressing artifacts. A multi-objective joint optimization (reconstruction loss, sparsity regularization, and computational efficiency constraints) ensures that the pruning and fine-tuning process balances accuracy, efficiency, and hardware limitations. For example, on the server side, γ=0.8 is relaxed to achieve higher reconstruction quality, while on the drone side, γ=0.6 is set to strictly limit computational effort to adapt to different deployment scenarios.

[0206] In an embodiment of the present application, the generation of the channel mask includes: performing global average pooling on the global response features to generate a channel feature vector. The channel feature vector is mapped to a corresponding probability value using a sigmoid function, which serves as a binarization threshold for the channel mask. The number of input and output channels of each layer of the backbone network is dynamically trimmed based on the channel mask.

[0207] Specifically, the channel mask generation process is divided into the following three key steps: (1) Global response feature extraction: First, global average pooling (GAP) is performed on the global response feature after the fusion of multi-scale dilated convolution and Softmax gating mechanism. Global average pooling converts the global response feature into a channel feature vector by calculating the feature mean of each channel. For example, for a feature map containing 256 channels, GAP will generate a vector of length 256, each element of which represents the global response intensity of the corresponding channel. (2) Channel importance mapping: The system inputs the channel feature vector into the Sigmoid function to generate a channel retention probability vector. The Sigmoid function maps the real value of the channel feature vector to the interval [0,1] to form a probability distribution. For example, the local texture channel of the insulator crack area may obtain a higher channel retention probability value (such as 0.9), while the channel value of the sky background area is lower (such as 0.3). (3) Binarization and mask generation: Generate a binary channel mask based on the probability vector. Specifically, set the binarization threshold τ (like τ =0.5), if p ( i )≥ τ ,but (Reserve channel), otherwise (Pruning channel). For example, if a layer channel retains the probability vector p =[0.9,0.4,0.7,0.2], threshold τ = 0.5, the generated mask is , retaining the front and back two channels. The final channel retention ratio is used to dynamically trim the number of channels in the backbone network

[0208] It is understood that the channel mask generation method in the embodiments of this application overcomes the limitations of traditional structured pruning, where channel retention rates are fixed or manually set, through global response feature perception and probability mapping binarization. This significantly improves defect detection accuracy in power inspection scenarios compared to traditional fixed pruning ratio methods. This technology has been deployed in power grid companies' drone inspection systems, supporting millimeter-level defect recognition and low-latency transmission of aerial video footage, providing key technical support for the refined operation and maintenance of smart grids.

[0209] In an embodiment of the present application, the dynamic prediction pruning strategy based on the feature complexity of the image block includes: calculating the intra-frame gradient intensity of the image block using the Sobel operator to quantify the texture complexity. The formula is:

[0210]

[0211] in, For the image block at position The gradient, is the L2 norm.

[0212] Specifically, the system uses the Sobel operator to extract the intensity of the intra-frame gradient of an image block to quantify texture complexity. Specifically, the Sobel operator involves convolving the image block with a horizontal kernel and a vertical kernel, respectively, to generate a gradient magnitude matrix that reflects the distribution of local edges and textures. The system then calculates the texture complexity of the image block by taking the mean of the gradient magnitudes and normalizing them to the range [0, 1].

[0213] The motion amplitude is quantified by the difference norm of adjacent frames, and the formula is:

[0214]

[0215] in, is the Frobenius norm.

[0216] Specifically, the Frobenius norm of the difference between adjacent frames is used to quantify the motion amplitude to adapt to dynamic disturbances in power grid inspection videos. The specific steps are: performing pixel-level differences between the current frame block and the previous frame block at the same location, generating a difference matrix, and calculating its Frobenius norm. This metric reflects the intensity of dynamic changes such as transmission line sway and cloud movement. For example, the motion amplitude in areas with violent conductor sway can reach values ​​between 0.3 and 0.5, while the motion amplitude in static background areas is less than 0.05.

[0217] The block-level channel retention rate is generated based on the weighted sum of the intra-frame gradient intensity and motion amplitude. The formula is:

[0218]

[0219] in, is the benchmark calculation amount, is the intra-frame gradient strength, is the range of motion, This is used to smooth intensity variations and avoid imbalanced resource allocation caused by local peaks. For example, in frames where conductors are swinging violently, the motion amplitude dominates, and the system automatically increases the block-level channel retention rate in this area to ensure temporal consistency. In static insulator regions, where the gradient intensity within the frame is dominant, the model prioritizes enhancing local details.

[0220] It can be understood that this application realizes the precise resource tilting to the key defect areas in the power inspection video by jointly perceiving the texture complexity and motion intensity of the image blocks. Specifically, in areas with high texture complexity such as insulator cracks and wire corrosion, the local gradient extracted by the system through the Sobel operator is significantly enhanced, triggering a higher texture complexity index. At this time, the block-level calculation intensity coefficient will be greatly improved due to the high gradient value, thereby driving the model to retain more channels and weight connections. For example, in the insulator crack area, the system dynamically increases the channel retention rate to 0.8~0.9 to ensure the clear reconstruction of local edges and small defects. This "key area key calculation" strategy makes the reconstruction quality of key defects significantly better than the traditional fixed pruning method, the PSNR index is improved by more than 1.5dB, and the false detection rate is reduced by 40%, providing a reliable guarantee for the accurate identification of millimeter-level cracks.

[0221] For uniform background areas such as the sky and vegetation, the system dynamically compresses its computing resource investment by quantifying low texture complexity and low motion amplitude. For example, in the sky area, the system compresses the baseline computing power to below 0.5G, the channel retention rate drops from 0.8 to 0.4, and the parameter volume is reduced by 60%. This "simple area lightweight" design not only reduces redundant calculations, but also releases more computing power for high-priority defect areas. On drone edge devices, this strategy significantly improves the overall inference efficiency, enabling the model to maintain high-resolution reconstruction capabilities under limited hardware resources, while reducing power consumption and heat generation, and extending device life.

[0222] Figure 2 Schematic diagram of a lightweight video super-resolution system based on dynamic pruning of a super-network provided by an embodiment of the present application. Figure 2 The lightweight video super-resolution system 10 based on hypernetwork dynamic pruning shown in the figure includes at least the following parts: an image acquisition module 100, an image processing module 200, a hole convolution layer 300, a hypernetwork structure 400, and an image restoration module 500.

[0223] In the embodiment of the present application, the image acquisition module 100 is used to acquire a video sequence of power inspection. The video sequence includes multiple frames of images and timing information between images. Figure 1 And the corresponding description will not be repeated here.

[0224] In the embodiment of the present application, the image processing module 200 is used to divide each frame image in the video sequence into multiple image blocks of fixed size; obtain the timing information between the multiple image blocks; construct a layered patch diffusion model based on the timing information between the multiple image blocks, please refer to Figure 1 And the corresponding description will not be repeated here.

[0225] In the embodiment of the present application, the dilated convolution layer 300 is used to perform multi-scale dilated convolution processing on multiple image blocks in the layered patch diffusion model to extract the spatiotemporal features of each image block; and to perform weighted fusion of the spatiotemporal features through the Softmax gating mechanism to generate a global response feature. For details, please refer to Figure 1 And the corresponding description will not be repeated here.

[0226] In the embodiment of the present application, the super network structure 400 is used to obtain global response features. The super network includes multiple candidate sub-network structures with pruning modules and a channel mask generator. The super network is used to dynamically predict the pruning strategy according to the feature complexity of the image block, and generate the corresponding sub-network structure to perform adaptive reasoning. For details, please refer to Figure 1 And the corresponding description will not be repeated here.

[0227] In the embodiment of the present application, the image restoration module 500 is used to perform deconvolution on the feature map processed by the sub-network to reconstruct the high-resolution image block, and splice the original image position to output a complete super-resolution video frame. For details, please refer to Figure 1 And the corresponding description will not be repeated here.

[0228] Figure 3 This is an electronic device 20 provided in one embodiment of the present application. Figure 3 As shown, the electronic device 20 includes at least the following parts: a processor 21 and a memory 22 .

[0229] In the embodiment of the present application, the memory 22 is used to store instructions executable by the processor 21. The processor 21 is configured to execute the instructions to implement the following Figure 1 The lightweight video super-resolution method based on dynamic pruning of hypernetwork is shown.

[0230] The program running in electronic device 20 according to one embodiment of the present application may be a program that controls a central processing unit (CPU) and the like to implement the functions of the aforementioned embodiment according to one aspect of the present invention (a program that causes a computer to function). Information processed by these devices is temporarily stored in random access memory (RAM) during processing, and then stored in various ROMs such as read-only memory (Flash ROM) and hard disk drives (HDDs), where it is read, modified, and written as needed by the CPU.

[0231] It should be noted that a portion of the electronic device 20 of the above embodiment may be implemented by a computer. In this case, a program for implementing the control function may be recorded on a computer-readable recording medium and implemented by reading the program recorded on the recording medium into a computer and executing it.

[0232] It should be noted that the "computer" mentioned here refers to the computer built into the electronic device 20, and is a computer including hardware such as an operating system and peripheral devices. In addition, "computer-readable recording medium" refers to removable media such as floppy disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into the computer.

[0233] Furthermore, "computer-readable recording media" may include: media that dynamically store programs for a short period of time, such as communication lines when transmitting programs via networks such as the Internet or communication lines such as telephone lines; and media that store programs for a fixed period of time, such as volatile memory within computers acting as servers or clients in this context. Furthermore, the aforementioned program may be a program for implementing a portion of the aforementioned functions, or a program that can achieve the aforementioned functions by combining with a program already stored in a computer.

[0234] Furthermore, the electronic device 20 in the above-described embodiment can also be implemented as a collection of multiple devices (a device group). Each device comprising the device group may include some or all of the functions or functional blocks of the electronic device 20 in the above-described embodiment. A device group only needs to include all of the functions or functional blocks of the electronic device 20.

[0235] It is understood that the lightweight video super-resolution method and system 10 based on hypernetwork dynamic pruning provided in the embodiments of the present application achieves efficient compression and accurate reconstruction of power inspection video super-resolution through hypernetwork dynamic pruning and content-aware mechanisms. By integrating structured and unstructured pruning strategies, the number of parameters is reduced by over 70%, achieving an inference speed of 28fps, and adapting to drone edge devices. Computing resources are dynamically allocated, and the PSNR in key defect areas is stabilized at over 31.0dB, with inter-frame consistency improved by 25%. Rain, fog, and light interference are effectively suppressed, meeting the requirements of millimeter-level crack detection, and significantly outperforming traditional fixed pruning methods.

[0236] Those skilled in the art should recognize that the above embodiments are merely intended to illustrate the present application and are not intended to limit the present application. As long as they are within the spirit of the present application, appropriate changes and modifications to the above embodiments are within the scope of protection claimed in the present application.

Claims

1. A lightweight video super-resolution method based on super-network dynamic pruning, applied to power inspection video processing, characterized by: The method comprises: Acquire a video sequence of power inspection, wherein the video sequence includes multiple frames of images and timing information between the images; Dividing each frame of image in the video sequence into a plurality of image blocks of fixed size; Acquiring timing information between the plurality of image blocks; constructing a hierarchical patch diffusion model based on temporal information between the plurality of image blocks; Performing multi-scale dilated convolution processing on the plurality of image blocks in the layered patch diffusion model through a dilated convolution layer to extract spatiotemporal features of each image block; The spatiotemporal features are weightedly fused through a Softmax gating mechanism to generate a global response feature; Inputting the global response feature and the image block into a super-network structure, the super-network including a plurality of candidate sub-network structures with pruning modules and a channel mask generator, the super-network being used to dynamically predict a pruning strategy based on the feature complexity of the image block and generate a corresponding sub-network structure to perform adaptive inference; Deconvolution is performed on the feature map processed by the sub-network to reconstruct a high-resolution image block, and the complete super-resolution video frame is output by splicing according to the original image position; Structured channel pruning adjustments include: The number of input and output channels of each layer of the backbone network is dynamically trimmed according to the channel mask, and the channel retention ratio is: ； in, Reserve ratio for the channel, Channel mask , is the original number of channels in a certain layer of the backbone network, Indicates the number of channels retained for statistics; The number of channels after cropping is: ； in, is the original number of channels, This is a rounding operation to ensure that the number of channels after clipping is an integer; The parameter reduction ratio is: ； represents the product of the retention ratios of all layers, is the ratio of reduction in the number of model parameters.

2. The lightweight video super-resolution method based on super-network dynamic pruning according to claim 1 is characterized in that: The method further comprises: Constructing a multi-objective joint optimization function, wherein the multi-objective joint optimization function includes a reconstruction loss term, a sparsity regularization term, and a computational efficiency constraint term; The formula of the multi-objective joint optimization function is: ； in, is the reconstruction loss term, is the sparse regularization term, is the computational efficiency constraint.

3. The lightweight video super-resolution method based on super-network dynamic pruning according to claim 2 is characterized in that: The reconstruction loss term is used to constrain the pixel-level difference between the super-resolution result and the true high-resolution frame; The expression of the reconstruction loss term is: ； in, represents a real high-resolution video frame, represents the super-resolution video frame reconstructed by the model, It represents the sum of squares of all pixel differences.

4. The lightweight video super-resolution method based on super-network dynamic pruning according to claim 2, characterized in that: The sparse regularization term is used to constrain the sparsity of the channel mask and the weight sparse matrix; The expression of the sparse regularization term is: ； in, , are the weight coefficients of the channel pruning binary mask and the weight retention probability matrix, represents the L1 norm, is the channel pruning binary mask, The L1 norm of corresponds to the total number of retained channels, Keep the probability matrix for the weights, The L1 norm of corresponds to the sum of the cumulative probabilities of the retained weights.

5. The lightweight video super-resolution method based on super-network dynamic pruning according to claim 2 is characterized in that: The computational efficiency constraint is used to limit the total computational effort to be below a preset upper limit; The expression of the computational efficiency constraint term is: ； in, Indicates the The floating point operations consumed by the block in the super-resolution inference stage, is a hyperparameter, and .

6. The lightweight video super-resolution method based on super-network dynamic pruning according to claim 1, characterized in that: The multi-scale dilated convolution process includes: Extracting multi-scale features through the cascaded dilated convolutional layers, wherein the dilation rate of the dilated convolutional layers increases exponentially; Perform linear mapping on each scale feature to generate an importance score; Weighting the importance scores by a weight matrix and a bias term; The importance score is normalized into a dynamic weight coefficient through the Softmax function, and the features under different void rates are weighted and fused. The formula for normalizing the importance score is: ； in, Indicates the The weight coefficient corresponding to the void rate is is the importance score, is the learnable weight matrix, is the global average pooling operation, is the bias term.

7. The lightweight video super-resolution method based on super-network dynamic pruning according to claim 2, characterized in that: The generation of the channel mask includes: Perform global average pooling on the global response features to generate a channel feature vector; The channel feature vector is mapped to the corresponding probability value through the Sigmoid function, which serves as the binarization threshold of the channel mask; The number of input and output channels of each layer of the backbone network is dynamically trimmed according to the channel mask.

8. The lightweight video super-resolution method based on super-network dynamic pruning according to claim 7 is characterized in that: Dynamically predicting pruning strategies based on the feature complexity of image blocks includes: The intra-frame gradient intensity of the image block is calculated by the Sobel operator to quantify the texture complexity. The formula is: ； in, For the image block at position The gradient, is the L2 norm; The motion amplitude is quantified by the difference norm of adjacent frames, and the formula is: ； in, is the Frobenius norm; The block-level channel retention rate is generated according to the weighted sum of the intra-frame gradient strength and the motion amplitude, and the formula is: ； in, is the benchmark calculation amount, is the intra-frame gradient strength, is the range of motion, Used to smooth intensity changes.

9. A lightweight video super-resolution system based on dynamic pruning of hypernetwork, characterized by: The system comprises: An image acquisition module is used to acquire a video sequence of power inspection, wherein the video sequence includes multiple frames of images and timing information between the images; An image processing module is configured to divide each frame of the video sequence into a plurality of image blocks of a fixed size; obtain timing information between the plurality of image blocks; and construct a layered patch diffusion model based on the timing information between the plurality of image blocks; a dilated convolution layer, configured to perform multi-scale dilated convolution processing on the plurality of image blocks within the hierarchical patch diffusion model to extract the spatiotemporal features of each image block; and to perform weighted fusion of the spatiotemporal features through a Softmax gating mechanism to generate a global response feature; A hypernetwork structure for obtaining the global response features, the hypernetwork comprising a plurality of candidate subnetwork structures with prunable modules and a channel mask generator, the hypernetwork being configured to dynamically predict pruning strategies based on the feature complexity of the image blocks, generate corresponding subnetwork structures, and perform adaptive inference; An image restoration module is used to perform deconvolution on the feature map processed by the sub-network to reconstruct a high-resolution image block, and splice it according to the original image position to output a complete super-resolution video frame; Structured channel pruning adjustments include: The number of input and output channels of each layer of the backbone network is dynamically trimmed according to the channel mask, and the channel retention ratio is: ； in, Reserve ratio for the channel, Channel mask , is the original number of channels in a certain layer of the backbone network, Indicates the number of channels retained for statistics; The number of channels after cropping is: ； in, is the original number of channels, This is a rounding operation to ensure that the number of channels after clipping is an integer; The parameter reduction ratio is: ； represents the product of the retention ratios of all layers, is the ratio of reduction in the number of model parameters.

Images