Video super-resolution reconstruction method, device, equipment and medium for transmission channel

Abstract

The present application discloses a method, apparatus, device and medium for super-resolution reconstruction of a video of a power transmission channel, which relates to the field of power defect detection. The method comprises: cutting each frame of a low-resolution video in a video data set into multiple square areas, flattening the multiple square areas to obtain multiple video frame sequences, inputting the multiple video frame sequences into an optical flow pre-estimation network, and predicting position distribution maps of the multiple video frame sequences; extracting feature vectors of the low-resolution video, fusing the feature vectors with the position distribution map to obtain a fusion vector; inputting the fusion vector into a Transformer neural network for trajectory attention operation to obtain a trajectory attention representation encoding vector; inputting the trajectory attention representation encoding vector into a pre-trained upsampling reconstruction network for processing to obtain a reconstructed sequence; sampling any frame of the low-resolution video to obtain a sampling sequence of any frame; and generating a super-resolution video of the power transmission channel based on the reconstruction sequence and the sampling sequence.

Description

Technical Field

[0001] The present application relates to the field of power defect detection, and more specifically, to a method, apparatus, device, and medium for super-resolution reconstruction of video of a power transmission channel. Background Art

[0002] The clarity and fidelity of transmission channel video have become crucial for power transmission monitoring. Traditional methods for acquiring and processing transmission channel video typically rely on high-resolution cameras and high-performance computers. This approach not only consumes significant resources and costs but also suffers from shortcomings such as susceptibility to environmental interference and difficulty in real-time processing. To overcome these shortcomings, super-resolution reconstruction technology has gained widespread application for enhancing the clarity and fidelity of transmission channel video. Super-resolution reconstruction utilizes artificial intelligence techniques such as deep learning to recover high-resolution video sequences from low-resolution ones. It can effectively improve the quality and visibility of transmission channel video, thereby enabling condition monitoring and fault diagnosis.

[0003] However, due to inherent issues with transmission channel videos, such as blur, noise, and motion, super-resolution reconstruction of videos faces many challenges. To improve the effectiveness and efficiency of super-resolution reconstruction of transmission channel videos, it is generally necessary to preprocess the transmission channel videos, such as denoising, enhancement, and correction. To improve the accuracy of target detection and recognition in videos, it is generally necessary to utilize inter-frame information in video sequences, such as motion and structural information, for feature extraction and fusion, and to utilize the spatiotemporal information of the videos to improve the expressiveness of target features. To improve the effectiveness of target labeling and description in videos, it is generally necessary to utilize artificial intelligence technologies, such as deep learning and natural language processing, to automatically analyze, identify, and label targets in videos, thereby realizing the intelligence of transmission channel videos.

[0004] When achieving super-resolution reconstruction of transmission channel video, inter-frame information in the video sequence is a key factor, determining the quality and efficiency of the reconstructed video. Traditional methods for extracting and utilizing inter-frame information typically rely on explicit motion estimation and compensation. This approach not only requires high computational complexity and relies on motion estimation, but is also prone to blurring and distortion between frames. Summary of the Invention

[0005] The purpose of this application is to provide a method, device, equipment and medium for super-resolution reconstruction of video of a transmission channel. This application solves the problems of poor reconstruction performance and high computational complexity in the current video super-resolution reconstruction process.

[0006] In a first aspect of the present application, a method for super-resolution reconstruction of a video of a transmission channel is provided, the method comprising:

[0007] Acquire a video dataset of a transmission channel, wherein the video dataset refers to video data of a target object of the transmission channel;

[0008] Each frame of a low-resolution video in a video dataset is cut into multiple square regions, and the multiple square regions of each frame are flattened to obtain multiple video frame sequences. The multiple video frame sequences are input into an optical flow pre-estimation network to predict the position distribution maps of the multiple video frame sequences.

[0009] Extract the feature vector of the low-resolution video, fuse the feature vector with the position distribution map, and obtain the fusion vector;

[0010] The fusion vector is input into the Transformer neural network for trajectory attention operation to obtain the trajectory attention representation encoding vector;

[0011] The trajectory attention representation encoding vector is input into the pre-trained upsampling reconstruction network for processing to obtain the reconstructed sequence;

[0012] Bicubic interpolation is used to sample any frame of the low-resolution video to obtain a sampling sequence of any frame;

[0013] Based on the reconstruction sequence and the sampling sequence, a super-resolution video of the transmission channel is generated.

[0014] In an implementation of the first aspect of the present application, each frame of the low-resolution video corresponds to a video frame sequence, and each square area corresponds to a coordinate position;

[0015] Multiple video frame sequences are input into the optical flow pre-estimation network to predict the position distribution maps of multiple video frame sequences, including:

[0016] Based on the optical flow pre-estimation network, the optical flow information of each square area in each video frame sequence corresponding to the change in the square area in the next frame of the low-resolution video is predicted;

[0017] According to the optical flow information, each frame of the low-resolution video is decomposed into trajectories to obtain multiple trajectories; where a trajectory refers to a visual unit where the target object of the transmission channel changes continuously;

[0018] Generate a position map for each trajectory, where the size of the position map is C×H×W, where C represents the length of the trajectory, H and W are the height and width of the low-resolution video frame, respectively. The position map is a binary matrix of 0 or 1, where 1 represents the coordinate position of the visual unit in the low-resolution video frame, and 0 represents the coordinate position of the visual unit not in the low-resolution video frame.

[0019] The position maps of each trajectory are combined to obtain position distribution maps of multiple video frame sequences, where the position distribution maps are used to indicate the position distribution of different trajectories in the low-resolution video.

[0020] In an implementation of the first aspect of the present application, extracting a feature vector of a low-resolution video includes:

[0021] Inputting the low-resolution video and any frame of the low-resolution video into the embedding network respectively, extracting a feature vector of the low-resolution video, wherein the feature vector includes the first key vector and the first value vector of each frame of the low-resolution video and the first query vector of any frame;

[0022] The feature vector is fused with the position distribution map to obtain a fusion vector, including:

[0023] fusing the first query vector, the first key vector, and the first value vector with the position distribution map respectively to obtain a fused vector, wherein the fused vector includes the second key vector of the position distribution map, and the second query vector and the second value vector of any frame;

[0024] The fusion vector is input into the Transformer neural network for trajectory attention operation to obtain the trajectory attention representation encoding vector. Specifically, the second query vector, the second key vector and the second value vector are input into the Transformer neural network for trajectory attention operation to obtain the trajectory attention representation encoding vector.

[0025] In one implementation of the first aspect of the present application, fusing the first query vector, the first key vector, and the first value vector with the position distribution map to obtain a fused vector includes:

[0026] Multiplying the position map of any frame of the position distribution map by the first query vector and the first value vector respectively to obtain a second query vector and a second value vector of the any frame;

[0027] The first bond vector and the position distribution map are multiplied to obtain a second bond vector corresponding to the first bond vector and the position distribution map.

[0028] In one implementation of the first aspect of the present application, the expression of the trajectory attention operation is: in, represents the second query vector, τ i represents the i-th track, h represents the height of the low-resolution video frame, w represents the width of the low-resolution video frame, Represents the location distribution map, represents the second key vector, Represents the second value vector, v represents the first value vector, K represents the first key vector, q represents the first query vector, softmax represents the normalization function, d represents the number of square areas, T represents the trajectory attention representation encoding vector, and t represents the tth frame of the low-resolution video.

[0029] In one implementation of the first aspect of the present application, the pre-trained upsampling reconstruction network includes:

[0030] Constructing an upsampling reconstruction network and a loss function; wherein the loss function is obtained by weighted summation of pixel loss, perceptual loss, and trajectory loss;

[0031] Configure the optimizer, calculate the loss value of the loss function, and use the optimizer to update the network parameters of the upsampling reconstruction network until the loss function converges to obtain the pre-trained upsampling reconstruction network.

[0032] In an implementation of the first aspect of the present application, generating a super-resolution video of a transmission channel based on a reconstruction sequence and a sampling sequence includes:

[0033] Adding the sampling sequence of any frame of the low-resolution video to the reconstruction sequence bit by bit to obtain a high-resolution video frame of any frame of the low-resolution video;

[0034] A super-resolution video of the transmission channel is generated based on the high-resolution video frame for each frame of the low-resolution video.

[0035] In a second aspect of the present application, a device for super-resolution reconstruction of a video of a transmission channel is provided, the device comprising:

[0036] A video data acquisition module, configured to acquire a video data set of a power transmission channel, wherein the video data set refers to video data of a target object of the power transmission channel;

[0037] A position distribution map prediction module is used to cut each frame of a low-resolution video in a video dataset into multiple square areas, flatten the multiple square areas of each frame to obtain multiple video frame sequences, input the multiple video frame sequences into an optical flow pre-estimation network, and predict the position distribution maps of the multiple video frame sequences;

[0038] Feature extraction and fusion module, used to extract feature vectors of low-resolution videos and fuse the feature vectors with the position distribution map to obtain a fusion vector;

[0039] The operation module is used to input the fusion vector into the Transformer neural network to perform trajectory attention operation and obtain the trajectory attention representation encoding vector;

[0040] The reconstruction module is used to input the trajectory attention representation encoding vector into the pre-trained upsampling reconstruction network for processing to obtain the reconstructed sequence;

[0041] A sampling module is used to sample any frame of the low-resolution video using a bicubic interpolation method to obtain a sampling sequence of any frame;

[0042] The super-resolution video generation module is used to generate a super-resolution video of the transmission channel based on the reconstruction sequence and the sampling sequence.

[0043] The third aspect of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of a video super-resolution reconstruction method for a transmission channel provided in the first aspect of the present application are implemented.

[0044] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of a video super-resolution reconstruction method for a transmission channel provided in the first aspect of the present application are implemented.

[0045] Compared with the prior art, this application has the following beneficial effects:

[0046] In a video super-resolution reconstruction method for a transmission channel provided in the present application, the present application utilizes a trajectory perception method to effectively extract dynamic information from the video, enhance the spatiotemporal consistency of the video, and thus improve the quality and efficiency of the video super-resolution reconstruction process. The present application does not need to perform explicit motion estimation and compensation operations, but instead implicitly aligns and fuses information between frames through a trajectory-aware Transformer neural network, thereby reducing computational complexity and dependence on motion estimation. First, the transmission line in the transmission channel video is used as a priori for the trajectory, and then the trajectory is learned and utilized through the Transformer neural network, thereby improving the super-resolution reconstruction effect of the video; further, the t-th frame of the low-resolution video is upsampled through a bicubic interpolation algorithm. The sample sequence is obtained by performing the same operation; the trajectory attention representation coding vector output by the Transformer neural network is input into the upsampling reconstruction network for reconstruction to obtain a reconstructed sequence. Then, the reconstructed sequence is added bit by bit to the sampling sequence of the tth frame to obtain a high-resolution video frame of the transmission channel. This operation can further improve the clarity and authenticity of the reconstructed video. For the reconstruction operation of the trajectory attention representation coding vector output by the Transformer neural network, the present application also constructs a loss function and optimizer including pixel-level loss, perception-level loss and trajectory-level loss. The optimizer can adaptively adjust the learning rate of the upsampling reconstruction network parameters according to the gradient of the loss function, and add a weight attenuation term to prevent overfitting, thereby guiding the upsampling reconstruction network to learn better reconstruction results. Therefore, compared with the current super-resolution reconstruction method based on inter-frame interpolation, the present application has better reconstruction performance and lower computational complexity, and is suitable for real-time transmission channel video monitoring scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the embodiments of the present application, constitute a part of the present application, and do not constitute a limitation of the embodiments of the present application. In the drawings:

[0048] Figure 1 A schematic diagram of a flow chart of a method for super-resolution reconstruction of a video transmission channel provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of the structure of the video super-resolution reconstruction process provided in an embodiment of the present application;

[0050] Figure 3 This is a block diagram of the principle of the video super-resolution reconstruction device for the power transmission channel provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of this application more clear, the present application is further described in detail below in conjunction with examples and drawings. The schematic implementation methods of this application and their descriptions are only used to explain this application and are not intended to limit this application.

[0053] It should be noted that the terms "including" or "may include" used in various embodiments of the present application indicate the presence of the claimed function, operation or element, and do not limit the addition of one or more functions, operations or elements. In addition, as used in various embodiments of the present application, the terms "including", "having" and their cognates are only intended to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the presence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or the possibility of adding one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0054] In various embodiments of the present application, the expression "or" or "at least one of B or / and C" includes any or all combinations of the words listed simultaneously. For example, the expression "B or C" or "at least one of B or / and C" may include B, may include C, or may include both B and C.

[0055] It should be understood that terms such as "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0056] Please refer to Figure 1 , Figure 1 A flow chart of a video super-resolution reconstruction method for a transmission channel provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method includes:

[0057] 101. Acquire a video dataset of a power transmission channel, where the video dataset refers to video data of a target object of the power transmission channel.

[0058] In this embodiment, the transmission channels are mainly divided into two types: underground lines and overhead lines. This application is applied to the super-resolution reconstruction of the video data set of the overhead channels. The video data set can be acquired by an aircraft (such as a drone) carrying a camera and flying along the overhead line, thereby acquiring the video data set of the transmission channel. The target objects of the video data include the video data of the transmission line, the transmission tower, and the equipment installed on the transmission tower. Specifically, the equipment includes insulators, shock absorbers, lightning arresters, etc., and of course other auxiliary equipment, such as hardware, tower accessories, etc., which will not be described in detail in this embodiment. It should be noted that since the video data set is acquired by the drone, each frame of the video data records the information of the target object of the transmission channel, such as location information.

[0059] 102. Cut each frame of a low-resolution video in a video dataset into multiple square areas, flatten the multiple square areas of each frame to obtain multiple video frame sequences, input the multiple video frame sequences into an optical flow pre-estimation network, and predict position distribution maps of the multiple video frame sequences.

[0060] In some embodiments, each frame of the low-resolution video corresponds to a video frame sequence, and each square area corresponds to a coordinate position. Specifically, since this embodiment cuts each frame into multiple square areas and flattens the multiple square areas of each frame, a video frame sequence can be obtained. In general, the number of frames of low-resolution video in a video dataset is greater than 1. Therefore, for a low-resolution video, the above operation generally results in multiple video frame sequences. Secondly, the determination of low-resolution videos is a conventional technical means in this field. For example, low-resolution videos in a video dataset can be determined based on video resolution, bit rate, frame rate, etc.

[0061] In some embodiments, multiple video frame sequences are input into an optical flow pre-estimation network to predict position distribution maps of the multiple video frame sequences, including: predicting, based on the optical flow pre-estimation network, optical flow information of changes in the square area corresponding to the next frame of the low-resolution video for each square area of ​​each video frame sequence; decomposing each frame of the low-resolution video into trajectories based on the optical flow information to obtain multiple trajectories; wherein the trajectory refers to a continuously changing visual unit of the target object of the transmission channel; generating a position map for each trajectory, wherein the size of the position map is C×H×W, C represents the length of the trajectory, H and W are the height and width of the frame of the low-resolution video, respectively, and the position map is a binary matrix of 0 or 1, 1 represents that the visual unit is at the coordinate position of the frame of the low-resolution video, and 0 represents that the visual unit is not at the coordinate position of the frame of the low-resolution video; combining the position maps of each trajectory to obtain position distribution maps of the multiple video frame sequences, wherein the position distribution map is used to indicate the position distribution of different trajectories in the low-resolution video.

[0062] like Figure 2 As shown in Figure 1, the optical flow pre-estimation network is the optical flow network. Specifically, each frame in the low-resolution video of the transmission channel is divided into multiple square regions, each corresponding to a coordinate, resulting in a video frame sequence of length d. The position maps of all trajectories are combined into a set, namely the position distribution map O, which indicates the position distribution of different trajectories in the video frame. The size of O is A × C × H × W, where A is the number of trajectories. O can be used for indexing operations in the self-attention mechanism, thereby achieving trajectory-aware feature learning.

[0063] This embodiment uses the optical flow estimation Perceiver IO pre-trained model as the optical flow pre-estimation network. Specifically, the optical flow estimation Perceiver IO pre-trained model includes optical flow estimation and Perceiver IO pre-trained models, which are two relatively independent but can be used in combination. First of all, optical flow estimation is mainly used to describe the motion information of objects in the image. It is based on the assumption of constant brightness and estimates the motion trajectory of the object by calculating the displacement of pixels in time. PerceiverIO is a deep learning model that aims to solve problems of various input and output types and provide a unified processing framework for various real-world applications such as language, vision, and multimodal understanding.

[0064] When it comes to applying the Perceiver IO pre-trained model to optical flow estimation, it can be done in the following way: First, the Perceiver IO model is pre-trained using a large-scale dataset to learn rich motion feature representations. These pre-training datasets can include video sequences showing various object motions. Then, the pre-trained Perceiver IO model is applied to the optical flow estimation task. Specifically, the model receives consecutive video frames as input and outputs a motion vector for each pixel. By comparing the differences between the input frames, the model learns the pixel motion patterns and estimates the optical flow information based on them. During the training process, an appropriate loss function can be used to optimize the model's performance, such as the error or accuracy of the optical flow estimation. Through iterative training, the model can gradually improve the accuracy and robustness of the optical flow estimation. Ultimately, the trained Perceiver IO model can be used in practical optical flow estimation tasks such as object tracking. By utilizing the motion feature representation learned by the model, the trajectory information of the target object in the video image can be more accurately predicted. Therefore, this embodiment uses the trajectory perception method to effectively extract dynamic information in the video, enhance the spatiotemporal consistency of the video, and thus improve the quality and efficiency of super-resolution reconstruction.

[0065] 103 , extracting a feature vector of the low-resolution video, and fusing the feature vector with the position distribution map to obtain a fusion vector.

[0066] In some embodiments, extracting a feature vector of the low-resolution video includes: inputting the low-resolution video and any frame of the low-resolution video into an embedding network, respectively, and extracting a feature vector of the low-resolution video, wherein the feature vector includes a first key vector and a first value vector of each frame of the low-resolution video, and a first query vector of any frame;

[0067] Fusing the feature vector with the position distribution map to obtain a fused vector includes: fusing the first query vector, the first key vector, and the first value vector with the position distribution map to obtain a fused vector. Specifically, obtaining the fused vector includes: multiplying the position map of any frame of the position distribution map with the first query vector and the first value vector to obtain a second query vector and a second value vector of the frame; multiplying the first key vector with the position distribution map to obtain a second key vector corresponding to the first key vector and the position distribution map;

[0068] like Figure 2 As shown, extracting an embedding network for a video's Q (query vector), K (key vector), and V (value vector) is a conventional technique in the art. Embedding networks are typically used in the context of implementing attention mechanisms (such as self-attention mechanisms), such as Transformer neural networks. For example, the embedding network includes an input layer, an embedding layer, a position encoding layer, a self-attention mechanism layer, a feedforward neural network, and an output layer. The input layer decomposes the video into a series of frames and uses each frame as input. Each frame can be raw pixel data or a feature map extracted by a convolutional neural network (CNN). The embedding layer converts the input data (whether raw pixels or feature maps) into a fixed-size vector. The embedding layer can be a fully connected neural network layer. Regarding the position encoding layer, since Transformer neural networks themselves do not have the ability to process sequential position information, position encoding is required to capture the sequential relationship between frames. This can be achieved through position encoding implemented using sine and cosine functions or through learned position embeddings. For the self-attention mechanism layer, after obtaining the embedding vector and position encoding, the Q, K, and V vectors can be calculated. These vectors are typically derived from the embedding vector through a linear transformation. These vectors are then used to perform a self-attention mechanism to capture the dependencies between video frames.

[0069] Further, such as Figure 2 As shown, for the Q, K, V vectors of each frame extracted, that is, the first key vector and the first value vector of each frame of the low-resolution video, and the first query vector of any frame, Q is combined with the position map of the t-th frame to obtain the second query vector of the t-th frame Combine V with the position map of the tth frame to get the second value vector of the tth frame

[0070] Combine the first bond vector K with the position graph to obtain the second bond vector corresponding to K with respect to the position graph. The expression is as follows:

[0071]

[0072] 104, input the fusion vector into the Transformer neural network to perform trajectory attention operation to obtain the trajectory attention representation encoding vector.

[0073] In this embodiment, the second query vector, the second key vector, and the second value vector are input into the Transformer neural network to perform trajectory attention operation to obtain a trajectory attention representation encoding vector.

[0074] In some embodiments, the expression of the trajectory attention operation is: in, represents the second query vector, τ i represents the i-th track, h represents the height of the low-resolution video frame, w represents the width of the low-resolution video frame, Represents the location distribution map, represents the second key vector, Represents the second value vector, v represents the first value vector, K represents the first key vector, q represents the first query vector, softmax represents the normalization function, d represents the number of square areas, T represents the trajectory attention representation encoding vector, and t represents the tth frame of the low-resolution video.

[0075] 105, the trajectory attention representation encoding vector is input into the pre-trained upsampling reconstruction network for processing to obtain the reconstructed sequence.

[0076] In this embodiment, the upsampling reconstruction network is a conventional technical means applied to the video super-resolution reconstruction task, so this embodiment does not describe its network structure in detail. However, for the training of the upsampling reconstruction network, this embodiment combines the encoding vector output by the Transformer neural network with trajectory perception ability, and proposes a training method that combines pixel loss, perceptual loss and trajectory loss. Specifically as follows: In some embodiments, the pre-trained upsampling reconstruction network includes: constructing an upsampling reconstruction network and a loss function; wherein the loss function is obtained by weighted summation of pixel loss, perceptual loss and trajectory loss; configuring an optimizer, calculating the loss value of the loss function, and using the optimizer to update the network parameters of the upsampling reconstruction network until the loss function converges, thereby obtaining a pre-trained upsampling reconstruction network.

[0077] In this embodiment, we first define a suitable loss function for the upsampling reconstruction network, which includes pixel loss, perceptual loss, and trajectory loss. The formula of the loss function is as follows: L(θ) = αL pixel +βL perceptual +γL trajectory , where L pixel , L perceptual , L trajectory They are pixel loss, perceptual loss, and trajectory loss. α, β, and γ are three adjustable parameters.

[0078] The expression of pixel loss function is: Where T represents the total number of frames in the video, N is the number of regions per frame, is the predicted value of the i-th region in the t-th frame, is the corresponding true value.

[0079] The expression of the perceptual loss function is: Among them, Φ l represents the features of the lth layer extracted from the pre-trained network, L is the set of selected layers, C, H, and W represent the number of channels corresponding to the video frame, and the height and width of the video frame respectively.

[0080] The expression of the trajectory loss function is: Among them, and T j (t) denote the predicted and true positions of the j-th object in the t-th frame, respectively.

[0081] Furthermore, we define a suitable optimizer, such as AdamW, which can adaptively adjust the learning rate of model parameters according to the gradient of the loss function and add a weight decay term to prevent overfitting. The formula of the AdamW optimizer is as follows:

[0082] where θ t+1 are model parameters, θ t+1 is the learning rate, and are the first-order and second-order moment estimates of the gradient, ε is a small constant, λ is the weight decay coefficient, and t is the number of iterations.

[0083] Finally, the value of the loss function is calculated, and the network parameters of the network are updated using the optimizer. The above steps are repeated multiple times until the loss value of the loss function converges to a small or stable level, thereby obtaining a pre-trained upsampling reconstruction network of the transmission channel.

[0084] 106 , sampling any frame of the low-resolution video using a bicubic interpolation method to obtain a sampling sequence of the arbitrary frame.

[0085] In this embodiment, the bicubic interpolation algorithm is used to perform an upsampling operation on the tth frame of the original low-resolution video to obtain a sampling sequence. The bicubic interpolation algorithm is a conventional algorithm for upsampling operations, so this embodiment does not provide a redundant description.

[0086] 107, generating a super-resolution video of the transmission channel based on the reconstruction sequence and the sampling sequence.

[0087] In some embodiments, a super-resolution video of the transmission channel is generated based on the reconstruction sequence and the sampling sequence, including: adding the sampling sequence of any frame of the low-resolution video to the reconstruction sequence bit by bit to obtain a high-resolution video frame of any frame of the low-resolution video; and generating a super-resolution video of the transmission channel based on the high-resolution video frame of each frame of the low-resolution video.

[0088] Specifically, the reconstructed sequence and the sampling sequence are of the same size, so the elements of the reconstructed sequence and the sampling sequence are added bit by bit to obtain a high-resolution video frame of any reconstructed transmission channel. The reconstructed sequence of all low-resolution video frames is added bit by bit to the sampling sequence, and then arranged in the order of the video frames to obtain a super-resolution video of the transmission channel.

[0089] In summary, the video super-resolution reconstruction method of a transmission channel described in the above embodiments, this embodiment uses a trajectory perception method to effectively extract dynamic information in the video, enhance the spatiotemporal consistency of the video, and thus improve the quality and efficiency of the video super-resolution reconstruction process. It does not require explicit motion estimation and compensation operations, but implicitly aligns and fuses information between frames through a trajectory-aware Transformer neural network, thereby reducing computational complexity and dependence on motion estimation. First, the transmission line in the transmission channel video is used as a priori for the trajectory, and then the trajectory is learned and utilized through the Transformer neural network, thereby improving the super-resolution reconstruction effect of the video; further, the t-th frame of the low-resolution video is upsampled through a bicubic interpolation algorithm. The sample sequence is obtained by performing the same operation; the trajectory attention representation encoding vector output by the Transformer neural network is input into the upsampling reconstruction network for reconstruction to obtain a reconstructed sequence. Then, the reconstructed sequence is added bit by bit to the sampling sequence of the tth frame to obtain a high-resolution video frame of the transmission channel. This operation can further improve the clarity and authenticity of the reconstructed video. For the reconstruction operation of the trajectory attention representation encoding vector output by the Transformer neural network, the present application also constructs a loss function and optimizer including pixel-level loss, perception-level loss and trajectory-level loss. The optimizer can adaptively adjust the learning rate of the upsampling reconstruction network parameters according to the gradient of the loss function, and add a weight attenuation term to prevent overfitting, thereby guiding the upsampling reconstruction network to learn better reconstruction results. Therefore, compared with the current super-resolution reconstruction method based on inter-frame interpolation, it has better reconstruction performance and lower computational complexity, and is suitable for real-time transmission channel video monitoring scenarios.

[0090] Please refer to Figure 3 , Figure 3 The principle block diagram of the video super-resolution reconstruction device for the transmission channel provided in the embodiment of the present application is as follows: Figure 3 As shown, the device includes:

[0091] A video data acquisition module 310 is configured to acquire a video data set of a power transmission channel, wherein the video data set refers to video data of a target object of the power transmission channel;

[0092] A position distribution map prediction module 320 is configured to slice each frame of a low-resolution video in a video dataset into a plurality of square regions, flatten the plurality of square regions of each frame to obtain a plurality of video frame sequences, input the plurality of video frame sequences into an optical flow pre-estimation network, and predict position distribution maps of the plurality of video frame sequences;

[0093] A feature extraction and fusion module 330 is used to extract feature vectors of the low-resolution video and fuse the feature vectors with the position distribution map to obtain a fusion vector;

[0094] Operation module 340, used to input the fusion vector into the Transformer neural network to perform trajectory attention operation to obtain the trajectory attention representation encoding vector;

[0095] The reconstruction module 350 is used to input the trajectory attention representation encoding vector into a pre-trained upsampling reconstruction network for processing to obtain a reconstructed sequence;

[0096] The sampling module 360 ​​is used to sample any frame of the low-resolution video using a bicubic interpolation method to obtain a sampling sequence of the any frame;

[0097] The super-resolution video generation module 370 is configured to generate a super-resolution video of the transmission channel based on the reconstruction sequence and the sampling sequence.

[0098] It can be seen that the embodiment of the present application provides a video super-resolution reconstruction device for a transmission channel. The present application uses a trajectory perception method to effectively extract dynamic information from the video, enhance the spatiotemporal consistency of the video, and thus improve the quality and efficiency of the video super-resolution reconstruction process. The present application does not need to perform explicit motion estimation and compensation operations, but implicitly aligns and fuses information between frames through a trajectory-aware Transformer neural network, thereby reducing computational complexity and dependence on motion estimation. First, the transmission line in the transmission channel video is used as a priori for the trajectory, and then the trajectory is learned and utilized through the Transformer neural network, thereby improving the super-resolution reconstruction effect of the video; further, the bicubic interpolation algorithm is performed on the tth frame of the low-resolution video. Upsampling operation to obtain a sampling sequence; the trajectory attention representation encoding vector output by the Transformer neural network is input into the upsampling reconstruction network for reconstruction to obtain a reconstructed sequence, and then the reconstructed sequence is added bit by bit to the sampling sequence of the tth frame to obtain a high-resolution video frame of the transmission channel. This operation can further improve the clarity and authenticity of the reconstructed video. For the reconstruction operation of the trajectory attention representation encoding vector output by the Transformer neural network, this application also constructs a loss function and optimizer including pixel-level loss, perception-level loss and trajectory-level loss. The optimizer can adaptively adjust the learning rate of the upsampling reconstruction network parameters according to the gradient of the loss function, and add a weight attenuation term to prevent overfitting, thereby guiding the upsampling reconstruction network to learn better reconstruction results. Therefore, compared with the current super-resolution reconstruction method based on inter-frame interpolation, this application has better reconstruction performance and lower computational complexity, and is suitable for real-time transmission channel video monitoring scenarios.

[0099] Please refer to Figure 4 , Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 400 includes a processor 410, a memory 420, a communication interface 430, and at least one communication bus for connecting the processor 410, the memory 420, and the communication interface 430. The memory 420 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (PROM), or a compact disc read-only memory (CD-ROM), and is used for related instructions and data.

[0100] Communication interface 430 is used to receive and send data. Processor 410 can be one or more CPUs. If processor 410 is a CPU, the CPU can be a single-core CPU or a multi-core CPU. Processor 410 in electronic device 400 is used to read one or more programs 421 stored in memory 420 and perform the following operations: obtaining a video dataset of a transmission channel, where the video dataset refers to video data of a target object of the transmission channel; cutting each frame of a low-resolution video in the video dataset into multiple square regions, flattening the multiple square regions of each frame to obtain multiple video frame sequences, inputting the multiple video frame sequences into an optical flow pre-estimation network to predict position distribution maps of the multiple video frame sequences; extracting feature vectors of the low-resolution video, fusing the feature vectors with the position distribution map to obtain a fused vector; inputting the fused vector into a Transformer neural network for trajectory attention operation to obtain a trajectory attention representation encoding vector; inputting the trajectory attention representation encoding vector into a pre-trained upsampling reconstruction network for processing to obtain a reconstructed sequence; sampling any frame of the low-resolution video using bicubic interpolation to obtain a sampling sequence of any frame; and generating a super-resolution video of the transmission channel based on the reconstructed sequence and the sampling sequence.

[0101] It should be noted that the specific implementation of each operation can be as described above. Figure 1 The corresponding description of the method embodiment shown is that the electronic device 400 can be used to execute a video super-resolution reconstruction method of a transmission channel of the above method embodiment of the present application, which will not be described in detail here.

[0102] In an embodiment of the present disclosure, a computer-readable storage medium is also provided. The computer-readable storage medium is a memory device in a computer device, used to store programs and data. It is understood that the computer-readable storage medium herein may include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides storage space, which stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by a processor. These instructions may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium herein may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The processor may load and execute the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the video super-resolution reconstruction method for a power transmission channel described in the above-mentioned embodiment. Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0103] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A video super-resolution reconstruction method for a transmission channel, characterized in that: Methods include: Acquire a video dataset of a transmission channel, wherein the video dataset refers to video data of a target object of the transmission channel; Each frame of a low-resolution video in a video data set is cut into multiple square areas, and the multiple square areas of each frame are flattened to obtain multiple video frame sequences. The multiple video frame sequences are input into an optical flow pre-estimation network to predict position distribution maps of the multiple video frame sequences; wherein each frame of the low-resolution video corresponds to a video frame sequence, and each square area corresponds to a coordinate position; wherein the multiple video frame sequences are input into the optical flow pre-estimation network to predict position distribution maps of the multiple video frame sequences, including: predicting, based on the optical flow pre-estimation network, the optical flow information of each square area of ​​each video frame sequence, which changes in the square area corresponding to the next frame of the low-resolution video; based on the optical flow information, decomposing each frame of the low-resolution video into trajectories to obtain multiple trajectories; wherein the trajectory refers to a continuously changing visual unit of the target object of the transmission channel; generating a position map of each trajectory, wherein the size of the position map is C H W and C represent the length of the trajectory, H and W are the height and width of the low-resolution video frame, respectively. The position map is a binary matrix of 0 or 1, where 1 indicates that the visual unit is at the coordinate position of the low-resolution video frame, and 0 indicates that the visual unit is not at the coordinate position of the low-resolution video frame. The position maps of each trajectory are combined to obtain position distribution maps of multiple video frame sequences, where the position distribution maps are used to indicate the position distribution of different trajectories in the low-resolution video. Extracting a feature vector of the low-resolution video, fusing the feature vector with the position distribution map to obtain a fused vector; wherein extracting the feature vector of the low-resolution video includes: inputting the low-resolution video and any frame of the low-resolution video into the embedding network respectively, extracting the feature vector of the low-resolution video, wherein the feature vector includes the first key vector and the first value vector of each frame of the low-resolution video, and the first query vector of any frame; fusing the feature vector with the position distribution map to obtain a fused vector, including: fusing the first query vector, the first key vector and the first value vector with the position distribution map respectively to obtain a fused vector, wherein the fused vector includes the second key vector of the position distribution map, and the second query vector and the second value vector of any frame; multiplying the position map of any frame of the position distribution map with the first query vector and the first value vector respectively to obtain the second query vector and the second value vector of any frame; multiplying the first key vector with the position distribution map to obtain a second key vector corresponding to the first key vector and the position distribution map; The fusion vector is input into the Transformer neural network for trajectory attention operation to obtain the trajectory attention representation encoding vector; wherein, the second query vector, the second key vector and the second value vector are input into the Transformer neural network for trajectory attention operation to obtain the trajectory attention representation encoding vector; wherein, the expression of trajectory attention operation is: ,in, represents the second query vector, Indicates the i A trajectory, h Indicates the height of the low-resolution video frame. w Indicates the width of the low-resolution video frame. Represents the location distribution map, represents the second key vector, represents the second value vector, v represents the first value vector, K represents the first bond vector, q represents the first query vector, represents the normalization function, d represents the number of square areas, T represents the trajectory attention representation encoding vector, t represents the tth frame of the low-resolution video; Inputting the trajectory attention representation encoding vector into a pre-trained upsampling reconstruction network for processing to obtain a reconstructed sequence; wherein the pre-trained upsampling reconstruction network includes: constructing an upsampling reconstruction network and a loss function; wherein the loss function is obtained by weighted summation of pixel loss, perceptual loss, and trajectory loss; configuring an optimizer, calculating a loss value of the loss function, and using the optimizer to update network parameters of the upsampling reconstruction network until the loss function converges, thereby obtaining the pre-trained upsampling reconstruction network; Bicubic interpolation is used to sample any frame of the low-resolution video to obtain a sampling sequence of any frame; Based on the reconstruction sequence and the sampling sequence, a super-resolution video of the transmission channel is generated.

2. The video super-resolution reconstruction method of a transmission channel according to claim 1, characterized in that: Generate super-resolution video of the transmission channel based on the reconstruction sequence and sampling sequence, including: Adding the sampling sequence of any frame of the low-resolution video to the reconstruction sequence bit by bit to obtain a high-resolution video frame of any frame of the low-resolution video; A super-resolution video of the transmission channel is generated based on the high-resolution video frame for each frame of the low-resolution video.

3. A video super-resolution reconstruction device for a transmission channel, characterized in that: The device includes: A video data acquisition module, configured to acquire a video data set of a power transmission channel, wherein the video data set refers to video data of a target object of the power transmission channel; A position distribution map prediction module is used to cut each frame of a low-resolution video in a video data set into multiple square areas, flatten the multiple square areas of each frame to obtain multiple video frame sequences, input the multiple video frame sequences into an optical flow pre-estimation network, and predict the position distribution maps of the multiple video frame sequences; wherein each frame of the low-resolution video corresponds to a video frame sequence, and each square area corresponds to a coordinate position; wherein the multiple video frame sequences are input into the optical flow pre-estimation network, and the position distribution maps of the multiple video frame sequences are predicted, including: predicting, based on the optical flow pre-estimation network, the optical flow information of each square area of ​​each video frame sequence, the change of the square area corresponding to the next frame of the low-resolution video; based on the optical flow information, decomposing each frame of the low-resolution video into trajectories to obtain multiple trajectories; wherein the trajectory refers to a continuously changing visual unit of the target object of the transmission channel; generating a position map for each trajectory, wherein the size of the position map is C H W and C represent the length of the trajectory, H and W are the height and width of the low-resolution video frame, respectively. The position map is a binary matrix of 0 or 1, where 1 indicates that the visual unit is at the coordinate position of the low-resolution video frame, and 0 indicates that the visual unit is not at the coordinate position of the low-resolution video frame. The position maps of each trajectory are combined to obtain position distribution maps of multiple video frame sequences, where the position distribution maps are used to indicate the position distribution of different trajectories in the low-resolution video. A feature extraction and fusion module is used to extract feature vectors of low-resolution videos, and fuse the feature vectors with the position distribution map to obtain a fusion vector; wherein, extracting the feature vectors of low-resolution videos includes: inputting the low-resolution video and any frame of the low-resolution video into the embedding network respectively, extracting the feature vectors of the low-resolution video, wherein the feature vectors include the first key vector and the first value vector of each frame of the low-resolution video, and the first query vector of any frame; fusing the feature vectors with the position distribution map to obtain a fusion vector, including: fusing the first query vector, the first key vector and the first value vector with the position distribution map respectively, to obtain a fusion vector, wherein the fusion vector includes the second key vector of the position distribution map, and the second query vector and the second value vector of any frame; multiplying the position map of any frame of the position distribution map with the first query vector and the first value vector respectively, to obtain the second query vector and the second value vector of any frame; multiplying the first key vector with the position distribution map to obtain a second key vector corresponding to the first key vector and the position distribution map; The operation module is used to input the fusion vector into the Transformer neural network for trajectory attention operation to obtain the trajectory attention representation encoding vector; wherein, the second query vector, the second key vector and the second value vector are input into the Transformer neural network for trajectory attention operation to obtain the trajectory attention representation encoding vector; wherein, the expression of the trajectory attention operation is: ,in, represents the second query vector, Indicates the i A trajectory, h Indicates the height of the low-resolution video frame. w Indicates the width of the low-resolution video frame. Represents the location distribution map, represents the second key vector, represents the second value vector, v represents the first value vector, K represents the first bond vector, q represents the first query vector, represents the normalization function, d represents the number of square areas, T represents the trajectory attention representation encoding vector, t represents the tth frame of the low-resolution video; A reconstruction module is configured to input the trajectory attention representation encoding vector into a pre-trained upsampling reconstruction network for processing to obtain a reconstructed sequence; wherein the pre-trained upsampling reconstruction network includes: constructing an upsampling reconstruction network and a loss function; wherein the loss function is a weighted sum of pixel loss, perceptual loss, and trajectory loss; configuring an optimizer to calculate the loss value of the loss function, and using the optimizer to update the network parameters of the upsampling reconstruction network until the loss function converges, thereby obtaining the pre-trained upsampling reconstruction network; A sampling module is used to sample any frame of the low-resolution video using a bicubic interpolation method to obtain a sampling sequence of any frame; The super-resolution video generation module is used to generate a super-resolution video of the transmission channel based on the reconstruction sequence and the sampling sequence.

4. An electronic device, characterized in that: The electronic device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the video super-resolution reconstruction method of a transmission channel according to any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the video super-resolution reconstruction method of a power transmission channel according to any one of claims 1 to 2 are implemented.

Images