Video super-resolution reconstruction method and device and wearable equipment

Abstract

The invention relates to the technical field of video processing, and discloses a video super-resolution reconstruction method and device and wearable equipment, and the method comprises the steps: obtaining a to-be-reconstructed low-resolution video; reconstructing the to-be-reconstructed low-resolution video through a preset reconstruction model to obtain a target high-resolution video; the preset reconstruction model at least comprises a dense module provided with a first network and a second network, and the first network is used for performing depth separable convolution operation on the low-resolution video to be reconstructed by adopting an inverted residual mode to obtain local features; and the second network is used for performing global feature extraction on the to-be-reconstructed low-resolution video by adopting a self-attention mechanism, and reconstructing the to-be-reconstructed low-resolution video according to the local features and the global features. Compared with the prior art, the number of parameters required by the self-attention mechanism, the depth separable convolution and the inverted residual error technology is small, so that the use parameters can be reduced, the reasoning speed can be improved, and the user experience can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video super-resolution reconstruction method and device and wearable equipment. BACKGROUND

[0002] At present, when performing super-resolution (SR) reconstruction on a low-resolution (LR) video to obtain a high-resolution (HR) video, a hybrid model of a convolutional neural network (CNN) and a Transformer model can be used, that is, local feature extraction is performed on the low-resolution video by using the CNN, global feature extraction is performed on the low-resolution video by using the Transformer model, and then the high-resolution video is obtained by decoding the fusion of the local features and the global features, thereby realizing the super-resolution reconstruction.

[0003] However, when the above method is applied to an augmented reality (AR) glasses, the AR glasses have limited computing resources, while the CNN+Transformer method requires a large number of parameters, which means that the model needs to occupy more storage space, and more memory is needed to store and access these parameters during the inference process, thereby causing slow inference speed of the AR glasses and poor user experience. SUMMARY

[0004] The main purpose of the present application is to provide a video super-resolution reconstruction method, device and wearable equipment, which aims to solve the technical problem that the AR glasses have limited computing resources, but the CNN+Transformer method requires a large number of parameters, thereby causing slow inference speed of the AR glasses and poor user experience.

[0005] To achieve the above purpose, the present application provides a video super-resolution reconstruction method, which comprises the following steps:

[0006] obtaining a low-resolution video to be reconstructed;

[0007] reconstructing the low-resolution video to be reconstructed by using a preset reconstruction model to obtain a target high-resolution video;

[0008] The preset reconstruction model at least comprises a dense module provided with a first network and a second network, the first network is used for performing deep separable convolution operation on the low-resolution video to be reconstructed in an inverted residual manner to obtain local features, and the second network is used for performing global feature extraction on the low-resolution video to be reconstructed in a self-attention mechanism and reconstructing the low-resolution video to be reconstructed according to the local features and the global features.

[0009] In an embodiment, the preset reconstruction model further comprises a non-local residual module, the non-local residual module is connected with the second network;

[0010] The step of reconstructing the low-resolution video to be reconstructed by the preset reconstruction model to obtain the target high-resolution video comprises:

[0011] The first network is used for performing deep separable convolution operation on the low-resolution video to be reconstructed in an inverted residual manner to obtain local features;

[0012] The second network is used for performing global feature extraction on the low-resolution video to be reconstructed in a self-attention mechanism and reconstructing the low-resolution video to be reconstructed according to the local features and the global features to obtain an initial high-resolution video;

[0013] The non-local residual module is used for performing detail adjustment on the initial high-resolution video to obtain the target high-resolution video.

[0014] In an embodiment, the non-local residual module comprises a non-local attention module and a hierarchical residual module, the non-local attention module is connected with the second network and the hierarchical residual module;

[0015] The step of performing detail adjustment on the initial high-resolution video by the non-local residual module to obtain the target high-resolution video comprises:

[0016] The non-local attention module is used for performing global calibration on the initial high-resolution video;

[0017] The hierarchical residual module is used for performing hierarchical fusion on the calibrated initial high-resolution video to obtain the target high-resolution video.

[0018] In an embodiment, the dense module further comprises a disparity map generation module, the non-local residual module further comprises a disparity map refinement regression module, the disparity map generation module is connected with the second network and the non-local attention module, and the disparity map refinement regression module is connected with the hierarchical residual module;

[0019] The step of globally extracting features of the low-resolution video to be reconstructed by the second network using a self-attention mechanism further comprises:

[0020] extracting depth information of the low-resolution video to be reconstructed by the disparity map generation module to obtain a disparity map;

[0021] The step of fusing the calibrated initial high-resolution video by the hierarchical residual module further comprises:

[0022] Based on the disparity map, the initial high-resolution video is linearly added by the disparity map refinement regression module to obtain a target high-resolution video.

[0023] In an embodiment, the preset reconstruction model further comprises a low-resolution video super-resolution module connected with the non-local residual module;

[0024] The step of adjusting the initial high-resolution video by the non-local residual module further comprises:

[0025] The adjusted initial high-resolution video is color restored by the low-resolution video super-resolution module, and the restored initial high-resolution video is resolution enhanced to obtain a target high-resolution video.

[0026] In an embodiment, the low-resolution video super-resolution module comprises a color restoration module and a resolution enhancement module connected in sequence, and the color restoration module is further connected with the non-local residual module;

[0027] The step of color restoring the adjusted initial high-resolution video by the low-resolution video super-resolution module, and resolution enhancing the restored initial high-resolution video to obtain a target high-resolution video comprises:

[0028] The adjusted initial high-resolution video is color restored by the color restoration module;

[0029] The restored initial high-resolution video is resolution enhanced by the resolution enhancement module to obtain a target high-resolution video.

[0030] In an embodiment, the preset reconstruction model further comprises an image preprocessing module connected with the first network;

[0031] The step of reconstructing the low-resolution video to be reconstructed by the preset reconstruction model comprises:

[0032] The image preprocessing module is used for preprocessing the low-resolution video to be reconstructed, and the preprocessing includes at least one of key frame extraction, bad frame elimination, resolution adjustment, and data enhancement.

[0033] The first network is used for extracting local features of the preprocessed low-resolution video to be reconstructed.

[0034] The second network is used for extracting global features of the preprocessed low-resolution video to be reconstructed, and the low-resolution video to be reconstructed is reconstructed according to the local features and the global features.

[0035] In an embodiment, the first network includes a MobileNet V3 network, and the second network includes a MobileVit network.

[0036] In addition, to achieve the above object, the present application further provides a video super-resolution reconstruction device, which comprises:

[0037] A video acquisition unit is configured to acquire a low-resolution video to be reconstructed.

[0038] A resolution reconstruction unit is configured to reconstruct the low-resolution video to be reconstructed by using a preset reconstruction model, so as to obtain a target high-resolution video.

[0039] The preset reconstruction model at least includes a dense module provided with a first network and a second network, the first network is used for performing a depth separable convolution operation on the low-resolution video to be reconstructed in an inverted residual manner, so as to obtain local features, and the second network is used for extracting global features of the low-resolution video to be reconstructed by using a self-attention mechanism, and reconstructing the low-resolution video to be reconstructed according to the local features and the global features.

[0040] In addition, to achieve the above object, the present application further provides a wearable device, which comprises a memory, a processor, and a video super-resolution reconstruction program stored in the memory and executable on the processor, and the video super-resolution reconstruction program is used to implement the steps of the video super-resolution reconstruction method when executed by the processor.

[0041] The application provides a video super-resolution reconstruction method, device and wearable equipment, the method comprises the following steps: acquiring a low-resolution video to be reconstructed; and reconstructing the low-resolution video to be reconstructed by a preset reconstruction model to obtain a target high-resolution video; wherein the preset reconstruction model at least comprises a dense module provided with a first network and a second network, the first network is used for performing a depth separable convolution operation on the low-resolution video to be reconstructed in an inverted residual manner to obtain local features, and the second network is used for performing global feature extraction on the low-resolution video to be reconstructed in a self-attention mechanism, and reconstructing the low-resolution video to be reconstructed according to the local features and the global features. After the low-resolution video to be reconstructed is acquired, the first network in the preset reconstruction model can be used to perform a depth separable convolution operation on the low-resolution video to be reconstructed in an inverted residual manner to obtain local features, and then the second network can be used to perform global feature extraction on the low-resolution video to be reconstructed in a self-attention mechanism, and the low-resolution video to be reconstructed is reconstructed according to the local features and the global features to obtain a target high-resolution video. Compared with the existing CNN+Transformer mode which needs a large number of parameters, the number of parameters required by the self-attention mechanism, the depth separable convolution and the inverted residual technology is small, so that the application can reduce the use of parameters, improve the inference speed, and thus improve the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required to be used in the embodiments or the prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0044] Figure 1 The device structure schematic diagram of the hardware running environment involved in the embodiment of the present application;

[0045] Figure 2 The flowchart of the first embodiment of the video super-resolution reconstruction method of the present application;

[0046] Figure 3 The structure schematic diagram of the preset reconstruction model in the first embodiment of the video super-resolution reconstruction method of the present application;

[0047] Figure 4 The structure schematic diagram of the dense module and the network module in the first embodiment of the video super-resolution reconstruction method of the present application;

[0048] Figure 5 Flowchart of the second embodiment of the video super-resolution reconstruction method of the present application.

[0049] Figure 6 Structural diagram of the preset reconstruction model in the second embodiment of the video super-resolution reconstruction method of the present application.

[0050] Figure 7 Structural diagram of the non-local residual module in the second embodiment of the video super-resolution reconstruction method of the present application.

[0051] Figure 8 Structural diagram of the non-local attention module in the second embodiment of the video super-resolution reconstruction method of the present application.

[0052] Figure 9 Process diagram of the sub-pixel convolution module in the second embodiment of the video super-resolution reconstruction method of the present application.

[0053] Figure 10 Flowchart of the third embodiment of the video super-resolution reconstruction method of the present application.

[0054] Figure 11 Structural diagram of the preset reconstruction model in the third embodiment of the video super-resolution reconstruction method of the present application.

[0055] Figure 12 Structural diagram of the channel attention mechanism module in the third embodiment of the video super-resolution reconstruction method of the present application.

[0056] Figure 13 Structural diagram of the multi-scale channel attention module in the third embodiment of the video super-resolution reconstruction method of the present application.

[0057] Figure 14 Structural diagram of the gating mechanism in the third embodiment of the video super-resolution reconstruction method of the present application.

[0058] Figure 15 Structural block diagram of the first embodiment of the video super-resolution reconstruction device of the present application.

[0059] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0061] Reference Figure 1 , Figure 1 Structural diagram of the device of the hardware running environment involved in the embodiment scheme of the present application.

[0062] like Figure 1 As shown, the hardware operating environment involved in this application embodiment is a wearable device, which may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize communication between these components. The user interface 1003 may include a display screen, and optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. In this application, the wired interface of the user interface 1003 may be a USB interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0063] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0064] like Figure 1 As shown, the memory 1005, which is identified as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a video super-resolution reconstruction program.

[0065] exist Figure 1 In the device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the user equipment; the device calls the video super-resolution reconstruction program stored in the memory 1005 through the processor 1001 and executes the video super-resolution reconstruction method provided in the embodiments of this application.

[0066] It should be noted that currently, when performing super-resolution (SR) reconstruction on low-resolution (LR) videos to obtain high-resolution (HR) videos, it is generally achieved through a hybrid model of convolutional neural networks (CNN) and Transformer models. That is, CNN is used to extract local features from the low-resolution video, while Transformer is used to extract global features from the low-resolution video. Then, the local and global features are fused and decoded to obtain the high-resolution video, thus achieving super-resolution reconstruction.

[0067] However, when this approach is applied to augmented reality (AR) glasses, the limited computing resources of AR glasses, coupled with the large number of parameters required by the CNN+Transformer method, result in slow inference speed and a poor user experience for AR glasses.

[0068] Therefore, to address the aforementioned shortcomings, this embodiment provides a video super-resolution reconstruction method. After acquiring the low-resolution video to be reconstructed, the first network in the preset reconstruction model performs depthwise separable convolution operations using an inverted residual method to obtain local features. Then, a second network uses a self-attention mechanism to extract global features from the low-resolution video to be reconstructed. Based on the local and global features, the low-resolution video is reconstructed to obtain the target high-resolution video. Compared to the existing CNN+Transformer approach, which requires a large number of parameters, this embodiment reduces the number of parameters required by the self-attention mechanism, depthwise separable convolution, and inverted residual technique, thereby improving inference speed and enhancing user experience.

[0069] For ease of understanding, the following is combined with Figures 2 to 15 The video super-resolution reconstruction method provided in the embodiments of this application will be described in detail.

[0070] This application provides a video super-resolution reconstruction method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the video super-resolution reconstruction method of this application.

[0071] In this embodiment, the video super-resolution reconstruction method includes the following steps:

[0072] Step S10: Obtain the low-resolution video to be reconstructed.

[0073] It is understood that the method of this embodiment can be applied to scenarios where wearable devices perform super-resolution reconstruction, such as AR glasses, or other devices. This embodiment does not limit this application. The executing entity of this embodiment can be a super-resolution reconstruction device with functions of super-resolution reconstruction, data processing, network communication, and program execution, such as AR glasses, or other devices capable of performing the same or similar functions. The following describes this embodiment and the following embodiments in detail using the aforementioned AR glasses (hereinafter referred to as the device).

[0074] It should be understood that the aforementioned low-resolution video to be reconstructed can be the video that the aforementioned device needs to display. Since AR glasses can generally be set to a display resolution at the factory, such as 1080P, and will display at that resolution in subsequent use, the resolution of the aforementioned low-resolution video to be reconstructed can be the set resolution. However, to improve the visual effect, the low-resolution video to be reconstructed can be reconstructed to increase its resolution, for example, to 2K.

[0075] Step S20: Reconstruct the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video.

[0076] It is important to emphasize that the aforementioned preset reconstruction mode can be a model used to upscale low-resolution videos to be reconstructed. However, traditional hybrid models combining CNN and Transformer require a large number of parameters, resulting in slow inference speeds. Therefore, in this embodiment, the aforementioned preset reconstruction model includes at least: a dense module comprising a first network and a second network. The first network is used to perform depthwise separable convolution operations on the low-resolution video to be reconstructed using inverted residuals to obtain local features. The second network is used to extract global features from the low-resolution video to be reconstructed using a self-attention mechanism, and reconstruct the low-resolution video based on the local and global features.

[0077] The aforementioned first network can be a network with depthwise separable convolution and inverted residual modules. In conventional convolution operations, each convolution kernel can simultaneously process multiple input channels of the input tensor and mix them into a single output channel. However, the depthwise separable convolution in this embodiment decomposes this process into two steps: depthwise convolution and pointwise convolution. Depthwise convolution performs a convolution operation on each input channel using its own convolution kernel, without cross-channel mixing. Pointwise convolution mixes the output channels of the depthwise convolution using a 1x1 convolution kernel to generate the final output channel. This reduces the number of convolution kernels and the computational cost of convolution operations.

[0078] Traditional ResNet residual modules typically contain two convolutional layers, where the output dimension of the second convolutional layer is the same as the input dimension, allowing the input to "jump" directly to the output, forming a residual connection. In this embodiment, however, the pointwise convolutional layer in the inverted residual module is placed first, followed by the depthwise convolutional layer. That is, the number of channels is adjusted first, and then this number is maintained in the depthwise convolution, thereby further reducing the number of parameters and computational cost.

[0079] Therefore, the first network in this embodiment may include depthwise separable convolution and inverted residual modules. In practical use, after the device acquires the low-resolution video to be reconstructed, it can be input into the preset reconstruction model. The first network can extract local features from the low-resolution video to be reconstructed using depthwise separable convolution and inverted residual techniques, thereby reducing the number of parameters required. The aforementioned local features can be features within a small range of the low-resolution video to be reconstructed, which can be used to reflect details such as edges and textures. This embodiment does not impose any specific limitations on this.

[0080] The second network described above can be a network with a self-attention mechanism. This self-attention mechanism can capture the relationship between any two elements and extract information about each location in the low-resolution video to be reconstructed. Furthermore, this embodiment can learn complex feature information at each location through a multi-attention mechanism to obtain global features. These global features can be overall information, such as the overall layout, relationships between objects, and context; this embodiment does not impose any limitations on this. Since the parameters in the self-attention mechanism can be shared, the number of parameters can be further reduced. Therefore, in practical use, after the first network extracts local features, the second network can utilize the self-attention mechanism to extract global features from the low-resolution video to be reconstructed, thus obtaining global features.

[0081] After obtaining the aforementioned local and global features, the dense module can then use the learned local and global features to perform super-resolution reconstruction of the low-resolution video to be reconstructed, thereby obtaining the target high-resolution video.

[0082] Furthermore, to construct the aforementioned first and second networks, in this embodiment, the first network may include a MobileNet V3 network, and the second network may include a MobileVit network. That is, in this embodiment, the dense module can be constructed using a dense block network architecture, combining MobileVit's attention mechanism, a MobileNet V3 convolutional network with a residual network.

[0083] This dense module, through the aforementioned structure, fully leverages the advantages of attention mechanisms to obtain global information from keyframe images. It utilizes the self-attention mechanism of the MobileVit network to acquire location information for each frame in the low-resolution video to be reconstructed, and employs a multi-attention mechanism to learn complex feature information at each location in the image. Simultaneously, it can identify dependencies between different regions, thereby performing global feature calibration. It can learn the temporal continuity and spatial correlation in the video, completing spatiotemporal consistency calibration and ensuring smooth transitions and natural connections between frames.

[0084] This embodiment utilizes the MobileNet V3 network in the dense module to extract local features using depthwise separable convolutions and inverted residual modules, and then extracts global features using the MobileVit attention mechanism. Finally, it reconstructs the target high-resolution video using the obtained local and global features. Compared to existing methods employing CNN+Transformer, which require a large number of parameters, this embodiment reduces the number of parameters needed due to the smaller parameter count required by self-attention mechanisms, depthwise separable convolutions, and inverted residuals, thus improving inference speed and enhancing the user experience.

[0085] Meanwhile, this embodiment uses fewer parameters, thus reducing the computational load and consequently lowering the power consumption of the AR glasses. Furthermore, because this embodiment performs super-resolution reconstruction, compared to AR glasses without super-resolution reconstruction, it solves the video quality degradation problem caused by user movement while wearing the glasses, further improving the user experience.

[0086] Secondly, since this embodiment can achieve super-resolution reconstruction within AR glasses, compared to existing methods that rely on improving hardware performance (e.g., using high-definition cameras and high-performance light sensors), this embodiment can reduce the cost of hardware.

[0087] Furthermore, in order to improve the quality of the low-resolution video to be reconstructed, in this embodiment, the above-mentioned preset reconstruction model further includes: an image preprocessing module, which is connected to the first network;

[0088] The step of reconstructing the low-resolution video to be reconstructed using a preset reconstruction model includes:

[0089] The image preprocessing module preprocesses the low-resolution video to be reconstructed, including at least one of keyframe extraction, bad frame removal, resolution adjustment, and data augmentation. The first network extracts local features from the preprocessed low-resolution video to be reconstructed. The second network extracts global features from the preprocessed low-resolution video to be reconstructed, and the low-resolution video to be reconstructed is reconstructed based on the local features and the global features.

[0090] It should be noted that, for ease of understanding, refer to Figure 3 To explain, Figure 3 This is a schematic diagram of the structure of the preset reconstruction model in the first embodiment of the video super-resolution reconstruction method of this application. Figure 3 As shown, in this embodiment, after obtaining the low-resolution video to be reconstructed (i.e., Figure 3 After processing low- to medium-quality videos, they can be input into the image preprocessing module of the preset reconstruction model.

[0091] In this image preprocessing module, keyframes can be extracted from the low-resolution video to be reconstructed at preset intervals (i.e., ...). Figure 3 (Frame extraction). For example, if the preset interval is 3 frames, then images at intervals of 3 frames in the low-resolution video to be reconstructed can be used as keyframe images (i.e., ... Figure 3 The keyframes in the low-resolution video to be reconstructed are extracted. Alternatively, keyframe extraction can be omitted, and all frames in the low-resolution video to be reconstructed can be used as keyframes for further processing. Whether to extract keyframes and the preset interval frame number when extracting them can be set according to the actual situation, and this embodiment does not impose any restrictions on this.

[0092] After extracting the keyframes, some images may be blurry or of low quality. Therefore, the image preprocessing module described above can remove bad frames from the extracted low-resolution video to be reconstructed (i.e., ...). Figure 3 Remove bad frames from the video (i.e., remove frames that do not meet the requirements) to obtain the low-resolution video to be reconstructed (i.e., the video itself). Figure 3 (The final keyframe in the middle);

[0093] After removing bad frames, in order to unify all the obtained frame images, the aforementioned image preprocessing module can also adjust the resolution of each frame image in the low-resolution video to be reconstructed after removal, adjusting it to the consistent resolution required subsequently (e.g., 1080P). Figure 3 (Medium resolution adjustment);

[0094] After adjusting the resolution, data augmentation (i.e., data enhancement) can be performed on the adjusted low-resolution video to be reconstructed to enrich the data. Figure 3 Data augmentation can be performed in various ways, including but not limited to spatial transformation, color transformation, cropping and flipping, adding Gaussian noise and Gaussian blur, etc. This embodiment does not limit these methods.

[0095] After data augmentation, the augmented low-resolution video to be reconstructed can be transmitted to the dense module (i.e., Figure 3 The MobileNet V3 network (comprised of a dense module consisting of MobileVit attention mechanism and MobileNet V3 convolutional blocks) performs local feature extraction, which is then transmitted to the MobileVit network for global feature extraction. Based on the local and global features, the low-resolution video to be reconstructed is then reconstructed.

[0096] However, to improve inference speed before input, in this embodiment, a segmentation module can be set after the image preprocessing module and before the dense module. Figure 3 (not shown in the image) and a first attention mechanism layer (i.e. Figure 3 The attention mechanism convolutional layer before the dense module (hereinafter, all attention mechanism layers are attention mechanism convolutional layers) means that the preset reconstruction model also includes: a segmentation module and a first attention mechanism layer, the first attention mechanism layer contains the second network, and the first attention mechanism layer is connected to the segmentation module and the dense module respectively.

[0097] After obtaining the preprocessed low-resolution video to be reconstructed, the segmentation module can first adjust the low-resolution video to make each frame image of it a preset size. The specific preset size is not limited in this embodiment. Then, the adjusted frame images are divided into several image blocks. The size of the image block can be 3×3, or other sizes. This embodiment can refer to the Vision Transformer network architecture and uses 3×3 for illustration.

[0098] After obtaining several image blocks, they can be sorted according to their position in the adjusted frame image to flatten the two-dimensional low-resolution image blocks into a one-dimensional sequence. Each image block is treated as a separate token, and an initial feature vector, denoted as I, is obtained from the pixel values ​​of each flattened image block through a linear transformation.LR Then, position embeddings are added to each flattened image patch (i.e., each token) to provide the position information of each image patch in the original image (i.e., the frame image in the low-resolution video to be reconstructed), and the final target feature vector containing position information is obtained, denoted as I. z This process can be obtained through the following first formula (i.e., Formula 1):

[0099] I z =WI LR Formula 1;

[0100] Among them I z Then it is the vector containing location information mentioned above, I LR Then, the eigenvectors are as described above, and W is the learnable weight matrix.

[0101] After obtaining the target feature vector I containing the location information mentioned above. z Then, the target feature vector can be transmitted to the first attention mechanism layer, where the MobileVit network processes the target feature vector I. z Perform self-attention calculation to complete feature extraction, obtain the first extraction result, and further improve the subsequent results.

[0102] In this first attention mechanism layer, the query (Q), key (K), and value (V) of each token can be obtained by multiplying them with the corresponding learning weight matrix. Specifically, this can be obtained through the following second formula (i.e., Formula 2):

[0103] Q = I z W Q K = I z W K V = I z W V Formula 2;

[0104] Where Q represents the query, W... Q To query the corresponding learning weight matrix, K is the key, W K Let V be the learning weight matrix corresponding to the key, and W be the value. V The value is the learning weight matrix corresponding to the value.

[0105] After obtaining the query (Q), key (K), and value (V), an attention score can be calculated to measure the similarity between the query Q and each key K. Then, a softmax function is used for normalization, followed by a weighted summation to obtain the feature output by the first attention mechanism layer (i.e., the first extraction result mentioned above), denoted as F. s Specifically, this can be achieved through the following third formula (i.e., formula 3) and fourth formula (i.e., formula ...).

[0106] Equation 4) yields:

[0107]

[0108] F s =S w Formula 4;

[0109] Where score is the normalized result, K T d is the transpose of the key, and I is the transpose of the key. z The dimension, S w For the attention weight (i.e., score) corresponding to value V, F s The features output by the convolutional layer of this attention mechanism (i.e., the weighted summation result, which is also the first extraction result mentioned above).

[0110] The first extraction result can then be input into the dense module for feature extraction. To facilitate dense connections, it is combined with... Figure 3 , Figure 3 The medium-density module may include at least one set of sequentially connected network modules and a first feature cascade layer (i.e., Figure 4 The network module, the first feature cascade layer, and the second attention mechanism layer are connected in a dense connection manner, that is, each first feature cascade layer is not only connected to the network module in its group, but also connected to the input of the network modules of all previous groups.

[0111] The network module can include a convolutional layer and an attention mechanism layer. The convolutional layer can use the first network, namely the MobileNet V3 convolutional network mentioned above, and the attention mechanism layer can use the second network, namely the MobileVit attention mechanism mentioned above. Therefore, local information of the input frame image can be extracted through MobileNet V3, and information of each pixel in the frame image and complex feature information of each pixel can be extracted through MobileVit to form global features.

[0112] To improve performance, the network module in this embodiment may also include multiple convolutional layers and multiple attention mechanism layers. (See reference...) Figure 4 , Figure 4 This is a schematic diagram of the structure of the dense module and the network module in the first embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 3 As shown, in this embodiment, the dense module can be described using three sequentially connected network modules, a first feature cascade layer, and a second attention mechanism layer.

[0113] The network module may include the aforementioned first network and the aforementioned second network. After obtaining the first extraction result through the first attention mechanism layer, the first network module can perform feature extraction on the first extraction result, obtain local features, and then transmit them together with the first extraction result to the first feature concatenation layer. Next, feature concatenation is performed on the global features and local features in the extracted feature map (i.e.,...). Figure 4 The first feature concatenation layer (using cascaded features) yields a first stitched result, which can be stitched along the channel dimension to enhance feature richness. This first stitched result is then processed by a second attention mechanism layer using a MobileVit network for feature extraction, yielding global features. Since the dense modules in this implementation employ dense connections, during feature stitching in the subsequent first feature concatenation layer, the local features output by the current network module, the global features output by the previous second attention mechanism layer, and the first extraction result from the first network module can be stitched together to further extract and refine features. Finally, video super-resolution reconstruction can be performed based on the extracted local and global features to obtain the target high-resolution video.

[0114] The specific process described above can be obtained through the following fifth formula (i.e., formula 5):

[0115] F D =AC n (Cat(B1, B2, ..., B) n )) Formula 5;

[0116] Among them B n Here, AC represents the local features extracted by the nth network module, and Cat represents the spatial feature concatenation operation (i.e., the feature concatenation mentioned above). n For the feature extraction operation of the second attention mechanism layer mentioned above, F D These are the global features mentioned above. This operation, which is easy to implement in deep learning, can complement the extracted deep features and combine them to improve the generalization performance of the model.

[0117] In order to obtain the above local feature B n , continue as Figure 4 As shown, in this embodiment, the network module includes: an extraction module, a second feature cascade layer, and a first convolutional layer. The first convolutional layer carries the first network. The second feature cascade layer is connected to the extraction module and the first convolutional layer, respectively. The first convolutional layer is connected to the first feature cascade layer.

[0118] In the process of obtaining local features by using the network module to extract features from the first extraction result or the global features output by the previous second attention mechanism layer through the first network, the extraction module can first extract features from the first extraction result or the global features output by the previous second attention mechanism layer to obtain the second extraction result. Then, the second feature concatenation layer can concatenate the features of the second extraction result to obtain the second concatenation result. Finally, the first convolutional layer can use the first network to extract features from the second concatenation result to obtain local features.

[0119] To further improve reasoning speed, such as Figure 4 As shown, in this embodiment, the extraction module includes at least two extraction units, each of which is connected sequentially and then connected to the second feature cascade layer in a skip connection manner;

[0120] It should be noted that in this embodiment, each extraction unit is connected to the second feature cascade layer using a jump connection method, such as... Figure 4 As shown, this embodiment uses two extraction units for illustration, referred to as the first extraction unit and the second extraction unit respectively. The input of the first extraction unit is connected not only to the second attention mechanism layer of the previous group, but also to the second feature cascade layer in the network module. When it is the first network module, the first extraction unit can also be connected to the first attention mechanism layer. The output of the first extraction unit can be connected to the input of the second extraction unit and the second feature cascade layer. The output of the second feature extraction unit can also be connected to the second feature cascade layer.

[0121] In practical use, the first extraction unit can perform feature extraction on the first extraction result or the global features output by the previous second attention mechanism layer to obtain the second extraction result corresponding to the first extraction unit, and transmit the second extraction result to the second extraction unit and the second feature cascade layer. The second extraction unit can then perform feature extraction on the second extraction result to obtain the second extraction result corresponding to the second extraction unit, and transmit it to the second feature cascade layer. The second feature cascade layer then concatenates the second extraction result output by the first extraction unit and the second extraction result output by the second extraction unit, as well as the global features output by the first extraction result or the previous second attention mechanism layer, to obtain the second concatenated result.

[0122] If we denote the local features output by the network module as B n Then, for the local feature B output by the above network module n The feature extraction process can be obtained through the following sixth formula (i.e., formula 6):

[0123]

[0124] The above For the 3rd MobileNet V3 convolutional layer in the nth network module (i.e. Figure 4 The first convolutional layer), Cat is the spatial feature concatenation operation (i.e., the second feature cascade layer), X n The input to the nth network module (i.e., the first extraction result or the global features output by the previous second attention mechanism layer) is used. Since this embodiment has two extraction units and one first convolutional layer, there are three feature extractions. For X n The second extraction result output after the first feature extraction (i.e., the first extraction unit) is as follows: The second extraction result is output after the second feature extraction (i.e., the second extraction unit).

[0125] To further improve speed, continue to refer to Figure 3 In this embodiment, the extraction unit may include: a third attention mechanism layer with a second network and a second convolutional layer with a first network, wherein the third attention mechanism layer and the second convolutional layer are connected in sequence, and the second convolutional layer is also connected to the second feature cascade layer.

[0126] It should be noted that when the third attention mechanism layer is the first extraction unit of the first network module, feature extraction can be performed on the first extraction result output by the first attention mechanism layer. When the third attention mechanism layer is not the first extraction unit in the first network module, feature extraction can be performed on the global features obtained by the previous second attention mechanism layer. When the third attention mechanism layer is not the first extraction unit, feature extraction can be performed on the second extraction result output by the previous extraction unit. Furthermore, in this embodiment, the feature extraction result of the third attention mechanism layer is referred to as the third extraction result.

[0127] After obtaining the third extraction result, the first network is used to extract features from the third extraction result through the second convolutional layer to obtain the second extraction result of the extraction unit. This can be specifically expressed by the following seventh formula (i.e., formula 7) and eighth formula (i.e., formula 8):

[0128]

[0129] Among the above For the third attention mechanism layer within the first extraction unit of the nth network module, the above For the second convolutional layer within the first extraction unit of the nth network module, the above For the third attention mechanism layer within the second extraction unit of the nth network module, the above This is the second convolutional layer within the second extraction unit of the nth network module. F can be obtained using the first to eighth formulas described above. D .

[0130] Secondly, continue to refer to Figure 3 After the dense module, upsampling and depth sampling can be added. Upsampling can include a deconvolution layer, which can also be composed of MobileNet V3 convolutional layers. Upsampling can further extract and refine the enlarged features. Depth sampling can be used to restore the feature map to the desired size. These two operations can enlarge the residual image before image reconstruction and linear feature addition (i.e.,...) Figure 4 (The linear features are added together) to obtain a high-resolution video of the target.

[0131] When adding linear features, bicubic interpolation can be used, which can be expressed by the following ninth formula (i.e., formula 9):

[0132] I SR =Bic(I SR )+OC 0 (DeC(F D )) Formula 9;

[0133] Among them I SR For high-resolution video, Bic is bicubic interpolation, and OC is... 0 For deep sampling operations, DeC is the aforementioned deconvolution layer.

[0134] It needs to be emphasized that, such as Figure 4 As shown, in this embodiment, the output of each second convolutional layer in the network module is directly connected to the subsequent second feature cascade layer. In the dense module, the input of each network module is directly connected to all subsequent first feature cascade layers. This means that the input of each first feature cascade layer is the union of the outputs of all previous layers. This design allows the network to reuse the features of previous layers, instead of simply passing the features to the next layer as in traditional networks. This connection method can be called dense connection, which further reduces the amount of computation.

[0135] It should also be emphasized that when the preset reconstruction model is in the learning phase, its input includes not only the low-resolution video to be reconstructed, but also its corresponding high-resolution video. This video undergoes preprocessing operations by the aforementioned image preprocessing module before being transmitted to the dense module for feature extraction and learning. This preprocessing process corresponds one-to-one with the preprocessing process for the low-resolution video described above, and will not be elaborated upon in this embodiment. When adjusting the resolution, the high-resolution video can be adjusted to 2K, but other resolutions are also possible; this embodiment does not impose any limitations on this.

[0136] During the learning phase, the dense module learns image detail information to perform detailed super-resolution reconstruction of low-resolution keyframe images. Furthermore, this dense module can better leverage the long-term dependency modeling capabilities of the attention mechanism, helping it better capture dependencies between different locations in the image and learn more inter-block information. This avoids gradient propagation problems, better models long-term dependencies, and effectively improves the network's reconstruction performance by learning more inter-block information. In summary, this dense block network learns the mapping relationship and feature map information from low-resolution keyframes to high-resolution keyframes.

[0137] Furthermore, continue as Figure 5 As shown, in this embodiment, skip connections are used between each extraction unit and the second feature cascade layer. Since edge contour information and high-frequency texture information are particularly important during image restoration, treating the features extracted by the last feature extraction unit and the inter-layer features transmitted by the skip connections equally might lead the model to focus only on low-frequency information (such as color and lighting) while ignoring high-frequency information, resulting in suboptimal high-resolution video restoration. Therefore, in this embodiment, the second feature cascade layer may include a target loss function. This target loss function is constructed by setting a first target weight for the first loss function of the extraction unit and a second target weight for the second loss function of the skip connections. The first and second target weights are determined through adaptive training.

[0138] It should be noted that the first loss function of the extraction unit mentioned above can be the loss function used by the extraction unit during training. For ease of explanation later, since the extraction unit may include a third attention mechanism layer and a second convolutional layer, the first loss function corresponding to the extraction unit can be denoted as L. MobileNetV3 +L MobileVit L MobileNetV3 L is the loss function of the third attention mechanism layer. MobileVit This is the loss function for the second convolutional layer.

[0139] The second loss function for the aforementioned skip connections can be the loss function corresponding to the skip connections between each extraction unit and the second feature cascade layer, and can be denoted as L. Skip .

[0140] This embodiment can incorporate the aforementioned first target weight and second target weight to treat the image features extracted by the convolutional layer and the attention mechanism layer equally, while enriching the inter-layer information extracted by the residual network and the convolutional layer. The first target weight can be denoted as α, and the second target weight as β. Both weights are independent and learnable parameters, and they borrow from the task-oriented attention mechanism concept, adaptively fluctuating between [0, 1] in conjunction with the following target loss function. If the target loss function is denoted as L, then the specific target loss function is: L = α(L MobileNetV3 +L MobileVit )+βL Skip .

[0141] Therefore, during training, training data can be acquired first. This training data may include the low-resolution video to be reconstructed and its corresponding high-resolution video. The data is transmitted to the dense module in the initial reconstruction model for adaptive training. In the dense module, the second feature cascade layer can continuously adjust the first initial weight α and the second initial weight β according to the training results to obtain the first target weight α and the second target weight β. Thus, the target loss function mentioned above can be obtained, and the initial reconstruction model can also be transformed into the preset reconstruction model mentioned above.

[0142] During adaptive training, when the detailed and contextual information learned by skip connections at different levels is less important, while the image feature information transmitted by the attention mechanism and convolution is more important, β can be decreased and α increased; conversely, α can be decreased and β increased. This task-oriented mechanism ensures that in the MobileVit attention-based convolutional network, the hierarchical feature information learned by each residual connection positively impacts the performance of the video super-resolution network in learning low-resolution image features, further improving reconstruction quality.

[0143] In this embodiment, after acquiring the low-resolution video to be reconstructed, the device first performs depthwise separable convolution operations on it using the inverted residual method in the first network of the preset reconstruction model to obtain local features. Then, the second network uses a self-attention mechanism to extract global features from the low-resolution video to be reconstructed. Based on the local and global features, the low-resolution video to be reconstructed is reconstructed to obtain the target high-resolution video. Compared with the existing CNN+Transformer approach, which requires a large number of parameters, this embodiment reduces the number of parameters used and improves inference speed, thereby enhancing the user experience, because the self-attention mechanism, depthwise separable convolution, and inverted residual technique require fewer parameters.

[0144] refer to Figure 5 , Figure 5 This is a flowchart illustrating the second embodiment of the video super-resolution reconstruction method of this application.

[0145] To further capture global information from frame images, learn internal structure and texture information, thereby ensuring that the high-resolution image recovered from the low-resolution frame image has more natural brightness, clearer image, and more realistic edge contours, such as... Figure 6 As shown, in this embodiment, the preset reconstruction model mentioned above further includes: a nonlocal residual module, which is connected to the second network in the dense module;

[0146] The step of reconstructing the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video includes:

[0147] Step S21: The first network is used to perform depthwise separable convolution on the low-resolution video to be reconstructed using the inverted residual method to obtain local features;

[0148] Step S22: The second network uses a self-attention mechanism to extract global features from the low-resolution video to be reconstructed, and reconstructs the low-resolution video to be reconstructed based on the local features and the global features to obtain an initial high-resolution video;

[0149] Step S23: Perform detail adjustments on the initial high-resolution video using the non-local residual module to obtain the target high-resolution video.

[0150] For ease of understanding, please refer to Figure 6 To explain, Figure 6 This is a schematic diagram of the structure of the preset reconstruction model in the second embodiment of the video super-resolution reconstruction method of this application. Figure 6 As shown, after obtaining global features in the last second attention mechanism layer of the dense module, the features can be passed to upsampling and deep sampling. After upsampling and deep sampling are completed, image reconstruction can be performed to obtain the initial high-resolution video, which is then passed to the non-local residual module (i.e., Figure 6 In the nonlocal residual network based on the MobileVit attention mechanism, the aforementioned nonlocal residual module can be a nonlocal residual network based on the MobileVit attention mechanism. Specifically, it can be located after the linear summation of the features. It can capture the global information of the feature map and use structural self-similarity loss to determine the similarity between different positions in the feature map, learn the internal structure and texture information of the image, and at the same time learn the temporal continuity and spatial correlation between video frames. It focuses on the global information in the feature map, and then adaptively adjusts the feature map of the key frame, performs detailed adjustments and optimizations to the feature map, and performs global calibration, thereby ensuring that the high-resolution image recovered from the low-resolution key frame image has more natural brightness, clearer image, and more realistic edge contours.

[0151] Furthermore, in order to achieve the above-mentioned detailed adjustments, in this embodiment, the nonlocal residual module includes: a nonlocal attention module and a hierarchical residual module, wherein the nonlocal attention module is connected to the second network and the hierarchical residual module;

[0152] The step of performing detail adjustments on the initial high-resolution video using the non-local residual module to obtain the target high-resolution video includes:

[0153] The initial high-resolution video is globally calibrated using the non-local attention module; the calibrated initial high-resolution video is then hierarchically fused using the hierarchical residual module to obtain the target high-resolution video.

[0154] like Figure 6 As shown, the nonlocal residual module may include a nonlocal attention module based on an attention mechanism (i.e., Figure 7 Non-local modules) and hierarchical residual modules (i.e. Figure 7 The non-local attention module can be located after the linear feature addition and before the hierarchical residual module.

[0155] The aforementioned nonlocal attention module can be a module built based on the nonlocal residual module of MobileVit, and the aforementioned hierarchical residual module can be a module composed of multiple residual connections. In this embodiment, the nonlocal residual module based on MobileVit can be used as the basic unit and integrated into the MobileNet V3 convolutional neural network to form a nonlocal residual module.

[0156] Reconstructing high-quality images requires accurately restoring the structural and textural relationships between different regions. By introducing a non-local attention mechanism, the network can better learn the semantic information of the input image and the correlations between pixels. The hierarchical residual module utilizes residual connections to optimize feature propagation and gradient flow. By establishing residual connections between different layers, the network can more effectively reuse and fuse feature information from different layers, thereby enhancing the expressive power of features.

[0157] It should be noted that in this embodiment, the non-local attention module and the hierarchical residual module can be iterated multiple times and then integrated with the deep network, so that the network can give full play to the effect of the self-attention mechanism.

[0158] For ease of understanding, please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of the nonlocal residual module in the second embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 7As shown, the nonlocal residual module in this embodiment may include several nonlocal residual blocks. This embodiment uses four nonlocal residual blocks for illustration, and each nonlocal residual block may include: a nonlocal attention module and a hierarchical residual module.

[0159] In practical use, it can first be achieved through a MobileNet V3 convolutional neural network (i.e., Figure 8 The process of extracting frame images from the reconstructed initial high-resolution video using MobileNet V3 convolutional blocks can be represented by the following formula 10:

[0160] F s1 =f 0 (I LR1 ) Formula 10;

[0161] Among them I LR1 For the initial high-resolution video obtained after reconstruction, f 0 For MobileNet V3 convolutional blocks,

[0162] F s1 The extracted frame image;

[0163] Next, a non-local attention module is used for feature extraction to obtain global information and dependencies, completing global calibration. This ensures that the high-resolution image reconstructed from the low-resolution keyframe image has more natural brightness, clearer image, and more realistic edge contours. Then, a hierarchical residual module is used to perform hierarchical fusion of the extracted features. Low-level features help restore local details of the image, while high-level features help maintain the overall structure and semantic information of the image. Finally, the features before and after the convolutional network can be linearly stacked through global skip connections, thereby realizing the processing of the non-local residual network based on the attention mechanism. The multi-layer stacked hierarchical residual module can linearly align the low-dimensional and high-dimensional feature maps of the keyframe, realizing linear mapping of keyframe feature map pairs, which can reduce the speed of reconstructing low-resolution images into high-resolution images and improve the network's generalization performance.

[0164] Furthermore, continue as Figure 8 As shown, in this embodiment, the non-local attention module may include a second network. This second network employs a self-attention mechanism to correlate the extracted frames of the initial high-resolution video, and based on this correlation, global calibration of the initial high-resolution video can be performed. The specific process is described in [reference needed]. Figure 8 , Figure 8 This is a schematic diagram of the structure of the non-local attention module in the second embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 8 As shown, the nonlocal residual module in this embodiment may include several nonlocal residual blocks. This embodiment uses four blocks for illustration, and they are connected in sequence.

[0165] In practical applications, the extracted initial high-resolution video frame images are first subjected to dimensionality reduction using a non-local attention module. Let's assume the input frame image size to the non-local attention module is M×N, and the number of channels is C, denoted as X. M ×N×C (Right now Figure 8 In the middle (X), dimensionality reduction is performed using 1×1 convolution, reducing its feature dimension to X. M×N×C / 2 This effectively reduces the computational burden;

[0166] Next, the self-similarity of the dimensionality-reduced feature maps is calculated using a structural similarity function, which can be...

[0167]

[0168] In this process, a 3×3 MobileNet V3 convolutional layer is first used to extract features from the dimensionality-reduced frame image to obtain... as well as

[0169] To facilitate computation and enable the model to learn the dependencies between different regions within the feature map, an attention-weighted aggregation strategy is used to convert the obtained feature map information into a matrix. After

[0170] Then, with the nonlocal transformation coefficients w(x) j The self-similarity matrix of the frame image is obtained by transposing and multiplying. take;

[0171] Simultaneously, a 1×1 MobileNet V3 convolutional network (i.e., ...) is used in the non-local attention module. Figure 6 The algorithm performs calculations on each pixel block in the frame image to achieve cross-channel information integration and interaction, enhance the non-linear characteristics of the network, and generate a weight matrix containing information on the relationships between pixels at different locations in the frame image. N represents the number of pixels in the frame image used by the non-local attention module. Next, let z... s With z w Multiplying two matrices yields a nonlocal matrix.

[0172] After converting the nonlocal matrix into a 3D frame image, it is then subjected to a 1×1 convolution (i.e., Figure 7 (h) Obtain frame image Z containing self-similarity information M×N×C / 2 Finally, let X M×N×C / 2 With Z M×N×C / 2 Adding corresponding elements, corresponding to the initial channel descent, achieves global calibration. Then, using a 1×1 upscaling method, the feature map channels are increased to obtain the final result Y.M×N×C The convolution process in this step integrates the MobileNet V3 convolutional modules using grouped convolution. Grouped convolution significantly reduces the number of parameters by dividing the input feature map into multiple groups and performing convolution operations independently within each group. This makes the model more lightweight, reduces computational resource consumption, and helps prevent overfitting. Grouped convolution is a sparse operation, making it easier for the model to learn important feature representations during training, providing a certain degree of regularization, reducing computational complexity, and improving the model's inference speed.

[0173] After obtaining the initial high-resolution video Y after calibration M×N×C Then, it can be transmitted to the hierarchical residual module, which can be configured with several residual modules. In this embodiment, eight residual modules are used for illustration, that is, each residual module is connected to the non-local attention module in a network cascade manner; the residual module can include the first network and the second network mentioned above.

[0174] In practical use, the calibrated initial high-resolution video can be grouped first. In this embodiment, it can be grouped according to paired frame images and divided into 4 groups for explanation. Then, it is transmitted to the first network and the second network for feature extraction, and hierarchical fusion is performed based on the corresponding extraction results. The non-local residual module can adopt a global residual connection method, that is, the input of the first non-local attention module is connected to the output of the last non-local residual block. The above hierarchical fusion process can be expressed by the following formula 11:

[0175] F D1 =B 4 (B 3 (B 2 (B 1 (F s1 )))) Formula 11;

[0176] Among them B i (i = 1, 2, 3, 4) represents the i-th nonlocal residual block, F D1 This is the result of hierarchical fusion.

[0177] It needs to be emphasized that, continuing as Figure 9 as well as Figure 9 As shown, in order to generate the final super-resolution image, the nonlocal residual module in this embodiment does not use the traditional deconvolution method to enlarge the image.

[0178] While deconvolution can magnify images, its computation process can be complex and inefficient.

[0179] Therefore, this embodiment uses the subpixel convolution method in ESPCN to amplify the image. That is, after the hierarchical fusion, a subpixel convolution module can be set up. This subpixel convolution module can be connected to the hierarchical residual module of the last non-local residual block. After obtaining the fusion result, the fusion result can be amplified through the above-mentioned subpixel convolution module to obtain the target high-resolution video.

[0180] The subpixel convolutional layer in this subpixel convolution module upsamples low-resolution images, which can restore image details, rearrange the output of the convolutional layer, and reduce image blurring.

[0181] The enlarged initial high-resolution video can then be used as the target high-resolution video.

[0182] Specifically, refer to Figure 9 , Figure 9 This is a schematic diagram of the subpixel convolution module process in the second embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 6 As shown, the sub-pixel convolution module described above can rearrange the fusion result into an r×r matrix (i.e., Figure 7 W×H×r 2 C is processed by subpixel convolution to obtain rW×rH×C), where r is the parameter of the subpixel convolution layer, representing the number of subpixels occupied by each pixel. Each element in this matrix represents a pixel in the high-resolution image. The fusion result is upsampled through the subpixel convolution layer. Simultaneously, the subpixel convolution layer can recover the edge loss caused by the convolution operation, restoring edge information to the higher-resolution image. Furthermore, by increasing the number of channels in the input feature map, the subpixel convolution layer increases the expressive power of the features, allowing the model to learn on higher-resolution feature maps, thereby improving the model's performance and accuracy.

[0183] In this embodiment, there can be 8 subpixel convolution modules, but other numbers are also possible. This embodiment does not limit this number.

[0184] It should also be emphasized that the optimization function of the nonlocal residual module in this embodiment can be represented by the mean absolute error, which can be specifically expressed by the following formula 12:

[0185]

[0186] Among them, the above L SR For the target high-resolution video, N is the number of pixels in the fused frame image. For the i-th pixel in the frame image of the initial high-resolution video obtained after reconstruction, Let be the i-th pixel in the frame image of the fused result, and 1 be the L1 loss function.

[0187] Furthermore, considering that AR glasses generally display 3D video with a certain depth information, therefore, continuing as follows... Figure 6 and Figure 6 As shown, in this embodiment, the dense module further includes a disparity map generation module, and the nonlocal residual module further includes a disparity map refinement and regression module. The disparity map generation module is connected to the second network and the nonlocal attention module, and the disparity map refinement and regression module is connected to the hierarchical residual module.

[0188] After the step of extracting global features from the low-resolution video to be reconstructed using the self-attention mechanism through the second network, the method further includes:

[0189] The disparity map generation module extracts depth information from the low-resolution video to be reconstructed to obtain a disparity map.

[0190] After the step of performing hierarchical fusion on the calibrated initial high-resolution video using the hierarchical residual module, the method further includes:

[0191] Based on the disparity map, the initial high-resolution video is linearly added using the disparity map refinement regression module to obtain the target high-resolution video.

[0192] It should be noted that the disparity map generation module described above can be connected after the last second attention mechanism layer in the dense module and before upsampling. The disparity map refinement and regression module can be connected after the last non-local residual block.

[0193] The disparity map generation module mentioned above may include a regularization extraction layer (i.e. Figure 7 (Regularized extraction) and window matching computation layer (i.e.) Figure 10 (MSD window matching calculation), wherein the above regularization extraction layer can be connected to the last second attention mechanism layer and the window matching calculation layer in the dense module, respectively;

[0194] The regularization extraction layer can use the L1 regularization loss function to extract the regularization information of the frame image output by the last second attention mechanism layer, providing a positive reference for learning and processing the depth information of AR videos. This layer can constrain the complexity of the model and encourage parameter sparsity by adding the sum of the absolute values ​​of the model parameters as a regularization term to the loss function. L1 regularization imposes constraints on the weight parameters of the convolutional layer. Since the L1 regularization term is the sum of the absolute values ​​of the weights, its derivative will jump when the weight is zero (from -1 to 1 or from 1 to -1). This property makes it easier for some weight values ​​to be pushed to zero during the optimization process, thereby achieving weight sparsity.

[0195] The regularization extraction layer can transmit the obtained regularization information to the window matching calculation layer. The window matching calculation layer can include MSD window matching calculation. Based on the regularization information, the network using MSD window matching calculation can generate a low-resolution disparity map (i.e., the aforementioned disparity map). This disparity map can be a difference map of binocular imaging. In the human visual system, due to the distance between the two eyes, the image position of the same object on the retinas of the two eyes is slightly different; this difference is called disparity. Specifically, the window matching calculation layer can first determine a reference window and candidate windows for the frame image based on the regularization information. The reference window can be a window used for reference, selected from a small rectangular region in the frame image of the low-resolution video to be reconstructed before input to the regularization extraction layer. This reference window can contain the features or patterns to be matched. The candidate window can be a window used to compare with the reference window to determine depth information, and can be a window corresponding to a larger search region defined for the received frame image.

[0196] After determining the reference window and candidate windows, depth information can be extracted to generate a disparity map. The specific process is as follows: determine the number of pixels in the window, and determine the pixel values ​​of the pixels in the reference window and the pixel values ​​of the pixels in the candidate windows; determine the similarity score based on the number of pixels in the window, the pixel values ​​of the pixels in the reference window, and the pixel values ​​of the pixels in the candidate windows; generate a disparity map based on the similarity score.

[0197] The number of pixels in the aforementioned window can be the number of pixels contained in the reference window or the candidate window. The pixel value of the pixel in the aforementioned reference window can be the grayscale value or intensity value corresponding to the pixel in the reference window. The pixel value of the pixel in the aforementioned candidate window can be the grayscale value or intensity value corresponding to the pixel in the candidate window.

[0198] The similarity score mentioned above can be considered as the similarity score between the reference window and the candidate window. The specific calculation can be expressed by the following formula 13:

[0199]

[0200] Where MSD is the similarity score, N is the number of pixels in the window, l1(i) is the pixel value of the i-th pixel in the reference window, and l2(i) is the pixel value of the i-th pixel in the candidate window.

[0201] After obtaining the similarity score, the similarity scores of all candidate windows are compared. The window with the smallest MSD value can be selected from the candidate windows. This candidate window can be the window most similar to the reference window and is used as the best matching window.

[0202] After determining the optimal matching window, its position information in the frame image can be determined. The position information and the similarity score of the optimal matching window are output together to generate a disparity map and send it to the disparity refinement regression module. The disparity map generated by the disparity map generation module is a lightweight feature matching disparity map. This disparity map mainly contains the depth information in the keyframes, which prepares the input for the disparity refinement regression module in the next part of the non-local residual network and provides depth-level information reference for subsequent reconstruction of high-resolution video.

[0203] After obtaining the disparity map, it can be used as input to the disparity map refinement regression module. In the disparity map refinement regression module of this embodiment, the unsupervised structural similarity index (SSIM) loss function can be used to determine the image residual between the disparity map and the frame image of the low-resolution video to be reconstructed before the input regularization layer through an upsampling network. Then, the image residual is added to the disparity map to obtain the accurate disparity map. Finally, the obtained accurate disparity map is linearly superimposed with the frame image of the low-resolution video to be reconstructed before the input regularization layer using a global connection. The feature map obtained in this way not only contains the local and global feature information of the key frame image, but also contains the feature information at the depth level, thereby obtaining the target high-resolution video.

[0204] Simultaneously refer to Figure 10 In this embodiment, the disparity thinning regression module can also use a global residual connection to directly transmit the output to the subsequent linear addition. This allows the network to learn the residual between the input and output, focusing on the difference between the input and output images, rather than learning the entire image from scratch. This connection method helps prevent the network from losing key information during training, helps ensure the stability of training, and improves the quality of the reconstructed image. Furthermore, the amplification process of the sub-pixel convolution module can be represented by the following formula 14:

[0205] ISR = SUP(Fss + FD2) Formula 14;

[0206] F s2 For the frame image of the first non-local residual block as input, F D2 S represents the image residual output by the disparity refinement regression module. UP For the scaling operation of the subpixel convolution module, I SR For high-resolution video.

[0207] Secondly, considering both the speed and quality of the generated image, this embodiment does not use deconvolution to enlarge the image, but instead uses subpixel convolution in ESPCN, which has the advantage of being fast.

[0208] This embodiment can capture global information of the frame image through a nonlocal residual module, learn its internal structure and texture information, thereby ensuring that the high-resolution image recovered from the low-resolution frame image has more natural brightness, clearer image, and more realistic edge contours, further improving image quality. At the same time, through the disparity map generation module and the disparity map refinement regression module, the disparity map is linearly superimposed with the obtained initial high-resolution video to obtain the target high-resolution video, thereby realizing AR video super-resolution reconstruction.

[0209] refer to Figure 10 , Figure 11 This is a flowchart illustrating the third embodiment of the video super-resolution reconstruction method of this application.

[0210] To color grade the video and further improve its resolution, such as... Figure 11 As shown, in this embodiment, the above-mentioned preset reconstruction model further includes: a low-resolution video super-resolution module, which is connected to the non-local residual module;

[0211] After the step of performing detail adjustments on the initial high-resolution video using the non-local residual module, the method further includes:

[0212] Step S24: Perform color restoration on the adjusted initial high-resolution video using the low-resolution video super-resolution module, and then perform resolution enhancement on the restored initial high-resolution video to obtain the target high-resolution video.

[0213] Reference Figure 11 , Figure 12 This is a schematic diagram of the structure of the preset reconstruction model in the third embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 12 As shown, the low-resolution video super-resolution module in this embodiment can be connected after the sub-pixel convolution module in the non-local residual module. The low-resolution video super-resolution module can be a module of a deep network formed by continuously stacking a multi-channel attention mechanism module and MobileNet V3 convolutional blocks. In this embodiment, the low-resolution super-resolution module can improve and optimize the channel attention mechanism, which can color grade low-quality videos and improve resolution. The channel attention mechanism can make the network pay more attention to the channels with high information content in the input keyframe image, which can help the network better understand and utilize the feature information in the video frame, thereby improving the quality of super-resolution reconstruction.

[0214] This embodiment can construct a channel attention mechanism module using a channel attention mechanism, which in turn forms a color restoration module for color recovery and a resolution enhancement module for resolution enhancement (super-resolution). For the color restoration part, this module can focus on features such as color, brightness, and contrast; for the resolution enhancement part, this module can focus on details such as texture and edges. Therefore, the two modules differ only in parameters and depth to adapt to different task requirements.

[0215] To facilitate understanding the channel attention mechanism module, refer to... Figure 12 , Figure 12 This is a schematic diagram of the channel attention mechanism module in the third embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 12 As shown, the channel attention mechanism used in this embodiment may include a feature information extraction module and a linear addition layer;

[0216] The linear addition layer can be placed after the feature information extraction module. The feature information extraction module can extract features from the input frame image to obtain feature information. The linear addition layer can perform color restoration or resolution enhancement on the low-resolution video to be reconstructed based on the feature information to obtain the target high-resolution video.

[0217] For the feature information extraction module, a multi-scale channel attention module and a small residual module (i.e., ...) can be set up. Figure 13 Small and medium residual modules 1 to small residual module M) and large residual module (i.e. Figure 13 The medium-to-large residual modules 1 to large residual modules M, the multi-scale channel attention module, the small residual module, and the large residual module are set in parallel. That is, the inputs of the multi-scale channel attention module, the small residual module, and the large residual module are all connected to the sub-pixel convolution module, and the outputs are all connected to the linear additive layer.

[0218] In this embodiment, the multi-scale channel attention module can effectively distinguish between the feature maps extracted by the MobileVit attention mechanism and the MobileNet V3 convolutional network. It can differentiate and process different feature maps based on their importance and relevance, thereby extracting more accurate information useful for super-resolution reconstruction. Different feature maps contain different information, and some feature maps may be more important and require more attention. Therefore, when fitting nonlinear mappings, each feature map is processed in different ways to better utilize its information, learn richer feature map channel information, and effectively utilize the interdependencies between feature maps to provide prior guidance for network inference.

[0219] Reference Figure 13 , Figure X This is a schematic diagram of the multi-scale channel attention module in the third embodiment of the video super-resolution reconstruction method of this application, as shown below.Figure 13 As shown, this embodiment improves upon traditional channel attention by assuming that the features input to the multi-scale channel attention module are... Figure 13 The number of layers is C, and the size of each layer is W×H. They can be initially processed using two different sizes of MobileNet V3 convolutional layers (i.e., 3×3 and 5×5). Figure 13 The network performs convolution operations using 3×3 and 5×5 convolutions. These two types of convolution kernels can capture local details and broader spatial context information, respectively, thereby generating two new sets of features, U and V. The channel dimensions of U and V are the same as X, namely C×W×H. Subsequently, the multi-scale channel attention module merges these two sets of features U and V together by adding corresponding matrix elements. This merging strategy ensures the effective fusion of feature information at different scales, enabling the network to more comprehensively understand and process image content. The fusion process can be represented by the following formula 15:

[0220] Formula 15: M = U + V;

[0221] Where M represents the feature after adding the matrix elements.

[0222] The feature maps processed by the multi-scale channel attention module require further processing to extract deeper features. This is followed by global pooling (i.e., ...). Figure 13 Global pooling (GSP) is applied to the feature maps, compressing each feature map of size W×H into a single value (i.e., ...). Figure 12 (C×1). This step helps to aggregate global information from each feature map, providing a more representative feature representation for subsequent operations.

[0223] After global pooling, the resulting features are fed into two fully connected layers (i.e., Figure 12 The fully connected layers 1 and 2 perform mapping. The purpose of these two fully connected layers is to further transform the pooled features, ultimately mapping them to a 2C×1 dimension. This process can be represented by the following formula 16:

[0224] w = FC2(FC1(GP(M))) Formula 16;

[0225] Where GP is the global pooling operation, FC1 represents the first fully connected layer, FC2 represents the second fully connected layer, and w is the mapping result. After obtaining the feature w, a reshape operation is used to expand it from two dimensions to three dimensions. Then, it is passed through a preset activation function (such as the sigmoid function) to obtain an output with the same dimension as the transformed feature map. This output can be interpreted as the importance weight of each channel. This weight is a feature vector containing channel attention information, with a dimension of 2C×1×1. Next, the obtained feature vector is divided into two sub-vectors u and v corresponding to the two sets of features U and V obtained earlier. These sub-vectors are multiplied by U and V respectively and then added together, thus completing the entire process of multi-scale attention mechanism calibration of the feature map and obtaining the relevant feature information of the color channel (i.e., the red, green, and blue (RGB) channel), i.e., the color feature information, denoted as Y. The specific process can be expressed by the following formula 17:

[0226] Y = uU + vV (Formula 17)

[0227] Continue as Figure 12 As shown, a convolutional layer (i.e., ...) can be sequentially connected before the aforementioned multi-scale channel attention module. Figure 12 (MobileNet V3 convolutional block), and an activation function layer (i.e.) Figure 12 (ReLU)

[0228] and a convolutional layer (i.e.) Figure 12 In the MobileNet V3 convolutional block, all convolutional layers can be 1×1 convolutions. The first convolutional layer reduces the number of channels in the input feature map to 1 / r of the original number of channels, where r is the number of channels in the input feature map. This helps reduce computational cost and the number of parameters. Then, an activation function layer (such as ReLU activation function) activates the reduced feature map. A second convolutional layer restores the number of channels to the original dimension, generating attention coefficients with the same number of channels as the original feature map. The restored feature map is then passed to a multi-scale channel attention module for the above processing, obtaining a feature map with channel attention coefficients. Finally, an element-wise multiplication operation is performed with the original feature map. The multiplication process recalibrates the original feature map, adjusting the feature values ​​of each channel according to its importance. In this way, the model can focus more on the channel information that has a significant impact on super-resolution reconstruction, while suppressing the information of unimportant channels, thereby improving the performance and efficiency of video super-resolution reconstruction.

[0229] For small residual modules, hierarchical features of the input feature map can be extracted to obtain hierarchical feature information. For large residual modules, spatial features of the input feature map can be extracted to obtain spatial feature information. Finally, color adjustment or super-resolution tasks are completed based on color feature information, hierarchical feature information, and spatial feature information.

[0230] It should be emphasized that the aforementioned color feature information can be information about color channel features in the image, the aforementioned hierarchical feature information can be feature information between different layers of images, and the aforementioned spatial feature information can be feature information of key regions in the image.

[0231] In both small and large residual modules, a convolutional layer (i.e., ...) can be used first. Figure 12 The shallow feature extraction is performed using MobileNet V3 convolutional blocks, and this process can be represented by the following formula 18:

[0232] F s3 =C SF (I LR ) Formula 18;

[0233] Among them I LR C is the initial high-resolution video output by the subpixel convolution module. SF For convolution operations, F s3 This represents the shallow features obtained from the convolutional layer. These shallow features are then passed to the small residual module and the large residual module for deep feature extraction. This process can be represented by the following formula 19:

[0234] F D3 =R DF (F s3 ) Formula 19;

[0235] Where R DF For the extraction of the aforementioned deep features, F D3 The results are extracted.

[0236] It is important to emphasize that the small residual module and the large residual module can be multi-level residual structures. To balance the complexity of the network and the stability of training, the low-quality video super-resolution network adopts a design strategy of nesting small residual modules within large residual modules (i.e., Figure 12 (As shown). This nested structure not only deepens the network but also effectively prevents the gradient explosion problem during training. Simultaneously, by increasing the number of feature extraction units, the network can expand its receptive field, enhance its ability to capture image information, and further improve the accuracy of feature extraction.

[0237] It should also be emphasized that, if Figure 14The structure shown can be used for color correction tasks without needing a subpixel convolutional module after the linear summation layer. However, if used for super-resolution tasks, a subpixel convolutional module can be placed after the linear summation layer (i.e., ...). Figure 14 (Subpixel convolution), this subpixel convolution module may contain subpixel convolution layers. After the super-resolution reconstruction task is performed through linear summation layers, the obtained result can be upsampled and amplified through this subpixel convolution module to further improve the resolution. Finally, a MobileNet V3 convolution operation is performed to generate the final target high-resolution video. This process can be represented by the following formula 20:

[0238] I LR =C RE (S UP (F D3 )) Formula 20.

[0239] Where S UP For the upsampling operation of this subpixel convolution module, C RE For the convolution operation of this convolutional layer, I LR For high-resolution video.

[0240] Furthermore, to differentiate between color grading tasks and super-resolution tasks, this embodiment can set different loss functions for different tasks. For the channel attention mechanism module in the color grading task, the corresponding color grading stage loss function can be composed of two loss functions: mean absolute error and structural similarity error (SSIM), which can be expressed by the following formula 21:

[0241] L Color =αL1+βL SSIM Formula 21;

[0242] Where L Color L1 is the loss function for the color grading stage, and L2 is the L1 loss function. SSIM Let α be the structural similarity error loss function, and let β be the weight coefficients. These can be obtained through adaptive training. In color correction tasks, since structural similarity error can focus on the structural information of the image, it tends to retain more details such as edges and textures during the optimization process, making the reconstructed video more natural and realistic. Therefore, β will be smaller.

[0243] For the channel attention mechanism module in the super-resolution task, the corresponding super-resolution stage loss function can be constructed using the mean absolute error, and can be expressed by the following formula 22:

[0244]

[0245] Where L SR For the super-resolution stage loss function, Let be the frame image of the target super-resolution video after super-resolution, and 1 be the L1 loss function.

[0246] To further differentiate between color correction and super-resolution tasks, this embodiment can also incorporate a gating mechanism into the linear addition layer. Specifically, corresponding gates are set for the multi-scale channel attention module, the large residual module, and the small residual module, respectively, as shown in the reference. Figure 14 , Figure 14 This is a schematic diagram of the gating mechanism in the third embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 11 As shown, a channel gate can be set for the multi-scale channel attention module, a spatial gate can be set for the large residual module, and a hierarchical gate can be set for the small residual module. A target feature weight (i.e., ...) is set for each gate. Figure 11 The target feature weights for the channel gate (h), spatial gate (σ), and hierarchical gate (tanh) are obtained through adaptive training of the initial feature weights using training data. The channel gate controls the flow of information between channels by calculating the weight of each channel. This helps the model focus on important channel features while ignoring irrelevant channel information. The spatial gate controls the flow of spatial information by calculating the weight at each location, which helps the model focus on key regions in the image while suppressing the influence of background or noise regions. The hierarchical gate controls the interaction between information at different levels.

[0247] Before use, training data containing the target high-resolution video and the low-resolution video to be reconstructed can be obtained, and the initial feature weights set for each gate in the gating mechanism can be trained. For color grading tasks, color feature information may be more important, and the target feature weights obtained by the channel gate after self-learning may be higher. For super-resolution tasks, the target feature weights obtained by the hierarchical gate and spatial gate lock after self-learning may be higher.

[0248] Therefore, the aforementioned gating mechanism can learn the importance or weight of each channel and adjust the output of the feature map based on these weights. This helps improve the quality of super-resolution reconstruction, resulting in clearer and more detailed images. The gating mechanism can process all channels uniformly, meaning that regardless of the number of channels in the input feature map, the gating mechanism can use the same structure and parameters for computation. This uniform approach not only simplifies the model's complexity but also makes the model more flexible and scalable.

[0249] In summary, in this embodiment, the low-resolution video super-resolution module is sequentially connected to a color restoration module and a resolution enhancement module, and the color restoration module is also connected to the non-local residual module.

[0250] The steps of performing color restoration on the adjusted initial high-resolution video using a low-resolution video super-resolution module, and then performing resolution upscaling on the restored initial high-resolution video to obtain the target high-resolution video, include:

[0251] The color restoration module restores the color of the adjusted initial high-resolution video; the resolution enhancement module enhances the resolution of the restored initial high-resolution video to obtain the target high-resolution video.

[0252] It should be noted that the aforementioned color restoration module can be connected after the sub-pixel convolution module in the non-local residual module. Furthermore, the aforementioned color restoration module may include the aforementioned channel attention mechanism module. For ease of subsequent explanation, the channel attention mechanism module in the color restoration module will be referred to as the first channel attention mechanism module, and the first channel attention mechanism module may include the aforementioned gating mechanism and color grading stage loss function set for the color grading task.

[0253] The multi-scale channel attention module in the first channel attention mechanism module is denoted as the first multi-scale channel attention module, the small residual module is denoted as the first small residual module, the large residual module is denoted as the first large residual module, the convolutional layer and activation function layer before the first multi-scale channel attention module can be denoted as the third convolutional layer, the first activation function layer and the fourth convolutional layer, respectively, and the linear addition layer is denoted as the first linear addition layer.

[0254] In practical use, the input initial low-resolution video (i.e., the initial low-resolution video after being amplified by the sub-pixel convolution module) is down-channeled through the third convolutional layer, and the down-channeled initial low-resolution video is activated through the first activation function layer; the activated initial low-resolution video is restored through the fourth convolutional layer, and the color features of the restored initial low-resolution video are extracted through the first multi-scale channel attention module to obtain color feature information; the hierarchical features of the input initial low-resolution video are extracted through the first small residual module to obtain hierarchical feature information; the spatial features of the input initial low-resolution video are extracted through the first large residual module to obtain spatial feature information; and finally, the color is restored through the gating mechanism in the first linear addition layer based on the color feature information, hierarchical feature information, and spatial feature information according to the corresponding target feature weights to obtain the restored initial high-resolution video.

[0255] Similarly, the resolution enhancement module described above can be connected after the color restoration module. The resolution enhancement module can also include the aforementioned channel attention mechanism module, and its depth can be greater than that of the color restoration module (i.e.,...). Figures 12 to 14(7 layers in the middle). For ease of explanation, the channel attention mechanism module in the resolution enhancement module will be referred to as the second channel attention mechanism module. The second channel attention mechanism module may include the gating mechanism and the loss function set for the super-resolution task.

[0256] The multi-scale channel attention module in the second channel attention mechanism module is denoted as the second multi-scale channel attention module, the small residual module is denoted as the second small residual module, the large residual module is denoted as the second large residual module, the convolutional layer and activation function layer before the second multi-scale channel attention module can be denoted as the fifth convolutional layer, the second activation function layer and the sixth convolutional layer, respectively, and the linear addition layer is denoted as the second linear addition layer.

[0257] In practical application, the fifth convolutional layer performs channel downscaling on the restored initial high-resolution video, and the second activation function layer activates the downscaled initial low-resolution video. The sixth convolutional layer restores the channels of the activated initial low-resolution video, and the second multi-scale channel attention module extracts color features from the restored initial low-resolution video to obtain color feature information. The second small residual module extracts hierarchical features from the restored initial high-resolution video to obtain hierarchical feature information. The second large residual module extracts spatial features from the restored initial high-resolution video to obtain spatial feature information. Then, the gating mechanism in the second linear addition layer performs resolution upscaling based on color, hierarchical, and spatial feature information according to the corresponding target feature weights. Finally, a sub-pixel convolution is used for magnification, allowing for the processing of each magnified result (i.e.,...) Figure 15 By encoding high-quality keyframes (in medium to high quality) into the video, the aforementioned high-resolution video can be obtained.

[0258] It should be emphasized that both the first-channel attention mechanism module and the second-channel attention mechanism module mentioned above can be referenced. Figure 15 The relevant descriptions are not repeated in this embodiment.

[0259] This embodiment can perform color restoration and resolution enhancement through the aforementioned low-resolution video super-resolution module, thereby further improving video quality.

[0260] Reference Figure 15 , ​ This is a structural block diagram of the first embodiment of the video super-resolution reconstruction device of this application.

[0261] like ​ As shown, the video super-resolution reconstruction apparatus proposed in this application includes:

[0262] The video acquisition unit 151 is used to acquire the low-resolution video to be reconstructed.

[0263] The resolution reconstruction unit 152 is used to reconstruct the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video.

[0264] The preset reconstruction model includes at least a dense module with a first network and a second network. The first network is used to perform depthwise separable convolution operations on the low-resolution video to be reconstructed using an inverted residual method to obtain local features. The second network is used to extract global features from the low-resolution video to be reconstructed using a self-attention mechanism, and reconstruct the low-resolution video to be reconstructed based on the local features and the global features.

[0265] In this embodiment, after acquiring the low-resolution video to be reconstructed, the device first performs depthwise separable convolution operations on it using the inverted residual method in the first network of the preset reconstruction model to obtain local features. Then, the second network uses a self-attention mechanism to extract global features from the low-resolution video to be reconstructed. Based on the local and global features, the low-resolution video to be reconstructed is reconstructed to obtain the target high-resolution video. Compared with the existing CNN+Transformer approach, which requires a large number of parameters, this embodiment reduces the number of parameters used and improves inference speed, thereby enhancing the user experience, because the self-attention mechanism, depthwise separable convolution, and inverted residual technique require fewer parameters.

[0266] In one implementation, the first network includes a MobileNet V3 network, and the second network includes a MobileVit network.

[0267] As one implementation, the preset reconstruction model further includes: an image preprocessing module, which is connected to the first network;

[0268] The resolution reconstruction unit 152 is further configured to preprocess the low-resolution video to be reconstructed through the image preprocessing module, wherein the preprocessing includes at least one of keyframe extraction, bad frame removal, resolution adjustment, and data augmentation; to extract local features from the preprocessed low-resolution video to be reconstructed through the first network; to extract global features from the preprocessed low-resolution video to be reconstructed through the second network; and to reconstruct the low-resolution video to be reconstructed based on the local features and the global features.

[0269] Based on the first embodiment of the video super-resolution reconstruction apparatus described in this application, a second embodiment of the video super-resolution reconstruction apparatus of this application is proposed.

[0270] In this embodiment, the preset reconstruction model further includes a nonlocal residual module, which is connected to the second network;

[0271] The resolution reconstruction unit 152 is further configured to perform depthwise separable convolution operations on the low-resolution video to be reconstructed using the inverted residual method through the first network to obtain local features; extract global features from the low-resolution video to be reconstructed using the self-attention mechanism through the second network, and reconstruct the low-resolution video to be reconstructed based on the local features and the global features to obtain an initial high-resolution video; and perform detail adjustments on the initial high-resolution video through the non-local residual module to obtain a target high-resolution video.

[0272] In one implementation, the nonlocal residual module includes a nonlocal attention module and a hierarchical residual module, wherein the nonlocal attention module is connected to the second network and the hierarchical residual module;

[0273] The resolution reconstruction unit 152 is also used to perform global calibration on the initial high-resolution video through the non-local attention module; and to perform hierarchical fusion on the calibrated initial high-resolution video through the hierarchical residual module to obtain the target high-resolution video.

[0274] In one implementation, the dense module further includes a disparity map generation module, and the nonlocal residual module further includes a disparity map refinement and regression module. The disparity map generation module is connected to the second network and the nonlocal attention module, and the disparity map refinement and regression module is connected to the hierarchical residual module.

[0275] The resolution reconstruction unit 152 is also used to extract depth information from the low-resolution video to be reconstructed through the disparity map generation module to obtain a disparity map;

[0276] The resolution reconstruction unit 152 is also used to linearly add the initial high-resolution video based on the disparity map through the disparity map refinement regression module to obtain the target high-resolution video.

[0277] Based on the above embodiments of the video super-resolution reconstruction apparatus of this application, a third embodiment of the video super-resolution reconstruction apparatus of this application is proposed.

[0278] In this embodiment, the preset reconstruction model further includes a low-resolution video super-resolution module, which is connected to the nonlocal residual module.

[0279] The resolution reconstruction unit 152 is also used to perform color restoration on the adjusted initial high-resolution video through the low-resolution video super-resolution module, and to perform resolution enhancement on the restored initial high-resolution video to obtain the target high-resolution video.

[0280] As one implementation, the low-resolution video super-resolution module comprises a color restoration module and a resolution enhancement module connected in sequence, wherein the color restoration module is also connected to the non-local residual module;

[0281] The resolution reconstruction unit 152 is also used to perform color restoration on the adjusted initial high-resolution video through the color restoration module; and to perform resolution enhancement on the restored initial high-resolution video through the resolution enhancement module to obtain the target high-resolution video.

[0282] Other embodiments or specific implementations of the video super-resolution reconstruction apparatus of this application can be found in the above-described method embodiments, and will not be repeated here.

[0283] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0284] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0285] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0286] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for video super-resolution reconstruction, characterized in that, The method comprises: acquiring a low-resolution video to be reconstructed; reconstructing the low-resolution video to be reconstructed by a preset reconstruction model to obtain a target high-resolution video; wherein the preset reconstruction model at least comprises a dense module provided with a first network and a second network, the first network is configured to perform a depth separable convolution operation on the low-resolution video to be reconstructed in an inverted residual manner to obtain local features, and the second network is configured to perform global feature extraction on the low-resolution video to be reconstructed by using a self-attention mechanism and reconstruct the low-resolution video to be reconstructed according to the local features and the global features.

2. The method of claim 1, wherein, The preset reconstruction model further comprises a non-local residual module connected with the second network. The step of reconstructing the low-resolution video to be reconstructed by the preset reconstruction model to obtain the target high-resolution video comprises: performing a depth separable convolution operation on the low-resolution video to be reconstructed by the first network in an inverted residual manner to obtain local features; performing global feature extraction on the low-resolution video to be reconstructed by the second network by using a self-attention mechanism, and reconstructing the low-resolution video to be reconstructed according to the local features and the global features to obtain an initial high-resolution video; performing detail adjustment on the initial high-resolution video by the non-local residual module to obtain the target high-resolution video.

3. The method of claim 2, wherein, The non-local residual module comprises a non-local attention module and a hierarchical residual module, and the non-local attention module is connected with the second network and the hierarchical residual module. The step of performing detail adjustment on the initial high-resolution video by the non-local residual module to obtain the target high-resolution video comprises: performing global calibration on the initial high-resolution video by the non-local attention module; performing hierarchical fusion on the calibrated initial high-resolution video by the hierarchical residual module to obtain the target high-resolution video.

4. The method of claim 3, wherein, The dense module further comprises a disparity map generation module, the non-local residual module further comprises a disparity map refinement regression module, the disparity map generation module is connected with the second network and the non-local attention module, and the disparity map refinement regression module is connected with the hierarchical residual module. After the step of performing global feature extraction on the low-resolution video to be reconstructed by the second network by using a self-attention mechanism, the method further comprises: extracting depth information from the low-resolution video to be reconstructed by the disparity map generation module to obtain a disparity map; after the step of performing hierarchical fusion on the calibrated initial high-resolution video by the hierarchical residual module, the method further comprises: performing linear addition on the initial high-resolution video by the disparity map refinement regression module based on the disparity map to obtain the target high-resolution video.

5. The method of claim 2, wherein, The preset reconstruction model further comprises a low-resolution video super-resolution module connected with the non-local residual module. After the step of performing detail adjustment on the initial high-resolution video by the non-local residual module, the method further comprises: The low-resolution video super-resolution module is connected with the non-local residual module.

6. The method of claim 5, wherein, The low-resolution video super-resolution module is connected with the non-local residual module. The step of performing color restoration on the adjusted initial high-resolution video through the low-resolution video super-resolution module, and performing resolution enhancement on the restored initial high-resolution video to obtain a target high-resolution video, includes: The color restoration module is connected with the non-local residual module. The resolution enhancement module is connected with the non-local residual module.

7. The method of claim 1, wherein, The preset reconstruction model further includes an image preprocessing module connected with the first network. The step of reconstructing the low-resolution video to be reconstructed through the preset reconstruction model includes: The image preprocessing module is used to preprocess the low-resolution video to be reconstructed, and the preprocessing includes at least one of key frame extraction, bad frame elimination, resolution adjustment, and data enhancement. The first network is used to extract local features from the preprocessed low-resolution video to be reconstructed. The second network is used to extract global features from the preprocessed low-resolution video to be reconstructed, and reconstruct the low-resolution video to be reconstructed according to the local features and the global features.

8. The method of any one of claims 1 to 7, wherein, The first network includes a MobileNetV3 network, and the second network includes a MobileVit network.

9. An apparatus for video super-resolution reconstruction, the apparatus comprising: a video super-resolution reconstruction module configured to perform the method of any one of claims 1-8. The device includes: A video acquisition unit is configured to acquire a low-resolution video to be reconstructed. A resolution reconstruction unit is configured to reconstruct the low-resolution video to be reconstructed through a preset reconstruction model to obtain a target high-resolution video. The preset reconstruction model includes at least a dense module provided with a first network and a second network.

10. A wearable device, comprising: The wearable device includes a memory, a processor, and a video super-resolution reconstruction program stored on the memory and executable on the processor. The video super-resolution reconstruction program is executed by the processor to implement the steps of the video super-resolution reconstruction method according to any one of claims 1 to 8.

Images