Video super-resolution reconstruction method and device and wearable equipment

Abstract

The invention relates to the technical field of video processing, and discloses a video super-resolution reconstruction method and device and wearable equipment, and the method comprises the steps: obtaining a to-be-reconstructed low-resolution video; reconstructing the to-be-reconstructed low-resolution video through a preset reconstruction model to obtain a target high-resolution video; wherein the preset reconstruction model at least comprises a dense module and a non-local attention module, the dense module is used for performing feature extraction on the to-be-reconstructed low-resolution video and reconstructing the to-be-reconstructed low-resolution video according to the first extraction result to obtain an initial high-resolution video, and the non-local attention module is used for performing non-local attention on the initial high-resolution video; and the non-local attention module is used for determining the relevance of the initial high-resolution video by adopting a self-attention mechanism, and adjusting the initial high-resolution video based on the relevance to obtain a target high-resolution video. According to the method and the device, the relevance of different positions can be considered in the obtained target high-resolution video, so that the reconstruction effect is improved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video super-resolution reconstruction method, apparatus and wearable device. Background Technology

[0002] Currently, when performing super-resolution (SR) reconstruction on low-resolution (LR) videos to obtain high-resolution (HR) videos, the general approach is to extract features from the low-resolution video and then directly reconstruct based on the obtained features. However, since there are correlations between pixels at different locations, existing methods do not consider these correlations, resulting in poor reconstruction quality. Summary of the Invention

[0003] The main purpose of this application is to provide a video super-resolution reconstruction method, apparatus and wearable device, which aims to solve the technical problem that the correlation between pixels at different positions is not considered when performing super-resolution reconstruction, resulting in poor reconstruction quality.

[0004] To achieve the above objectives, this application provides a video super-resolution reconstruction method, the method comprising:

[0005] Acquire the low-resolution video to be reconstructed;

[0006] The low-resolution video to be reconstructed is reconstructed using a preset reconstruction model to obtain the target high-resolution video;

[0007] The preset reconstruction model includes at least a dense module and a non-local attention module. The dense module is used to extract features from the low-resolution video to be reconstructed and reconstruct the low-resolution video to be reconstructed based on the first extraction result to obtain an initial high-resolution video. The non-local attention module is used to determine the correlation of the initial high-resolution video using a self-attention mechanism and adjust the initial high-resolution video based on the correlation to obtain the target high-resolution video.

[0008] In one embodiment, the step of reconstructing the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video includes:

[0009] The dense module extracts features from the low-resolution video to be reconstructed, and reconstructs the low-resolution video to be reconstructed based on the first extraction result to obtain an initial high-resolution video.

[0010] The non-local attention module uses a self-attention mechanism to determine the correlation of the initial high-resolution video, and performs global calibration on the initial high-resolution video based on the correlation to obtain the target high-resolution video.

[0011] In one embodiment, the step of determining the correlation of the initial high-resolution video using a self-attention mechanism through the non-local attention module, and performing global calibration on the initial high-resolution video based on the correlation to obtain the target high-resolution video includes:

[0012] The initial high-resolution video is dimensionality reduced using the non-local attention module, and features are extracted from the dimensionality-reduced initial high-resolution video.

[0013] A self-attention mechanism is used to transform the extracted initial high-resolution video to obtain a self-similarity matrix;

[0014] The correlation of the initial high-resolution video is determined based on the self-similarity matrix, and the dimensionality-reduced initial high-resolution video is globally calibrated based on the correlation to obtain the target high-resolution video.

[0015] In one embodiment, the step of determining the correlation of the initial high-resolution video based on the self-similarity matrix includes:

[0016] A weight matrix is ​​obtained based on the dimensionality-reduced initial high-resolution video, and a nonlocal matrix is ​​obtained based on the weight matrix and the self-similar matrix.

[0017] The nonlocal matrix is ​​transformed, and the correlation of the initial high-resolution video is determined based on the transformation result.

[0018] In one embodiment, the preset reconstruction model further includes: a hierarchical residual module, which is connected to the nonlocal attention module;

[0019] The step of performing global calibration on the initial high-resolution video based on the correlation to obtain the target high-resolution video includes:

[0020] The initial high-resolution video is globally calibrated based on the correlation.

[0021] The hierarchical residual module is used to perform hierarchical fusion on the calibrated initial high-resolution video to obtain the target high-resolution video.

[0022] In one embodiment, the hierarchical residual module includes: a residual module comprising a first network that performs depthwise separable convolution operations using inverted residuals to extract local features and a second network that performs a self-attention mechanism to extract global features, wherein the non-local attention module is connected to the first network and the second network respectively.

[0023] The step of performing hierarchical fusion on the calibrated initial high-resolution video through the hierarchical residual module to obtain the target high-resolution video includes:

[0024] The calibrated initial high-resolution video is grouped using the hierarchical residual module;

[0025] The first network and the second network are used to extract features from the grouping results, and the corresponding extraction results are used for hierarchical fusion to obtain the target high-resolution video.

[0026] In one embodiment, the first network includes a MobileNet V3 network, and the second network includes a MobileVit network.

[0027] In one embodiment, the preset reconstruction model further includes a subpixel convolution module, which is connected to the hierarchical residual module;

[0028] The step of performing hierarchical fusion on the calibrated initial high-resolution video using the hierarchical residual module to obtain the target high-resolution video includes:

[0029] The hierarchical residual module is used to perform hierarchical fusion on the calibrated initial high-resolution video.

[0030] The fusion result is amplified by the subpixel convolution module to obtain the target high-resolution video.

[0031] Furthermore, to achieve the above objectives, this application also proposes a video super-resolution reconstruction apparatus, the apparatus comprising:

[0032] The video acquisition unit is used to acquire the low-resolution video to be reconstructed.

[0033] The resolution reconstruction unit is used to reconstruct the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video.

[0034] The preset reconstruction model includes at least a dense module and a non-local attention module. The dense module is used to extract features from the low-resolution video to be reconstructed and reconstruct the low-resolution video to be reconstructed based on the first extraction result to obtain an initial high-resolution video. The non-local attention module is used to determine the correlation of the initial high-resolution video using a self-attention mechanism and adjust the initial high-resolution video based on the correlation to obtain the target high-resolution video.

[0035] In addition, to achieve the above objectives, this application also proposes a wearable device, which includes: a memory, a processor, and a video super-resolution reconstruction program stored in the memory and executable on the processor. When the video super-resolution reconstruction program is executed by the processor, it implements the steps of the video super-resolution reconstruction method described above.

[0036] This application provides a video super-resolution reconstruction method, apparatus, and wearable device. The method includes: acquiring a low-resolution video to be reconstructed; reconstructing the low-resolution video using a preset reconstruction model to obtain a target high-resolution video; wherein the preset reconstruction model includes at least a dense module and a non-local attention module, the dense module being used to extract features from the low-resolution video to be reconstructed and reconstruct the low-resolution video based on a first extraction result to obtain an initial high-resolution video, and the non-local attention module being used to determine the correlation of the initial high-resolution video using a self-attention mechanism and adjust the initial high-resolution video based on the correlation to obtain the target high-resolution video. After acquiring the low-resolution video to be reconstructed, this application first extracts features using a dense module and reconstructs the video based on the first extraction result to obtain an initial high-resolution video. Then, the non-local attention module uses a self-attention mechanism to determine the correlation of the initial high-resolution video and adjusts it based on the correlation, thereby ensuring that the obtained target high-resolution video takes into account the correlation of different locations, thus improving the reconstruction effect. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application;

[0040] Figure 2 This is a flowchart illustrating the first embodiment of the video super-resolution reconstruction method of this application;

[0041] Figure 3 This is a schematic diagram of the structure of the preset reconstruction model in the first embodiment of the video super-resolution reconstruction method of this application;

[0042] Figure 4 This is a schematic diagram of the structure of the dense module and the network module in the first embodiment of the video super-resolution reconstruction method of this application;

[0043] Figure 5 This is a flowchart illustrating the second embodiment of the video super-resolution reconstruction method of this application;

[0044] Figure 6 This is a schematic diagram of the structure of the preset reconstruction model in the second embodiment of the video super-resolution reconstruction method of this application;

[0045] Figure 7 This is a schematic diagram of the structure of the nonlocal residual module in the second embodiment of the video super-resolution reconstruction method of this application;

[0046] Figure 8 This is a schematic diagram of the structure of the nonlocal attention module in the second embodiment of the video super-resolution reconstruction method of this application;

[0047] Figure 9 This is a schematic diagram of the subpixel convolution module in the second embodiment of the video super-resolution reconstruction method of this application;

[0048] Figure 10 This is a flowchart illustrating the third embodiment of the video super-resolution reconstruction method of this application;

[0049] Figure 11 This is a schematic diagram of the structure of the preset reconstruction model in the third embodiment of the video super-resolution reconstruction method of this application;

[0050] Figure 12 This is a schematic diagram of the channel attention mechanism module in the third embodiment of the video super-resolution reconstruction method of this application;

[0051] Figure 13 This is a schematic diagram of the structure of the multi-scale channel attention module in the third embodiment of the video super-resolution reconstruction method of this application;

[0052] Figure 14 This is a schematic diagram of the gating mechanism in the third embodiment of the video super-resolution reconstruction method of this application;

[0053] Figure 15 This is a structural block diagram of the first embodiment of the video super-resolution reconstruction device of this application.

[0054] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0056] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application.

[0057] like Figure 1 As shown, the hardware operating environment involved in this application embodiment is a wearable device, which may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize communication between these components. The user interface 1003 may include a display screen, and optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. In this application, the wired interface of the user interface 1003 may be a USB interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0058] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0059] like Figure 1 As shown, the memory 1005, which is identified as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a video super-resolution reconstruction program.

[0060] exist Figure 1In the device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the user equipment; the device calls the video super-resolution reconstruction program stored in the memory 1005 through the processor 1001 and executes the video super-resolution reconstruction method provided in the embodiments of this application.

[0061] It should be noted that currently, when performing super-resolution (SR) reconstruction on low-resolution (LR) videos to obtain high-resolution (HR) videos, the reconstruction is generally performed directly based on the features obtained after feature extraction from the low-resolution video. However, since there are correlations between pixels at different locations, the existing methods do not take this correlation into account, resulting in poor reconstruction quality.

[0062] Therefore, in order to solve the above-mentioned defects, this embodiment provides a video super-resolution reconstruction method. After obtaining the low-resolution video to be reconstructed, feature extraction can be performed first through a dense module, and reconstruction can be performed based on the first extraction result to obtain an initial high-resolution video. Then, the correlation of the initial high-resolution video is determined by a self-attention mechanism through a non-local attention module, and adjustments are made based on the correlation, so that the obtained target high-resolution video takes into account the correlation of different positions, thereby improving the reconstruction effect.

[0063] For ease of understanding, the following is combined with Figures 2 to 15 The video super-resolution reconstruction method provided in the embodiments of this application will be described in detail.

[0064] This application provides a video super-resolution reconstruction method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the video super-resolution reconstruction method of this application.

[0065] In this embodiment, the video super-resolution reconstruction method includes the following steps:

[0066] Step S10: Obtain the low-resolution video to be reconstructed.

[0067] It is understood that the method of this embodiment can be applied to scenarios where wearable devices perform super-resolution reconstruction, such as AR glasses, or other devices. This embodiment does not limit this application. The executing entity of this embodiment can be a super-resolution reconstruction device with functions of super-resolution reconstruction, data processing, network communication, and program execution, such as AR glasses, or other devices capable of performing the same or similar functions. The following describes this embodiment and the following embodiments in detail using the aforementioned AR glasses (hereinafter referred to as the device).

[0068] It should be understood that the aforementioned low-resolution video to be reconstructed can be the video that the aforementioned device needs to display. Since AR glasses can generally be set to a display resolution at the factory, such as 1080P, and will display at that resolution in subsequent use, the resolution of the aforementioned low-resolution video to be reconstructed can be the set resolution. However, to improve the visual effect, the low-resolution video to be reconstructed can be reconstructed to increase its resolution, for example, to 2K.

[0069] Step S20: Reconstruct the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video.

[0070] It is important to emphasize that the aforementioned preset reconstruction mode can be a model used to upscale low-resolution videos to be reconstructed. Traditionally, this can be achieved using a hybrid model of Convolutional Neural Networks (CNNs) and Transformers. Specifically, CNNs extract local features from the low-resolution video, while Transformers extract global features. The local and global features are then fused and decoded to obtain a high-resolution video, achieving super-resolution reconstruction. However, this method does not consider the correlation between pixels at different locations, resulting in low reconstruction quality. Therefore, the preset reconstruction model in this embodiment includes at least a dense module and a non-local attention module. The dense module extracts features from the low-resolution video to be reconstructed and reconstructs it based on the first extraction result to obtain an initial high-resolution video. The non-local attention module uses a self-attention mechanism to determine the correlation of the initial high-resolution video and adjusts it based on the correlation to obtain the target high-resolution video.

[0071] Therefore, in this embodiment, after obtaining the low-resolution video to be reconstructed, feature extraction can be performed first through a dense module, and reconstruction can be performed based on the first extraction result to obtain an initial high-resolution video. Then, the correlation of the initial high-resolution video is determined by a self-attention mechanism through a non-local attention module, and adjustments are made based on this correlation, so that the obtained target high-resolution video takes into account the correlation of different positions, thereby improving the reconstruction effect.

[0072] The aforementioned dense module can be a module for feature extraction of the low-resolution video to be reconstructed. The aforementioned non-local attention module can be equipped with a second network, which can be a network with the aforementioned self-attention mechanism. The self-attention mechanism can capture dependencies and thus extract the correlation between pixels.

[0073] However, before proceeding, it should be noted in this embodiment that when the hybrid model of the above-mentioned CNN and Transformer model is applied to augmented reality (AR) glasses, the AR glasses have limited computing resources, and the above-mentioned CNN+Transformer method requires a large number of parameters. A large number of parameters means that the model needs to occupy more storage space, and more memory is needed to store and access these parameters during the inference process. Therefore, the inference speed of AR glasses is slow and the user experience is poor.

[0074] Therefore, in this embodiment, the above-mentioned preset reconstruction model may further include at least: a dense module with a first network and a second network, wherein the first network is used to perform depthwise separable convolution operation on the low-resolution video to be reconstructed using an inverted residual method to obtain local features, and the second network is used to extract global features from the low-resolution video to be reconstructed using a self-attention mechanism, and reconstruct the low-resolution video to be reconstructed based on the local features and the global features (i.e., the first extraction result mentioned above).

[0075] The aforementioned first network can be a network with depthwise separable convolution and inverted residual modules. In conventional convolution operations, each convolution kernel can simultaneously process multiple input channels of the input tensor and mix them into a single output channel. However, the depthwise separable convolution in this embodiment decomposes this process into two steps: depthwise convolution and pointwise convolution. Depthwise convolution performs a convolution operation on each input channel using its own convolution kernel, without cross-channel mixing. Pointwise convolution mixes the output channels of the depthwise convolution using a 1x1 convolution kernel to generate the final output channel. This reduces the number of convolution kernels and the computational cost of convolution operations.

[0076] Traditional ResNet residual modules typically contain two convolutional layers, where the output dimension of the second convolutional layer is the same as the input dimension, allowing the input to "jump" directly to the output, forming a residual connection. In this embodiment, however, the pointwise convolutional layer in the inverted residual module is placed first, followed by the depthwise convolutional layer. That is, the number of channels is adjusted first, and then this number is maintained in the depthwise convolution, thereby further reducing the number of parameters and computational cost.

[0077] Therefore, the first network in this embodiment may include depthwise separable convolution and inverted residual modules. In practical use, after the device acquires the low-resolution video to be reconstructed, it can be input into the preset reconstruction model. The first network can extract local features from the low-resolution video to be reconstructed using depthwise separable convolution and inverted residual techniques, thereby reducing the number of parameters required. The aforementioned local features can be features within a small range of the low-resolution video to be reconstructed, which can be used to reflect details such as edges and textures. This embodiment does not impose any specific limitations on this.

[0078] The second network described above can be a network with a self-attention mechanism. This self-attention mechanism can capture the relationship between any two elements and extract information about each location in the low-resolution video to be reconstructed. Furthermore, this embodiment can learn complex feature information at each location through a multi-attention mechanism to obtain global features. These global features can be overall information, such as the overall layout, relationships between objects, and context; this embodiment does not impose any limitations on this. Since the parameters in the self-attention mechanism can be shared, the number of parameters can be further reduced. Therefore, in practical use, after the first network extracts local features, the second network can utilize the self-attention mechanism to extract global features from the low-resolution video to be reconstructed, thus obtaining global features.

[0079] After obtaining the aforementioned local and global features, the dense module can then use the learned local and global features to perform super-resolution reconstruction of the low-resolution video to be reconstructed, thereby obtaining an initial high-resolution video.

[0080] Furthermore, to construct the first and second networks in the aforementioned dense module, in this embodiment, the first network may include a MobileNet V3 network, and the second network may include a MobileVit network. That is, in this embodiment, the dense module can be constructed using a dense block network architecture, employing MobileVit's attention mechanism, a MobileNet V3 convolutional network combined with a residual network.

[0081] This dense module, through the aforementioned structure, fully leverages the advantages of attention mechanisms to obtain global information from keyframe images. It utilizes the self-attention mechanism of the MobileVit network to acquire location information for each frame in the low-resolution video to be reconstructed, and employs a multi-attention mechanism to learn complex feature information at each location in the image. Simultaneously, it can identify dependencies between different regions, thereby performing global feature calibration. It can learn the temporal continuity and spatial correlation in the video, completing spatiotemporal consistency calibration and ensuring smooth transitions and natural connections between frames.

[0082] This embodiment utilizes the MobileNet V3 network in the dense module to extract local features using depthwise separable convolutions and inverted residual modules, and then extracts global features using the MobileVit attention mechanism. Finally, it reconstructs the target high-resolution video using the obtained local and global features. Compared to existing methods employing CNN+Transformer, which require a large number of parameters, this embodiment reduces the number of parameters needed due to the smaller parameter count required by self-attention mechanisms, depthwise separable convolutions, and inverted residuals, thus improving inference speed and enhancing the user experience.

[0083] Meanwhile, this embodiment uses fewer parameters, thus reducing the computational load and consequently lowering the power consumption of the AR glasses. Furthermore, because this embodiment performs super-resolution reconstruction, compared to AR glasses without super-resolution reconstruction, it solves the video quality degradation problem caused by user movement while wearing the glasses, further improving the user experience.

[0084] Secondly, since this embodiment can achieve super-resolution reconstruction within AR glasses, compared to existing methods that rely on improving hardware performance (e.g., using high-definition cameras and high-performance light sensors), this embodiment can reduce the cost of hardware.

[0085] Furthermore, in order to improve the quality of the low-resolution video to be reconstructed, in this embodiment, the above-mentioned preset reconstruction model further includes: an image preprocessing module, which is connected to the first network;

[0086] The step of reconstructing the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video includes:

[0087] The image preprocessing module preprocesses the low-resolution video to be reconstructed, including at least one of keyframe extraction, bad frame removal, resolution adjustment, and data augmentation. The first network extracts local features from the preprocessed low-resolution video. The second network extracts global features from the preprocessed low-resolution video, and the video is reconstructed based on the local and global features to obtain an initial high-resolution video. The non-local attention module uses a self-attention mechanism to determine the correlation of the initial high-resolution video, and adjusts the initial high-resolution video based on the correlation to obtain a target high-resolution video.

[0088] It should be noted that, for ease of understanding, refer to Figure 3 To explain, Figure 3 This is a schematic diagram of the structure of the preset reconstruction model in the first embodiment of the video super-resolution reconstruction method of this application. Figure 3 As shown, in this embodiment, after obtaining the low-resolution video to be reconstructed (i.e., Figure 3 After processing low- to medium-quality videos, they can be input into the image preprocessing module of the preset reconstruction model.

[0089] In this image preprocessing module, keyframes can be extracted from the low-resolution video to be reconstructed at preset intervals (i.e., ...). Figure 3 (Frame extraction). For example, if the preset interval is 3 frames, then images at intervals of 3 frames in the low-resolution video to be reconstructed can be used as keyframe images (i.e., ... Figure 3 The keyframes in the low-resolution video to be reconstructed are extracted. Alternatively, keyframe extraction can be omitted, and all frames in the low-resolution video to be reconstructed can be used as keyframes for further processing. Whether to extract keyframes and the preset interval frame number when extracting them can be set according to the actual situation, and this embodiment does not impose any restrictions on this.

[0090] After extracting the keyframes, some images may be blurry or of low quality. Therefore, the image preprocessing module described above can remove bad frames from the extracted low-resolution video to be reconstructed (i.e., ...). Figure 3 Remove bad frames from the video (i.e., remove frames that do not meet the requirements) to obtain the low-resolution video to be reconstructed (i.e., the video itself). Figure 3 (The final keyframe in the middle);

[0091] After removing bad frames, in order to unify all the obtained frame images, the aforementioned image preprocessing module can also adjust the resolution of each frame image in the low-resolution video to be reconstructed after removal, adjusting it to the consistent resolution required subsequently (e.g., 1080P). Figure 3 (Medium resolution adjustment);

[0092] After adjusting the resolution, data augmentation (i.e., data enhancement) can be performed on the adjusted low-resolution video to be reconstructed to enrich the data. Figure 3 Data augmentation can be performed in various ways, including but not limited to spatial transformation, color transformation, cropping and flipping, adding Gaussian noise and Gaussian blur, etc. This embodiment does not limit these methods.

[0093] After data augmentation, the augmented low-resolution video to be reconstructed can be transmitted to the dense module (i.e., Figure 3 The MobileNet V3 network (comprising the MobileVit attention mechanism and MobileNet V3 convolutional blocks in the dense module) performs local feature extraction, which is then transmitted to the MobileVit network for global feature extraction. Based on the local and global features, the low-resolution video to be reconstructed is then reconstructed. However, to improve inference speed before input, in this embodiment, a segmentation module can be set after the image preprocessing module and before the dense module. Figure 3 (not shown in the image) and a first attention mechanism layer (i.e. Figure 3 The attention mechanism convolutional layer before the dense module (hereinafter, all attention mechanism layers are attention mechanism convolutional layers) means that the preset reconstruction model also includes: a segmentation module and a first attention mechanism layer, the first attention mechanism layer contains the second network, and the first attention mechanism layer is connected to the segmentation module and the dense module respectively.

[0094] After obtaining the preprocessed low-resolution video to be reconstructed, the segmentation module can first adjust the low-resolution video to make each frame image of it a preset size. The specific preset size is not limited in this embodiment. Then, the adjusted frame images are divided into several image blocks. The size of the image block can be 3×3, or other sizes. This embodiment can refer to the Vision Transformer network architecture and uses 3×3 for illustration.

[0095] After obtaining several image blocks, they can be sorted according to their position in the adjusted frame image to flatten the two-dimensional low-resolution image blocks into a one-dimensional sequence. Each image block is treated as a separate token, and an initial feature vector, denoted as I, is obtained from the pixel values ​​of each flattened image block through a linear transformation. LR Then, position embeddings are added to each flattened image patch (i.e., each token) to provide the position information of each image patch in the original image (i.e., the frame image in the low-resolution video to be reconstructed), and the final target feature vector containing position information is obtained, denoted as I. z This process can be obtained through the following first formula (i.e., Formula 1):

[0096] I z =WI LR Formula 1;

[0097] Among them I z Then it is the vector containing location information mentioned above, I LR Then, the eigenvectors are as described above, and W is the learnable weight matrix.

[0098] After obtaining the target feature vector I containing the location information mentioned above. z Then, the target feature vector can be transmitted to the first attention mechanism layer, where the MobileVit network processes the target feature vector I. z Perform self-attention calculation to complete feature extraction, obtain a second extraction result, and further improve the subsequent effect.

[0099] In this first attention mechanism layer, the query (Q), key (K), and value (V) of each token can be obtained by multiplying them with the corresponding learning weight matrix. Specifically, this can be obtained through the following second formula (i.e., Formula 2):

[0100] Q = I z W Q K = I z W K V = I z W V Formula 2;

[0101] Where Q represents the query, W... Q To query the corresponding learning weight matrix, K is the key, W K Let V be the learning weight matrix corresponding to the key, and W be the value. V The value is the learning weight matrix corresponding to the value.

[0102] After obtaining the query (Q), key (K), and value (V), an attention score can be calculated to measure the similarity between the query Q and each key K. Then, a softmax function is used for normalization, followed by a weighted summation to obtain the feature output by the first attention mechanism layer (i.e., the second extraction result mentioned above), denoted as F. s Specifically, this can be achieved through the following third formula (i.e., formula 3) and fourth formula (i.e., formula ...).

[0103] Equation 4) yields:

[0104]

[0105] F s =S w Formula 4;

[0106] Where score is the normalized result, KT d is the transpose of the key, and I is the transpose of the key. z The dimension, S w For the attention weight (i.e., score) corresponding to value V, F s The features output by the convolutional layer of this attention mechanism (i.e., the weighted summation result, i.e., the second extraction result mentioned above).

[0107] The second extraction result can then be input into the dense module for feature extraction. To facilitate dense connections, it is combined with... Figure 3 , Figure 3 The medium-density module may include at least one set of sequentially connected network modules and a first feature cascade layer (i.e., Figure 3 The network module, the first feature cascade layer, and the second attention mechanism layer are connected in a dense connection manner, that is, each first feature cascade layer is not only connected to the network module in its group, but also connected to the input of the network modules of all previous groups.

[0108] The network module can include a convolutional layer and an attention mechanism layer. The convolutional layer can use a first network, namely the MobileNet V3 convolutional network mentioned above, and the attention mechanism layer can use a second network, namely the MobileVit attention mechanism mentioned above. Therefore, local information of the input frame image can be extracted through MobileNet V3, and information of each pixel and complex feature information of each pixel can be extracted through MobileVit. To improve the performance, the network module in this embodiment can also include multiple convolutional layers and multiple attention mechanism layers.

[0109] Reference Figure 4 , Figure 4 This is a schematic diagram of the structure of the dense module and the network module in the first embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 4 As shown, in this embodiment, the dense module can be described using three sequentially connected network modules, a first feature cascade layer, and a second attention mechanism layer.

[0110] The network module may include the aforementioned first network. After obtaining the second extraction result through the first attention mechanism layer, the first network module can perform feature extraction on the second extraction result, obtain local features, and then transmit them together with the second extraction result to the first feature concatenation layer. Then, feature concatenation is performed on the global features and local features in the extracted feature map (i.e.,...). Figure 3The first feature concatenation layer (using cascaded features) yields a first concatenation result, which can be concatenated along the channel dimension to enhance feature richness. This first concatenation result is then processed by a second attention mechanism layer using a MobileVit network for feature extraction, yielding global features. Since the dense modules in this implementation employ dense connections, during feature concatenation in the subsequent first feature concatenation layer, the local features output by the current network module, the global features output by the previous second attention mechanism layer, and the second extraction result from the first network module can be concatenated together to further extract and refine features, obtaining the aforementioned first extraction result. Finally, video super-resolution reconstruction can be performed based on the extracted local and global features (i.e., the aforementioned first extraction result) to obtain an initial high-resolution video.

[0111] The specific process described above can be obtained through the following fifth formula (i.e., formula 5):

[0112] F D =AC n (Cat(B1, B2, ..., B) n )) Formula 5;

[0113] Among them B n Here, AC represents the local features extracted by the nth network module, and Cat represents the spatial feature concatenation operation (i.e., the feature concatenation mentioned above). n For the feature extraction operation of the second attention mechanism layer mentioned above, F D These are the global features mentioned above. This operation, which is easy to implement in deep learning, can complement the extracted deep features and combine them to improve the generalization performance of the model.

[0114] In order to obtain the above local feature B n , continue as Figure 4 As shown, in this embodiment, the network module includes: an extraction module, a second feature cascade layer, and a first convolutional layer. The first convolutional layer carries the first network. The second feature cascade layer is connected to the extraction module and the first convolutional layer, respectively. The first convolutional layer is connected to the first feature cascade layer.

[0115] In the process of obtaining local features by using the network module to extract features from the second extraction result or the global features output by the previous second attention mechanism layer through the first network, the extraction module can first extract features from the second extraction result or the global features output by the previous second attention mechanism layer to obtain the third extraction result. Then, the third extraction result is concatenated through the second feature cascade layer to obtain the second concatenated result. Finally, the first convolutional layer uses the first network to extract features from the second concatenated result to obtain local features.

[0116] To further improve reasoning speed, such as Figure 4 As shown, in this embodiment, the extraction module includes at least two extraction units, each of which is connected sequentially and then connected to the second feature cascade layer in a skip connection manner;

[0117] It should be noted that in this embodiment, each extraction unit is connected to the second feature cascade layer using a jump connection method, such as... Figure 4 As shown, this embodiment uses two extraction units for illustration, referred to as the first extraction unit and the second extraction unit respectively. The input of the first extraction unit is connected not only to the second attention mechanism layer of the previous group, but also to the second feature cascade layer in the network module. When it is the first network module, the first extraction unit can also be connected to the first attention mechanism layer. The output of the first extraction unit can be connected to the input of the second extraction unit and the second feature cascade layer. The output of the second feature extraction unit can also be connected to the second feature cascade layer.

[0118] In practical use, the first extraction unit can process the second extraction result F. s Alternatively, feature extraction operations can be performed on the global features output by the previous second attention mechanism layer to obtain the third extraction result corresponding to the first extraction unit, and this third extraction result is transmitted to the second extraction unit and the second feature cascade layer. The second extraction unit can then perform feature extraction operations on the third extraction result to obtain the third extraction result corresponding to the second extraction unit, and transmit it to the second feature cascade layer. The second feature cascade layer then concatenates the third extraction result output by the first extraction unit, the third extraction result output by the second extraction unit, and the second extraction result or the global features output by the previous second attention mechanism layer to obtain the second concatenated result.

[0119] If we denote the local features output by the network module as B n Then, for the local feature B output by the above network module n The feature extraction process can be obtained through the following sixth formula (i.e., formula 6):

[0120]

[0121] The above For the 3rd MobileNet V3 convolutional layer in the nth network module (i.e. Figure 4 The first convolutional layer), Cat is the spatial feature concatenation operation (i.e., the second feature cascade layer), X n The input to the nth network module (i.e., the second extraction result or the global features output by the previous second attention mechanism layer) is used. Since this embodiment has two extraction units and one first convolutional layer, there are three feature extraction processes. For X nThe second extraction result output after the first feature extraction (i.e., the first extraction unit) is as follows: The second extraction result is output after the second feature extraction (i.e., the second extraction unit).

[0122] To further improve speed, continue to refer to Figure 4 In this embodiment, the extraction unit may include: a third attention mechanism layer with a second network and a second convolutional layer with a first network, wherein the third attention mechanism layer and the second convolutional layer are connected in sequence, and the second convolutional layer is also connected to the second feature cascade layer.

[0123] It should be noted that when the third attention mechanism layer is the first extraction unit of the first network module, feature extraction can be performed on the second extraction result output by the first attention mechanism layer. When the third attention mechanism layer is not the first extraction unit in the first network module, feature extraction can be performed on the global features obtained by the previous second attention mechanism layer. When the third attention mechanism layer is not the first extraction unit, feature extraction can be performed on the third extraction result output by the previous extraction unit. Furthermore, in this embodiment, the feature extraction result of the third attention mechanism layer is referred to as the fourth extraction result.

[0124] After obtaining the fourth extraction result, the first network is used to extract features from the third extraction result through the second convolutional layer to obtain the third extraction result of this extraction unit. Specifically, this can be expressed by the following seventh formula (i.e., formula 7) and eighth formula (i.e., formula 8):

[0125]

[0126] Among the above For the third attention mechanism layer within the first extraction unit of the nth network module, the above For the second convolutional layer within the first extraction unit of the nth network module, the above For the third attention mechanism layer within the second extraction unit of the nth network module, the above This is the second convolutional layer within the second extraction unit of the nth network module. F can be obtained using the first to eighth formulas described above. D .

[0127] Secondly, continue to refer to Figure 3 After the dense module, upsampling and depth sampling can be added. Upsampling can include a deconvolution layer, which can also be composed of MobileNet V3 convolutional layers. Upsampling can further extract and refine the enlarged features. Depth sampling can be used to restore the feature map to the desired size. These two operations can enlarge the residual image before image reconstruction and linear feature addition (i.e.,...) Figure 3The linear features are added together, and then the video is transmitted to the non-local attention module. The non-local attention module extracts the correlations and adjusts the initial high-resolution video obtained by adding the features based on the correlations, thereby obtaining the target high-resolution video.

[0128] When adding linear features, bicubic interpolation can be used, which can be expressed by the following ninth formula (i.e., formula 9):

[0129] I SR =Bic(I SR )+OC 0 (DeC(F D )) Formula 9;

[0130] Among them I SR For high-resolution video, Bic is bicubic interpolation, and OC is... 0 For deep sampling operations, DeC is the aforementioned deconvolution layer.

[0131] It needs to be emphasized that, such as Figure 4 As shown, in this embodiment, the output of each second convolutional layer in the network module is directly connected to the subsequent second feature cascade layer. In the dense module, the input of each network module is directly connected to all subsequent first feature cascade layers. This means that the input of each first feature cascade layer is the union of the outputs of all previous layers. This design allows the network to reuse the features of previous layers, instead of simply passing the features to the next layer as in traditional networks. This connection method can be called dense connection, which further reduces the amount of computation.

[0132] It should also be emphasized that when the preset reconstruction model is in the learning phase, its input includes not only the low-resolution video to be reconstructed, but also its corresponding high-resolution video. This video undergoes preprocessing operations by the aforementioned image preprocessing module before being transmitted to the dense module for feature extraction and learning. This preprocessing process corresponds one-to-one with the preprocessing process for the low-resolution video described above, and will not be elaborated upon in this embodiment. When adjusting the resolution, the high-resolution video can be adjusted to 2K, but other resolutions are also possible; this embodiment does not impose any limitations on this.

[0133] During the learning phase, the dense module learns image detail information to perform detailed super-resolution reconstruction of low-resolution keyframe images. Furthermore, this dense module can better leverage the long-term dependency modeling capabilities of the attention mechanism, helping it better capture dependencies between different locations in the image and learn more inter-block information. This avoids gradient propagation problems, better models long-term dependencies, and effectively improves the network's reconstruction performance by learning more inter-block information. In summary, this dense block network learns the mapping relationship and feature map information from low-resolution keyframes to high-resolution keyframes.

[0134] Furthermore, continue as Figure 4 As shown, in this embodiment, skip connections are used between each extraction unit and the second feature cascade layer. Since edge contour information and high-frequency texture information are particularly important during image restoration, treating the features extracted by the last feature extraction unit and the inter-layer features transmitted by the skip connections equally might lead the model to focus only on low-frequency information (such as color and lighting) while ignoring high-frequency information, resulting in suboptimal high-resolution video restoration. Therefore, in this embodiment, the second feature cascade layer may include a target loss function. This target loss function is constructed by setting a first target weight for the first loss function of the extraction unit and a second target weight for the second loss function of the skip connections. The first and second target weights are determined through adaptive training.

[0135] It should be noted that the first loss function of the extraction unit mentioned above can be the loss function used by the extraction unit during training. For ease of explanation later, since the extraction unit may include a third attention mechanism layer and a second convolutional layer, the first loss function corresponding to the extraction unit can be denoted as L. MobileNetV3 +L MobileVit L MobileNetV3 L is the loss function of the third attention mechanism layer. MobileVit This is the loss function for the second convolutional layer.

[0136] The second loss function for the aforementioned skip connections can be the loss function corresponding to the skip connections between each extraction unit and the second feature cascade layer, and can be denoted as L. Skip .

[0137] This embodiment can incorporate the aforementioned first target weight and second target weight to treat the image features extracted by the convolutional layer and the attention mechanism layer equally, while enriching the inter-layer information extracted by the residual network and the convolutional layer. The first target weight can be denoted as α, and the second target weight as β. Both weights are independent and learnable parameters, and they borrow from the task-oriented attention mechanism concept, adaptively fluctuating between [0, 1] in conjunction with the following target loss function. If the target loss function is denoted as L, then the specific target loss function is: L = α(L MobileNetV3 +L MobileVit )+βL Skip .

[0138] Therefore, during training, training data can be acquired first. This training data may include the low-resolution video to be reconstructed and its corresponding high-resolution video. The data is transmitted to the dense module in the initial reconstruction model for adaptive training. In the dense module, the second feature cascade layer can continuously adjust the first initial weight α and the second initial weight β according to the training results to obtain the first target weight α and the second target weight β. Thus, the target loss function mentioned above can be obtained, and the initial reconstruction model can also be transformed into the preset reconstruction model mentioned above.

[0139] During adaptive training, when the detailed and contextual information learned by skip connections at different levels is less important, while the image feature information transmitted by the attention mechanism and convolution is more important, β can be decreased and α increased; conversely, α can be decreased and β increased. This task-oriented mechanism ensures that in the MobileVit attention-based convolutional network, the hierarchical feature information learned by each residual connection positively impacts the performance of the video super-resolution network in learning low-resolution image features, further improving reconstruction quality.

[0140] In this embodiment, the device described above can first extract features through a dense module after acquiring the low-resolution video to be reconstructed, and then reconstruct the video based on the first extraction result to obtain an initial high-resolution video. Then, a non-local attention module is used to determine the correlation of the initial high-resolution video using a self-attention mechanism, and adjustments are made based on this correlation. This allows the obtained target high-resolution video to take into account the correlation of different locations, thereby improving the reconstruction effect.

[0141] refer to Figure 5 , Figure 5 This is a flowchart illustrating the second embodiment of the video super-resolution reconstruction method of this application.

[0142] like Figure 5 As shown, in use, the step of reconstructing the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video includes:

[0143] Step S21: Extract features from the low-resolution video to be reconstructed using the dense module, and reconstruct the low-resolution video to be reconstructed based on the first extraction result to obtain an initial high-resolution video;

[0144] Step S22: The correlation of the initial high-resolution video is determined by the self-attention mechanism of the nonlocal residual module, and the initial high-resolution video is globally calibrated based on the correlation to obtain the target high-resolution video.

[0145] In the specific implementation, after obtaining local and global features through the first and second networks in the dense module, the initial high-resolution video is reconstructed based on the local and global features. Then, after upsampling, deep sampling, and linear addition, it is transmitted to the nonlocal residual module. The nonlocal residual module can extract the correlation of the initial high-resolution video after addition, and then perform global calibration on the initial high-resolution video after addition based on the correlation to obtain the target high-resolution video.

[0146] In order to further capture global information of the frame image, learn internal structure and texture information, and thus ensure that the high-resolution image recovered from the low-resolution frame image has more natural brightness, clearer image and more realistic edge contour, in this embodiment, the above-mentioned preset reconstruction model further includes: a non-local residual module, which is connected to the second network in the dense module, and the above-mentioned non-local attention module can be set in the above-mentioned non-local residual module.

[0147] For ease of understanding, please refer to Figure 6 To explain, Figure 6 This is a schematic diagram of the structure of the preset reconstruction model in the second embodiment of the video super-resolution reconstruction method of this application. Figure 6 As shown, after reconstructing local and global features in the second network of the dense module, an initial high-resolution video can be obtained, which is then transmitted to the non-local residual module (i.e., Figure 6 In the nonlocal residual network based on the MobileVit attention mechanism, the aforementioned nonlocal residual module can be a nonlocal residual network based on the MobileVit attention mechanism. Specifically, it can be located after the linear summation of the features. It can capture the global information of the feature map and use structural self-similarity loss to determine the similarity between different positions in the feature map, learn the internal structure and texture information of the image, and at the same time learn the temporal continuity and spatial correlation between video frames. It focuses on the global information in the feature map, and then adaptively adjusts the feature map of the key frame, performs detailed adjustments and optimizations to the feature map, and performs global calibration, thereby ensuring that the high-resolution image recovered from the low-resolution key frame image has more natural brightness, clearer image, and more realistic edge contours.

[0148] Furthermore, in order to take into account the correlation, the nonlocal residual module mentioned above in this embodiment may include the nonlocal attention module mentioned above, and the nonlocal attention module may be located after the addition of the linear features mentioned above.

[0149] Furthermore, in order to achieve the above-mentioned detailed adjustments, in this embodiment, the nonlocal residual module further includes a hierarchical residual module, and the nonlocal attention module is connected to the hierarchical residual module;

[0150] The step of performing global calibration on the initial high-resolution video based on the correlation to obtain the target high-resolution video includes:

[0151] The initial high-resolution video is globally calibrated based on the correlation; the calibrated initial high-resolution video is then fused hierarchically using the hierarchical residual module to obtain the target high-resolution video.

[0152] like Figure 6 As shown, the nonlocal residual module may include a nonlocal attention module based on an attention mechanism (i.e., Figure 6 Non-local modules) and hierarchical residual modules (i.e. Figure 6 The non-local attention module can be located after the linear feature addition and before the hierarchical residual module.

[0153] The aforementioned nonlocal attention module can be a module built based on the nonlocal residual module of MobileVit, and the aforementioned hierarchical residual module can be a module composed of multiple residual connections. In this embodiment, the nonlocal residual module based on MobileVit can be used as the basic unit and integrated into the MobileNet V3 convolutional neural network to form a nonlocal residual module.

[0154] This is because reconstructing a high-quality image requires accurately restoring the structural and textural relationships between different regions.

[0155] By introducing a nonlocal attention mechanism, the network can better learn the semantic information of the input image and the relationships between pixels. The hierarchical residual module utilizes residual connections to optimize feature propagation and gradient flow. By establishing residual connections between different layers, the network can more effectively reuse and fuse feature information from different layers, thereby enhancing the expressive power of features.

[0156] It should be noted that in this embodiment, the non-local attention module and the hierarchical residual module can be iterated multiple times and then integrated with the deep network, so that the network can give full play to the effect of the self-attention mechanism.

[0157] For ease of understanding, please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of the nonlocal residual module in the second embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 7 As shown, the nonlocal residual module in this embodiment may include several nonlocal residual blocks. This embodiment uses four nonlocal residual blocks for illustration, and each nonlocal residual block may include: a nonlocal attention module and a hierarchical residual module.

[0158] In practical use, it can first be achieved through a MobileNet V3 convolutional neural network (i.e., Figure 7 The process of extracting frame images from the reconstructed initial high-resolution video using MobileNet V3 convolutional blocks can be represented by the following formula 10:

[0159] F s1 =f 0 (I LR1 ) Formula 10;

[0160] Among them I LR1 For the initial high-resolution video obtained after reconstruction, f 0 For MobileNet V3 convolutional blocks,

[0161] F s1 The extracted frame image;

[0162] Next, a non-local attention module is used for feature extraction to obtain global information and dependencies, completing global calibration. This ensures that the high-resolution image reconstructed from the low-resolution keyframe image has more natural brightness, clearer image, and more realistic edge contours. Then, a hierarchical residual module is used to perform hierarchical fusion of the extracted features. Low-level features help restore local details of the image, while high-level features help maintain the overall structure and semantic information of the image. Finally, the features before and after the convolutional network can be linearly stacked through global skip connections, thereby realizing the processing of the non-local residual network based on the attention mechanism. The multi-layer stacked hierarchical residual module can linearly align the low-dimensional and high-dimensional feature maps of the keyframe, realizing linear mapping of keyframe feature map pairs, which can reduce the speed of reconstructing low-resolution images into high-resolution images and improve the network's generalization performance.

[0163] Furthermore, continue as Figure 7 As shown, in this embodiment, the non-local attention module may include a second network. This second network employs a self-attention mechanism to correlate the extracted frames of the initial high-resolution video, and based on this correlation, global calibration of the initial high-resolution video can be performed. The specific process is described in [reference needed]. Figure 8 , Figure 8 This is a schematic diagram of the structure of the non-local attention module in the second embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 8 As shown, the nonlocal residual module in this embodiment may include several nonlocal residual blocks. This embodiment uses four blocks for illustration, and they are connected in sequence.

[0164] Further, the steps described above, including determining the correlation of the initial high-resolution video using a self-attention mechanism through the non-local attention module and performing global calibration on the initial high-resolution video based on the correlation to obtain the target high-resolution video, include:

[0165] The initial high-resolution video is dimensionality reduced using the non-local attention module, and features are extracted from the dimensionality-reduced initial high-resolution video. A self-attention mechanism is used to transform the extracted initial high-resolution video to obtain a self-similarity matrix. The correlation of the initial high-resolution video is determined based on the self-similarity matrix, and global calibration is performed on the dimensionality-reduced initial high-resolution video based on the correlation to obtain the target high-resolution video.

[0166] In practical applications, the extracted initial high-resolution video frame images are first subjected to dimensionality reduction using a non-local attention module. Let's assume the input frame image size to the non-local attention module is M×N, and the number of channels is C, denoted as X. M ×N×C (Right now Figure 8 In the middle (X), dimensionality reduction is performed using 1×1 convolution, reducing its feature dimension to X. M×N×C / 2 This effectively reduces the computational burden;

[0167] Next, the self-similarity of the dimensionality-reduced feature maps is calculated using a structural similarity function, which can be...

[0168]

[0169] In this process, a 3×3 MobileNet V3 convolutional layer is first used to extract features from the dimensionality-reduced frame image to obtain... as well as

[0170] To facilitate computation and enable the model to learn the dependencies between different regions within the feature map, an attention-weighted aggregation strategy is used to convert the obtained feature map information into a matrix. and The self-similarity matrix of the frame image is then obtained by transposing and multiplying. The correlation can then be obtained from this self-similarity matrix.

[0171] Further, the step of determining the correlation of the initial high-resolution video based on the self-similarity matrix includes:

[0172] A weight matrix is ​​obtained based on the dimensionality-reduced initial high-resolution video, and a nonlocal matrix is ​​obtained based on the weight matrix and the self-similar matrix; the nonlocal matrix is ​​transformed, and the correlation of the initial high-resolution video is determined based on the transformation result.

[0173] In practical applications, the self-similarity matrix is ​​obtained. Then, it can be combined with the nonlocal transformation coefficient w(x) j Multiply by 1; simultaneously use a 1×1 MobileNet V3 convolutional network (i.e., ...) in the non-local attention module. Figure 8 The algorithm performs calculations on each pixel block in the frame image to achieve cross-channel information integration and interaction, enhance the non-linear characteristics of the network, and generate a weight matrix containing information on the relationships between pixels at different locations in the frame image. N represents the number of pixels in the frame image used by the non-local attention module. Next, let z... s With z w Multiplying two matrices yields a nonlocal matrix.

[0174] After converting the nonlocal matrix into a 3D frame image, it is then subjected to a 1×1 convolution (i.e., Figure 8 (h) Obtain frame image Z containing self-similarity information (i.e., the aforementioned correlation). M×N×C / 2 Finally, let X M×N×C / 2 With Z M×N×C / 2 Adding corresponding elements, corresponding to the initial channel descent, achieves global calibration. Then, using a 1×1 upscaling method, the feature map channels are increased to obtain the final result Y. M×N×C The convolution process in this step integrates the MobileNet V3 convolutional modules using grouped convolution. Grouped convolution significantly reduces the number of parameters by dividing the input feature map into multiple groups and performing convolution operations independently within each group. This makes the model more lightweight, reduces computational resource consumption, and helps prevent overfitting. Grouped convolution is a sparse operation, making it easier for the model to learn important feature representations during training, providing a certain degree of regularization, reducing computational complexity, and improving the model's inference speed.

[0175] After obtaining the initial high-resolution video Y after calibration M×N×C Then, it can be transmitted to the hierarchical residual module, which is connected to the nonlocal attention module;

[0176] The step of performing global calibration on the initial high-resolution video based on the correlation to obtain the target high-resolution video includes:

[0177] The initial high-resolution video is globally calibrated based on the correlation; the calibrated initial high-resolution video is then fused hierarchically using the hierarchical residual module to obtain the target high-resolution video.

[0178] In practical use, after global calibration using nonlocal attention, the initial high-resolution video after calibration can be fused hierarchically using a hierarchical residual module. Specifically, the hierarchical residual module can contain several residual modules. This embodiment uses eight residual modules for illustration, that is, each residual module is connected to the nonlocal attention module in a network cascade manner. The residual module can contain the first network and the second network mentioned above, and the nonlocal attention module is connected to the first network and the second network respectively.

[0179] The steps described above for performing hierarchical fusion on the calibrated initial high-resolution video using the hierarchical residual module to obtain the target high-resolution video include:

[0180] The initial high-resolution video after calibration is grouped using the hierarchical residual module; the grouping results are then feature-extracted using the first network and the second network respectively, and hierarchical fusion is performed based on the corresponding extraction results to obtain the target high-resolution video.

[0181] In practical use, the calibrated initial high-resolution video can be grouped first. In this embodiment, it can be grouped according to paired frame images and divided into 4 groups for explanation. Then, it is transmitted to the first network and the second network for feature extraction, and hierarchical fusion is performed based on the corresponding extraction results. The non-local residual module can adopt a global residual connection method, that is, the input of the first non-local attention module is connected to the output of the last non-local residual block. The above hierarchical fusion process can be expressed by the following formula 11:

[0182] F D1 =B 4 (B 3 (B 2 (B 1 (F s1 )))) Formula 11;

[0183] Among them B i (i = 1, 2, 3, 4) represents the i-th nonlocal residual block, F D1 This is the result of hierarchical fusion.

[0184] It needs to be emphasized that, continuing as Figure 6 as well as Figure 7 As shown, in order to generate the final super-resolution image, the nonlocal residual module in this embodiment does not use the traditional deconvolution method to enlarge the image.

[0185] While deconvolution can magnify images, its computation process can be complex and inefficient.

[0186] Therefore, this embodiment uses the subpixel convolution method in ESPCN to enlarge the image. That is, after the hierarchical fusion, a subpixel convolution module can be set up. This subpixel convolution module can be connected to the hierarchical residual module of the last non-local residual block.

[0187] The step of performing hierarchical fusion on the calibrated initial high-resolution video using the hierarchical residual module to obtain the target high-resolution video includes:

[0188] The initial high-resolution video after calibration is fused using the hierarchical residual module; the fusion result is amplified using the sub-pixel convolution module to obtain the target high-resolution video.

[0189] After obtaining the fusion result, the fusion result can be amplified through the subpixel convolution module mentioned above to obtain the target high-resolution video.

[0190] The subpixel convolutional layer in this subpixel convolution module upsamples low-resolution images, which can restore image details, rearrange the output of the convolutional layer, and reduce image blurring.

[0191] The enlarged initial high-resolution video can then be used as the target high-resolution video.

[0192] Specifically, refer to Figure 9 , Figure 9 This is a schematic diagram of the subpixel convolution module in the second embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 9 As shown, the sub-pixel convolution module described above can rearrange the fusion result into an r×r matrix (i.e., Figure 9 W×H×r 2 C is processed by subpixel convolution to obtain rW×rH×C), where r is the parameter of the subpixel convolution layer, representing the number of subpixels occupied by each pixel. Each element in this matrix represents a pixel in the high-resolution image. The fusion result is upsampled through the subpixel convolution layer. Simultaneously, the subpixel convolution layer can recover the edge loss caused by the convolution operation, restoring edge information to the higher-resolution image. Furthermore, the subpixel convolution layer increases the expressive power of features by expanding the number of channels in the input feature map, allowing the model to learn on higher-resolution feature maps, thereby improving the model's performance and accuracy. In this embodiment, eight subpixel convolution modules can be set, but other numbers are also possible; this embodiment does not impose any limitations on this.

[0193] It should also be emphasized that the optimization function of the nonlocal residual module in this embodiment can be represented by the mean absolute error, which can be specifically expressed by the following formula 12:

[0194]

[0195] Among them, the above L SR For the target high-resolution video, N is the number of pixels in the fused frame image. For the i-th pixel in the frame image of the initial high-resolution video obtained after reconstruction, Let be the i-th pixel in the frame image of the fused result, and 1 be the L1 loss function.

[0196] Furthermore, considering that AR glasses generally display 3D video with a certain depth information, therefore, continuing as follows... Figure 6 and Figure 7 As shown, in this embodiment, the dense module further includes a disparity map generation module, and the nonlocal residual module further includes a disparity map refinement and regression module. The disparity map generation module is connected to the second network and the nonlocal attention module, and the disparity map refinement and regression module is connected to the hierarchical residual module.

[0197] After the step of extracting global features from the low-resolution video to be reconstructed using the self-attention mechanism through the second network, the method further includes:

[0198] The disparity map generation module extracts depth information from the low-resolution video to be reconstructed to obtain a disparity map.

[0199] After the step of performing hierarchical fusion on the calibrated initial high-resolution video using the hierarchical residual module, the method further includes:

[0200] Based on the disparity map, the initial high-resolution video is linearly added using the disparity map refinement regression module to obtain the target high-resolution video.

[0201] It should be noted that the disparity map generation module described above can be connected after the last second attention mechanism layer in the dense module and before upsampling. The disparity map refinement and regression module can be connected after the last non-local residual block.

[0202] The disparity map generation module mentioned above may include a regularization extraction layer (i.e. Figure 6 (Regularized extraction) and window matching computation layer (i.e.) Figure 6 (MSD window matching calculation), wherein the above regularization extraction layer can be connected to the last second attention mechanism layer and the window matching calculation layer in the dense module, respectively;

[0203] The regularization extraction layer can use the L1 regularization loss function to extract the regularization information of the frame image output by the last second attention mechanism layer, providing a positive reference for learning and processing the depth information of AR videos. This layer can constrain the complexity of the model and encourage parameter sparsity by adding the sum of the absolute values ​​of the model parameters as a regularization term to the loss function. L1 regularization imposes constraints on the weight parameters of the convolutional layer. Since the L1 regularization term is the sum of the absolute values ​​of the weights, its derivative will jump when the weight is zero (from -1 to 1 or from 1 to -1). This property makes it easier for some weight values ​​to be pushed to zero during the optimization process, thereby achieving weight sparsity.

[0204] The regularization extraction layer can transmit the obtained regularization information to the window matching calculation layer. The window matching calculation layer can include MSD window matching calculation. Based on the regularization information, the network using MSD window matching calculation can generate a low-resolution disparity map (i.e., the aforementioned disparity map). This disparity map can be a difference map of binocular imaging. In the human visual system, due to the distance between the two eyes, the image position of the same object on the retinas of the two eyes is slightly different; this difference is called disparity. Specifically, the window matching calculation layer can first determine a reference window and candidate windows for the frame image based on the regularization information. The reference window can be a window used for reference, selected from a small rectangular region in the frame image of the low-resolution video to be reconstructed before input to the regularization extraction layer. This reference window can contain the features or patterns to be matched. The candidate window can be a window used to compare with the reference window to determine depth information, and can be a window corresponding to a larger search region defined for the received frame image.

[0205] After determining the reference window and candidate windows, depth information can be extracted to generate a disparity map. The specific process is as follows: determine the number of pixels in the window, and determine the pixel values ​​of the pixels in the reference window and the pixel values ​​of the pixels in the candidate windows; determine the similarity score based on the number of pixels in the window, the pixel values ​​of the pixels in the reference window, and the pixel values ​​of the pixels in the candidate windows; generate a disparity map based on the similarity score.

[0206] The number of pixels in the aforementioned window can be the number of pixels contained in the reference window or the candidate window. The pixel value of the pixel in the aforementioned reference window can be the grayscale value or intensity value corresponding to the pixel in the reference window. The pixel value of the pixel in the aforementioned candidate window can be the grayscale value or intensity value corresponding to the pixel in the candidate window.

[0207] The similarity score mentioned above can be considered as the similarity score between the reference window and the candidate window. The specific calculation can be expressed by the following formula 13:

[0208]

[0209] Where MSD is the similarity score, N is the number of pixels in the window, l1(i) is the pixel value of the i-th pixel in the reference window, and l2(i) is the pixel value of the i-th pixel in the candidate window.

[0210] After obtaining the similarity score, the similarity scores of all candidate windows are compared. The window with the smallest MSD value can be selected from the candidate windows. This candidate window can be the window most similar to the reference window and is used as the best matching window.

[0211] After determining the optimal matching window, its position information in the frame image can be determined. The position information and the similarity score of the optimal matching window are output together to generate a disparity map and send it to the disparity refinement regression module. The disparity map generated by the disparity map generation module is a lightweight feature matching disparity map. This disparity map mainly contains the depth information in the keyframes, which prepares the input for the disparity refinement regression module in the next part of the non-local residual network and provides depth-level information reference for subsequent reconstruction of high-resolution video.

[0212] After obtaining the disparity map, it can be used as input to the disparity map refinement regression module. In the disparity map refinement regression module of this embodiment, the unsupervised structural similarity index (SSIM) loss function can be used to determine the image residual between the disparity map and the frame image of the low-resolution video to be reconstructed before the input regularization layer through an upsampling network. Then, the image residual is added to the disparity map to obtain the accurate disparity map. Finally, the obtained accurate disparity map is linearly superimposed with the frame image of the low-resolution video to be reconstructed before the input regularization layer using a global connection. The feature map obtained in this way not only contains the local and global feature information of the key frame image, but also contains the feature information at the depth level, thereby obtaining the target high-resolution video.

[0213] Simultaneously refer to Figure 7 In this embodiment, the disparity thinning regression module can also use a global residual connection to directly transmit the output to the subsequent linear addition. This allows the network to learn the residual between the input and output, focusing on the difference between the input and output images, rather than learning the entire image from scratch. This connection method helps prevent the network from losing key information during training, helps ensure the stability of training, and improves the quality of the reconstructed image. Furthermore, the amplification process of the sub-pixel convolution module can be represented by the following formula 14:

[0214] I SR =S UP (F s2 +F D2 ) Formula 14;

[0215] F s2 For the frame image of the first non-local residual block as input, F D2 S represents the image residual output by the disparity refinement regression module. UP For the scaling operation of the subpixel convolution module, I SR For high-resolution video.

[0216] Secondly, considering both the speed and quality of the generated image, this embodiment does not use deconvolution to enlarge the image, but instead uses subpixel convolution in ESPCN, which has the advantage of being fast.

[0217] This embodiment can capture global information of the frame image through a nonlocal residual module, learn its internal structure and texture information, thereby ensuring that the high-resolution image recovered from the low-resolution frame image has more natural brightness, clearer image, and more realistic edge contours, further improving image quality. At the same time, through the disparity map generation module and the disparity map refinement regression module, the disparity map is linearly superimposed with the obtained initial high-resolution video to obtain the target high-resolution video, thereby realizing AR video super-resolution reconstruction.

[0218] refer to Figure 10 , Figure 10 This is a flowchart illustrating the third embodiment of the video super-resolution reconstruction method of this application.

[0219] To color grade the video and further improve its resolution, such as... Figure 10 As shown, in this embodiment, the above-mentioned preset reconstruction model further includes: a low-resolution video super-resolution module, which is connected to the non-local residual module;

[0220] Following the step of performing global calibration on the initial high-resolution video based on the correlation, the method further includes:

[0221] Step S23: Perform color restoration on the calibrated initial high-resolution video using the low-resolution video super-resolution module, and then perform resolution enhancement on the restored initial high-resolution video to obtain the target high-resolution video.

[0222] Reference Figure 11 , Figure 11 This is a schematic diagram of the structure of the preset reconstruction model in the third embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 11As shown, the low-resolution video super-resolution module in this embodiment can be connected after the sub-pixel convolution module in the non-local residual module. The low-resolution video super-resolution module can be a module of a deep network formed by continuously stacking a multi-channel attention mechanism module and MobileNet V3 convolutional blocks. In this embodiment, the low-resolution super-resolution module can improve and optimize the channel attention mechanism, which can color grade low-quality videos and improve resolution. The channel attention mechanism can make the network pay more attention to the channels with high information content in the input keyframe image, which can help the network better understand and utilize the feature information in the video frame, thereby improving the quality of super-resolution reconstruction.

[0223] This embodiment can construct a channel attention mechanism module using a channel attention mechanism, which in turn forms a color restoration module for color recovery and a resolution enhancement module for resolution enhancement (super-resolution). For the color restoration part, this module can focus on features such as color, brightness, and contrast; for the resolution enhancement part, this module can focus on details such as texture and edges. Therefore, the two modules differ only in parameters and depth to adapt to different task requirements.

[0224] To facilitate understanding the channel attention mechanism module, refer to... Figure 12 , Figure 12 This is a schematic diagram of the channel attention mechanism module in the third embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 12 As shown, the channel attention mechanism used in this embodiment may include a feature information extraction module and a linear addition layer;

[0225] The linear addition layer can be placed after the feature information extraction module. The feature information extraction module can extract features from the input frame image to obtain feature information. The linear addition layer can perform color restoration or resolution enhancement on the low-resolution video to be reconstructed based on the feature information to obtain the target high-resolution video.

[0226] For the feature information extraction module, a multi-scale channel attention module and a small residual module (i.e., ...) can be set up. Figure 12 Small and medium residual modules 1 to small residual module M) and large residual module (i.e. Figure 12 The medium-to-large residual modules 1 to large residual modules M, the multi-scale channel attention module, the small residual module, and the large residual module are set in parallel. That is, the inputs of the multi-scale channel attention module, the small residual module, and the large residual module are all connected to the sub-pixel convolution module, and the outputs are all connected to the linear additive layer.

[0227] In this embodiment, the multi-scale channel attention module can effectively distinguish between the feature maps extracted by the MobileVit attention mechanism and the MobileNet V3 convolutional network. It can differentiate and process different feature maps based on their importance and relevance, thereby extracting more accurate information useful for super-resolution reconstruction. Different feature maps contain different information, and some feature maps may be more important and require more attention. Therefore, when fitting nonlinear mappings, each feature map is processed in different ways to better utilize its information, learn richer feature map channel information, and effectively utilize the interdependencies between feature maps to provide prior guidance for network inference.

[0228] Reference Figure 13 , Figure 13 This is a schematic diagram of the multi-scale channel attention module in the third embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 13 As shown, this embodiment improves upon traditional channel attention by assuming that the features input to the multi-scale channel attention module are... Figure X The number of images is C, each with a size of W×H. They can be initially processed using two different sizes of MobileNet V3 convolutional layers: 3×3 and 5×5 (i.e., ...). Figure 13 The network performs convolution operations using 3×3 and 5×5 convolutions. These two types of convolution kernels can capture local details and broader spatial context information, respectively, thereby generating two new sets of features, U and V. The channel dimensions of U and V are the same as X, namely C×W×H. Subsequently, the multi-scale channel attention module merges these two sets of features U and V together by adding corresponding matrix elements. This merging strategy ensures the effective fusion of feature information at different scales, enabling the network to more comprehensively understand and process image content. The fusion process can be represented by the following formula 15:

[0229] Formula 15: M = U + V;

[0230] Where M represents the feature after adding the matrix elements.

[0231] The feature maps processed by the multi-scale channel attention module require further processing to extract deeper features. This is followed by global pooling (i.e., ...). Figure 13 Global pooling (GSP) is applied to the feature maps, compressing each feature map of size W×H into a single value (i.e., ...). Figure 13 (C×1). This step helps to aggregate global information from each feature map, providing a more representative feature representation for subsequent operations.

[0232] After global pooling, the resulting features are fed into two fully connected layers (i.e., Figure 13 Fully Connected 1

[0233] And then the fully connected layers (2) perform mapping. The role of these two fully connected layers is to further transform the pooled features and finally map them to a 2C×1 dimension. This process can be represented by the following formula 16:

[0234] w = FC2(FC1(GP(M))) Formula 16;

[0235] Where GP is the global pooling operation, FC1 represents the first fully connected layer, FC2 represents the second fully connected layer, and w is the mapping result. After obtaining the feature w, a reshape operation is used to expand it from two dimensions to three dimensions. Then, it is passed through a preset activation function (such as the sigmoid function) to obtain an output with the same dimension as the transformed feature map. This output can be interpreted as the importance weight of each channel. This weight is a feature vector containing channel attention information, with a dimension of 2C×1×1. Next, the obtained feature vector is divided into two sub-vectors u and v corresponding to the two sets of features U and V obtained earlier. These sub-vectors are multiplied by U and V respectively and then added together, thus completing the entire process of multi-scale attention mechanism calibration of the feature map and obtaining the relevant feature information of the color channel (i.e., the red, green, and blue (RGB) channel), i.e., the color feature information, denoted as Y. The specific process can be expressed by the following formula 17:

[0236] Y = uU + vV (Formula 17)

[0237] Continue as Figure 12 As shown, a convolutional layer (i.e., ...) can be sequentially connected before the aforementioned multi-scale channel attention module. Figure 12 (MobileNet V3 convolutional block), and an activation function layer (i.e.) Figure 12 (ReLU)

[0238] and a convolutional layer (i.e.) Figure 12 In MobileNet V3 convolutional blocks, all convolutional layers can be 1×1.

[0239] The process involves convolution, where the first convolutional layer reduces the number of channels in the input feature map to 1 / r of the original number of channels, where r is the number of channels in the input feature map. This helps reduce computational cost and the number of parameters. Then, an activation function layer (such as ReLU) activates the reduced feature map. A second convolutional layer restores the number of channels to the original dimension, generating attention coefficients with the same number of channels as the original feature map. The restored feature map is then passed to a multi-scale channel attention module for the same processing, resulting in a feature map with channel attention coefficients. Finally, an element-wise multiplication operation is performed with the original feature map. This multiplication process recalibrates the original feature map, adjusting the feature values ​​of each channel according to its importance. In this way, the model can focus more on channel information that has a significant impact on super-resolution reconstruction while suppressing less important channel information, thereby improving the performance and efficiency of video super-resolution reconstruction.

[0240] It should be emphasized that the aforementioned color feature information can be information about color channel features in the image, the aforementioned hierarchical feature information can be feature information between different layers of images, and the aforementioned spatial feature information can be feature information of key regions in the image.

[0241] For small residual modules, hierarchical features of the input feature map can be extracted to obtain hierarchical feature information. For large residual modules, spatial features of the input feature map can be extracted to obtain spatial feature information. Finally, color adjustment or super-resolution tasks are completed based on color feature information, hierarchical feature information, and spatial feature information.

[0242] In both small and large residual modules, a convolutional layer (i.e., ...) can be used first. Figure 12 The shallow feature extraction is performed using MobileNet V3 convolutional blocks, and this process can be represented by the following formula 18:

[0243] F s3 =C SF (I LR ) Formula 18;

[0244] Among them I LR C is the initial high-resolution video output by the subpixel convolution module. SF For convolution operations, F s3 This represents the shallow features obtained from the convolutional layer. These shallow features are then passed to the small residual module and the large residual module for deep feature extraction. This process can be represented by the following formula 19:

[0245] F D3 =R DF (F s3 ) Formula 19;

[0246] Where R DF For the extraction of the aforementioned deep features, F D3 For extracting results.

[0247] It is important to emphasize that the small residual module and the large residual module can be multi-level residual structures. To balance the complexity of the network and the stability of training, the low-quality video super-resolution network adopts a design strategy of nesting small residual modules within large residual modules (i.e., Figure 12 (As shown). This nested structure not only deepens the network but also effectively prevents the gradient explosion problem during training. Simultaneously, by increasing the number of feature extraction units, the network can expand its receptive field, enhance its ability to capture image information, and further improve the accuracy of feature extraction.

[0248] It should also be emphasized that, if Figure 12 The structure shown can be used for color correction tasks without needing a subpixel convolutional module after the linear summation layer. However, if used for super-resolution tasks, a subpixel convolutional module can be placed after the linear summation layer (i.e., ...). Figure 12 (Subpixel convolution), this subpixel convolution module may contain subpixel convolution layers. After the super-resolution reconstruction task is performed through linear summation layers, the obtained result can be upsampled and amplified through this subpixel convolution module to further improve the resolution. Finally, a MobileNet V3 convolution operation is performed to generate the final target high-resolution video. This process can be represented by the following formula 20:

[0249] I LR =C RE (S UP (F D3 )) Formula 20.

[0250] Where S UP For the upsampling operation of this subpixel convolution module, C RE For the convolution operation of this convolutional layer, I LR For high-resolution video.

[0251] Furthermore, to differentiate between color grading tasks and super-resolution tasks, this embodiment can set different loss functions for different tasks. For the channel attention mechanism module in the color grading task, the corresponding color grading stage loss function can be composed of two loss functions: mean absolute error and structural similarity error (SSIM), which can be expressed by the following formula 21:

[0252] L Color =αL1+βL SSIM Formula 21;

[0253] Where L CollrL1 is the loss function for the color grading stage, and L2 is the L1 loss function. SSIM Let α be the structural similarity error loss function, and let β be the weight coefficients. These can be obtained through adaptive training. In color correction tasks, since structural similarity error can focus on the structural information of the image, it tends to retain more details such as edges and textures during the optimization process, making the reconstructed video more natural and realistic. Therefore, β will be smaller.

[0254] For the channel attention mechanism module in the super-resolution task, the corresponding super-resolution stage loss function can be constructed using the mean absolute error, and can be expressed by the following formula 22:

[0255]

[0256] Where L SR For the super-resolution stage loss function, Let be the frame image of the target super-resolution video after super-resolution, and 1 be the L1 loss function.

[0257] To further differentiate between color correction and super-resolution tasks, this embodiment can also incorporate a gating mechanism into the linear addition layer. Specifically, corresponding gates are set for the multi-scale channel attention module, the large residual module, and the small residual module, respectively, as shown in the following example. Figure 14 , Figure 14 This is a schematic diagram of the gating mechanism in the third embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 14 As shown, a channel gate can be set for the multi-scale channel attention module, a spatial gate can be set for the large residual module, and a hierarchical gate can be set for the small residual module. A target feature weight (i.e., ...) is set for each gate. Figure 14 The target feature weights for the channel gate (h), spatial gate (σ), and hierarchical gate (tanh) are obtained through adaptive training of the initial feature weights using training data. The channel gate controls the flow of information between channels by calculating the weight of each channel. This helps the model focus on important channel features while ignoring irrelevant channel information. The spatial gate controls the flow of spatial information by calculating the weight at each location, which helps the model focus on key regions in the image while suppressing the influence of background or noise regions. The hierarchical gate controls the interaction between information at different levels.

[0258] Before use, training data containing the target high-resolution video and the low-resolution video to be reconstructed can be obtained, and the initial feature weights set for each gate in the gating mechanism can be trained. For color grading tasks, color feature information may be more important, and the target feature weights obtained by the channel gate after self-learning may be higher. For super-resolution tasks, the target feature weights obtained by the hierarchical gate and spatial gate lock after self-learning may be higher.

[0259] Therefore, the aforementioned gating mechanism can learn the importance or weight of each channel and adjust the output of the feature map based on these weights. This helps improve the quality of super-resolution reconstruction, resulting in clearer and more detailed images. The gating mechanism can process all channels uniformly, meaning that regardless of the number of channels in the input feature map, the gating mechanism can use the same structure and parameters for computation. This uniform approach not only simplifies the model's complexity but also makes the model more flexible and scalable.

[0260] In summary, in this embodiment, the low-resolution video super-resolution module is sequentially connected to a color restoration module and a resolution enhancement module, and the color restoration module is also connected to the non-local residual module.

[0261] The steps of performing color restoration on the calibrated initial high-resolution video using a low-resolution video super-resolution module, and then performing resolution upscaling on the restored initial high-resolution video to obtain the target high-resolution video include:

[0262] The color restoration module restores the color of the calibrated initial high-resolution video; the resolution enhancement module enhances the resolution of the restored initial high-resolution video to obtain the target high-resolution video.

[0263] It should be noted that the aforementioned color restoration module can be connected after the sub-pixel convolution module in the non-local residual module. Furthermore, the aforementioned color restoration module may include the aforementioned channel attention mechanism module. For ease of subsequent explanation, the channel attention mechanism module in the color restoration module will be referred to as the first channel attention mechanism module, and the first channel attention mechanism module may include the aforementioned gating mechanism and color grading stage loss function set for the color grading task.

[0264] The multi-scale channel attention module in the first channel attention mechanism module is denoted as the first multi-scale channel attention module, the small residual module is denoted as the first small residual module, the large residual module is denoted as the first large residual module, the convolutional layer and activation function layer before the first multi-scale channel attention module can be denoted as the third convolutional layer, the first activation function layer and the fourth convolutional layer, respectively, and the linear addition layer is denoted as the first linear addition layer.

[0265] In practical use, the input initial low-resolution video (i.e., the initial low-resolution video after being amplified by the sub-pixel convolution module) is down-channeled through the third convolutional layer, and the down-channeled initial low-resolution video is activated through the first activation function layer; the activated initial low-resolution video is restored through the fourth convolutional layer, and the color features of the restored initial low-resolution video are extracted through the first multi-scale channel attention module to obtain color feature information; the hierarchical features of the input initial low-resolution video are extracted through the first small residual module to obtain hierarchical feature information; the spatial features of the input initial low-resolution video are extracted through the first large residual module to obtain spatial feature information; and finally, the color is restored through the gating mechanism in the first linear addition layer based on the color feature information, hierarchical feature information, and spatial feature information according to the corresponding target feature weights to obtain the restored initial high-resolution video.

[0266] It should be emphasized that the above process can be referenced from the... Figures 12 to 14 The specific process will not be described in detail in this embodiment.

[0267] Similarly, the resolution enhancement module described above can be connected after the color restoration module. The resolution enhancement module can also include the aforementioned channel attention mechanism module, and its depth can be greater than that of the color restoration module (i.e.,...). Figure 11 (7 layers in the middle). For ease of explanation, the channel attention mechanism module in the resolution enhancement module will be referred to as the second channel attention mechanism module. The second channel attention mechanism module may include the gating mechanism and the loss function set for the super-resolution task.

[0268] The multi-scale channel attention module in the second channel attention mechanism module is denoted as the second multi-scale channel attention module, the small residual module is denoted as the second small residual module, the large residual module is denoted as the second large residual module, the convolutional layer and activation function layer before the second multi-scale channel attention module can be denoted as the fifth convolutional layer, the second activation function layer and the sixth convolutional layer, respectively, and the linear addition layer is denoted as the second linear addition layer.

[0269] In practical application, the fifth convolutional layer performs channel downscaling on the restored initial high-resolution video, and the second activation function layer activates the downscaled initial low-resolution video. The sixth convolutional layer restores the channels of the activated initial low-resolution video, and the second multi-scale channel attention module extracts color features from the restored initial low-resolution video to obtain color feature information. The second small residual module extracts hierarchical features from the restored initial high-resolution video to obtain hierarchical feature information. The second large residual module extracts spatial features from the restored initial high-resolution video to obtain spatial feature information. Then, the gating mechanism in the second linear addition layer performs resolution upscaling based on color, hierarchical, and spatial feature information according to the corresponding target feature weights. Finally, a sub-pixel convolution is used for magnification, allowing for the processing of each magnified result (i.e.,...) Figure 11 By encoding high-quality keyframes (in medium to high quality) into the video, the aforementioned high-resolution video can be obtained.

[0270] It should be emphasized that the above process can also refer to the... Figures 12 to 14 The specific process will not be described in detail in this embodiment.

[0271] This embodiment can perform color restoration and resolution enhancement through the aforementioned low-resolution video super-resolution module, thereby further improving video quality.

[0272] Reference Figure 15 , Figure 15 This is a structural block diagram of the first embodiment of the video super-resolution reconstruction device of this application.

[0273] like Figure 15 As shown, the video super-resolution reconstruction apparatus proposed in this application includes:

[0274] The video acquisition unit 151 is used to acquire the low-resolution video to be reconstructed.

[0275] The resolution reconstruction unit 152 is used to reconstruct the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video.

[0276] The preset reconstruction model includes at least a dense module and a non-local attention module. The dense module is used to extract features from the low-resolution video to be reconstructed and reconstruct the low-resolution video to be reconstructed based on the first extraction result to obtain an initial high-resolution video. The non-local attention module is used to determine the correlation of the initial high-resolution video using a self-attention mechanism and adjust the initial high-resolution video based on the correlation to obtain the target high-resolution video.

[0277] In this embodiment, the device described above can first extract features through a dense module after acquiring the low-resolution video to be reconstructed, and then reconstruct the video based on the first extraction result to obtain an initial high-resolution video. Then, a non-local attention module is used to determine the correlation of the initial high-resolution video using a self-attention mechanism, and adjustments are made based on this correlation. This allows the obtained target high-resolution video to take into account the correlation of different locations, thereby improving the reconstruction effect.

[0278] In one implementation, the resolution reconstruction unit 152 is further configured to extract features from the low-resolution video to be reconstructed using the dense module, and reconstruct the low-resolution video to be reconstructed based on the first extraction result to obtain an initial high-resolution video; determine the correlation of the initial high-resolution video using a self-attention mechanism through the non-local attention module, and perform global calibration on the initial high-resolution video based on the correlation to obtain the target high-resolution video.

[0279] In one implementation, the resolution reconstruction unit 152 is further configured to reduce the dimensionality of the initial high-resolution video using the non-local attention module, and extract features from the dimensionality-reduced initial high-resolution video; to transform the extracted initial high-resolution video using a self-attention mechanism to obtain a self-similarity matrix; to determine the correlation of the initial high-resolution video based on the self-similarity matrix, and to perform global calibration on the dimensionality-reduced initial high-resolution video based on the correlation to obtain the target high-resolution video.

[0280] In one implementation, the resolution reconstruction unit 152 is further configured to obtain a weight matrix based on the dimensionality-reduced initial high-resolution video, and obtain a nonlocal matrix based on the weight matrix and the self-similar matrix; transform the nonlocal matrix, and determine the correlation of the initial high-resolution video based on the transformation result.

[0281] As one implementation, the preset reconstruction model further includes: a hierarchical residual module, which is connected to the nonlocal attention module;

[0282] The resolution reconstruction unit 152 is also used to perform global calibration on the initial high-resolution video based on the correlation; and to perform hierarchical fusion on the calibrated initial high-resolution video through the hierarchical residual module to obtain the target high-resolution video.

[0283] As one implementation, the hierarchical residual module includes: a residual module, which includes a first network that performs depthwise separable convolution operations using inverted residuals to extract local features and a second network that uses a self-attention mechanism to extract global features, and a non-local attention module that is connected to the first network and the second network respectively.

[0284] The resolution reconstruction unit 152 is also used to group the calibrated initial high-resolution video through the hierarchical residual module; extract features from the grouping results through the first network and the second network respectively, and perform hierarchical fusion based on the corresponding extraction results to obtain the target high-resolution video.

[0285] In one implementation, the first network includes a MobileNet V3 network, and the second network includes a MobileVit network.

[0286] As one implementation, the preset reconstruction model further includes: a subpixel convolution module, which is connected to the hierarchical residual module;

[0287] The resolution reconstruction unit 152 is also used to perform hierarchical fusion on the calibrated initial high-resolution video through the hierarchical residual module; and to amplify the fusion result through the sub-pixel convolution module to obtain the target high-resolution video.

[0288] Other embodiments or specific implementations of the video super-resolution reconstruction apparatus of this application can be found in the above-described method embodiments, and will not be repeated here.

[0289] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0290] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0291] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0292] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for video super-resolution reconstruction, characterized in that, The method comprises: acquiring a low-resolution video to be reconstructed; reconstructing the low-resolution video to be reconstructed by a preset reconstruction model to obtain a target high-resolution video; wherein the preset reconstruction model at least comprises a dense module and a non-local attention module, the dense module is used for feature extraction on the low-resolution video to be reconstructed, and the low-resolution video to be reconstructed is reconstructed according to a first extraction result to obtain an initial high-resolution video, the non-local attention module is used for determining the correlation of the initial high-resolution video by using a self-attention mechanism, and the initial high-resolution video is adjusted based on the correlation to obtain the target high-resolution video.

2. The method of claim 1, wherein, The step of reconstructing the low-resolution video to be reconstructed by the preset reconstruction model to obtain the target high-resolution video comprises: extracting features of the low-resolution video to be reconstructed by the dense module, and reconstructing the low-resolution video to be reconstructed according to the first extraction result to obtain the initial high-resolution video; determining the correlation of the initial high-resolution video by using the self-attention mechanism of the non-local attention module, and globally calibrating the initial high-resolution video based on the correlation to obtain the target high-resolution video.

3. The method of claim 2, wherein, The step of determining the correlation of the initial high-resolution video by using the self-attention mechanism of the non-local attention module, and globally calibrating the initial high-resolution video based on the correlation to obtain the target high-resolution video comprises: dimension reduction is performed on the initial high-resolution video by the non-local attention module, and features of the dimension-reduced initial high-resolution video are extracted; the self-similarity matrix is obtained by converting the extracted initial high-resolution video by using the self-attention mechanism; the correlation of the initial high-resolution video is determined based on the self-similarity matrix, and the dimension-reduced initial high-resolution video is globally calibrated based on the correlation to obtain the target high-resolution video.

4. The method of claim 3, wherein, The step of determining the correlation of the initial high-resolution video based on the self-similarity matrix comprises: a weight matrix is obtained based on the dimension-reduced initial high-resolution video, and a non-local matrix is obtained based on the weight matrix and the self-similarity matrix; the non-local matrix is converted, and the correlation of the initial high-resolution video is determined according to the conversion result.

5. The method of claim 2, wherein, The preset reconstruction model further comprises a hierarchical residual module, and the hierarchical residual module is connected with the non-local attention module. The step of globally calibrating the initial high-resolution video based on the correlation to obtain the target high-resolution video comprises: globally calibrating the initial high-resolution video based on the correlation; the hierarchical fusion of the calibrated initial high-resolution video is performed by the hierarchical residual module to obtain the target high-resolution video.

6. The method of claim 5, wherein, The hierarchical residual module comprises a residual module including a first network performing a deep separable convolution operation in an inverted residual manner to perform local feature extraction and a second network adopting a self-attention mechanism to perform global feature extraction, and the non-local attention module is connected with the first network and the second network respectively. The step of obtaining the target high-resolution video by performing hierarchical fusion on the calibrated initial high-resolution video through the hierarchical residual module comprises: Grouping the calibrated initial high-resolution video through the hierarchical residual module; Performing feature extraction on the grouping result through the first network and the second network respectively, and performing hierarchical fusion according to the corresponding extraction results to obtain the target high-resolution video.

7. The method of claim 6, wherein, The first network comprises a MobileNet V3 network, and the second network comprises a MobileVit network.

8. The method of any one of claims 5 to 7, wherein, The preset reconstruction model further comprises a sub-pixel convolution module connected with the hierarchical residual module. The step of obtaining the target high-resolution video by performing hierarchical fusion on the calibrated initial high-resolution video through the hierarchical residual module comprises: Performing hierarchical fusion on the calibrated initial high-resolution video through the hierarchical residual module; Performing magnification on the fusion result through the sub-pixel convolution module to obtain the target high-resolution video.

9. An apparatus for video super-resolution reconstruction, the apparatus comprising: a video super-resolution reconstruction module configured to perform the method of any one of claims 1-8. The device comprises: a video acquisition unit configured to acquire a low-resolution video to be reconstructed; a resolution reconstruction unit configured to reconstruct the low-resolution video to be reconstructed through a preset reconstruction model to obtain a target high-resolution video; wherein the preset reconstruction model at least comprises a dense module and a non-local attention module, the dense module is configured to perform feature extraction on the low-resolution video to be reconstructed, and reconstruct the low-resolution video to be reconstructed according to a first extraction result to obtain an initial high-resolution video, and the non-local attention module is configured to determine the correlation of the initial high-resolution video by adopting a self-attention mechanism, and adjust the initial high-resolution video based on the correlation to obtain the target high-resolution video.

10. A wearable device, comprising: The wearable device comprises a memory, a processor, and a video super-resolution reconstruction program stored on the memory and executable on the processor, and the video super-resolution reconstruction program, when executed by the processor, implements the steps of the video super-resolution reconstruction method according to any one of claims 1 to 8.

Images