Video super-resolution reconstruction method and device and wearable equipment

Abstract

The invention relates to the technical field of video processing, and discloses a video super-resolution reconstruction method and device and wearable equipment, and the method comprises the steps: obtaining a to-be-reconstructed low-resolution video; reconstructing the to-be-reconstructed low-resolution video through a preset reconstruction model to obtain a target high-resolution video; wherein the preset reconstruction model at least comprises a channel attention mechanism module provided with a gating mechanism, the gating mechanism comprises target feature weights corresponding to the feature information, the target feature weights are determined through adaptive training, and the channel attention mechanism module is used for performing feature extraction on the low-resolution video to be reconstructed to obtain the feature information; and reconstructing the to-be-reconstructed low-resolution video according to the corresponding target feature weight based on each piece of feature information to obtain a target high-resolution video. When the method is used, reconstruction can be carried out through a gating mechanism according to the target feature weights corresponding to the feature information, so that interference can be reduced, and the reconstruction effect is improved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video super-resolution reconstruction method, apparatus and wearable device. Background Technology

[0002] Currently, when performing super-resolution (SR) reconstruction on low-resolution (LR) videos to obtain high-resolution (HR) videos, different feature information exists during feature extraction, such as feature information related to color features, feature information related to spatial features, and feature information related to hierarchical features.

[0003] Current methods treat these features equally. However, not all features are relevant to the current task. For example, color features may be more important for color restoration, while spatial and hierarchical features may be more important for resolution reconstruction. Therefore, the current equal treatment approach may allow irrelevant information to interfere with the current task, resulting in poor reconstruction results. Summary of the Invention

[0004] The main purpose of this application is to provide a video super-resolution reconstruction method, apparatus and wearable device, which aims to solve the technical problem in the prior art that treating each feature information equally may cause information that is invalid for the current task to interfere with the current task, resulting in poor reconstruction effect.

[0005] To achieve the above objectives, this application provides a video super-resolution reconstruction method, the method comprising:

[0006] Acquire the low-resolution video to be reconstructed;

[0007] The low-resolution video to be reconstructed is reconstructed using a preset reconstruction model to obtain the target high-resolution video;

[0008] The preset reconstruction model includes at least: a channel attention mechanism module with a gating mechanism, wherein the gating mechanism carries target feature weights corresponding to each feature information, and each target feature weight is determined through adaptive training. The channel attention mechanism module is used to extract features from the low-resolution video to be reconstructed, obtain each feature information, and reconstruct the low-resolution video to be reconstructed based on each feature information and the corresponding target feature weights to obtain the target high-resolution video.

[0009] In one embodiment, prior to the step of acquiring the low-resolution video to be reconstructed, the method further includes:

[0010] Acquire training data and adaptively train the initial reconstruction model using the training data;

[0011] Based on the training results, the initial feature weights corresponding to each feature information in the gating mechanism are adjusted to obtain a preset reconstruction model with target feature weights.

[0012] In one embodiment, the channel attention mechanism module includes: a feature information extraction module and a linear addition layer with a gating mechanism, wherein the linear addition layer is connected to the feature information extraction module;

[0013] The step of reconstructing the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video includes:

[0014] The feature information extraction module extracts features from the low-resolution video to be reconstructed to obtain various feature information.

[0015] The target high-resolution video is obtained by reconstructing the low-resolution video to be reconstructed by the linear addition layer based on each feature information and according to the corresponding target feature weights.

[0016] In one embodiment, the feature information extraction module includes a multi-scale channel attention module, a large residual module, and a small residual module, wherein the multi-scale channel attention module, the large residual module, and the small residual module are all connected to the linear addition layer;

[0017] The step of extracting features from the low-resolution video to be reconstructed using the feature information extraction module to obtain various feature information includes:

[0018] The color features of the low-resolution video to be reconstructed are extracted by the multi-scale channel attention module to obtain color feature information.

[0019] The hierarchical features of the low-resolution video to be reconstructed are extracted by the small residual module to obtain hierarchical feature information.

[0020] The large residual module is used to extract spatial features from the low-resolution video to be reconstructed to obtain spatial feature information.

[0021] The step of reconstructing the low-resolution video to be reconstructed based on each feature information according to the corresponding target feature weights through the linear summation layer to obtain the target high-resolution video includes:

[0022] The linear additive layer reconstructs the low-resolution video to be reconstructed based on the color feature information, the hierarchical feature information, and the spatial feature information according to the corresponding target feature weights, thereby obtaining the target high-resolution video.

[0023] In one embodiment, the step of extracting color features from the low-resolution video to be reconstructed using the multi-scale channel attention module includes:

[0024] The low-resolution video to be reconstructed is globally pooled using the multi-scale channel attention module, and the pooling result is mapped.

[0025] The mapping results are segmented, and the color features of the segmented results are extracted.

[0026] In one embodiment, the step of segmenting the mapping result and extracting color features from the segmented result includes:

[0027] The mapping result is expanded, and the expanded result is transformed using a preset activation function;

[0028] The transformation result is segmented, and the color features of the segmented result are extracted.

[0029] In one embodiment, the feature information extraction module further includes: a first convolutional layer, an activation function layer, and a second convolutional layer connected in sequence, wherein the second convolutional layer is also connected to the multi-scale channel attention module;

[0030] The step of extracting color features from the low-resolution video to be reconstructed using the multi-scale channel attention module to obtain color feature information includes:

[0031] The first convolutional layer performs channel reduction on the low-resolution video to be reconstructed, and the activation function layer activates the reduced low-resolution video to be reconstructed.

[0032] The second convolutional layer performs channel restoration on the activated low-resolution video to be reconstructed, and the multi-scale channel attention module extracts color features from the restored low-resolution video to obtain color feature information.

[0033] In one embodiment, the feature information extraction module further includes a subpixel convolution module, which is connected to the linear addition layer;

[0034] The step of reconstructing the low-resolution video to be reconstructed using the linear additive layer based on the color feature information, the hierarchical feature information, and the spatial feature information according to the corresponding target feature weights, to obtain the target high-resolution video, includes:

[0035] The low-resolution video to be reconstructed is reconstructed by the linear additive layer based on the color feature information, the hierarchical feature information, and the spatial feature information according to the corresponding target feature weights, so as to obtain an initial high-resolution video.

[0036] The initial high-resolution video is magnified by the subpixel convolution module to obtain the target high-resolution video.

[0037] Furthermore, to achieve the above objectives, this application also proposes a video super-resolution reconstruction apparatus, the apparatus comprising:

[0038] The video acquisition unit is used to acquire the low-resolution video to be reconstructed.

[0039] The resolution reconstruction unit is used to reconstruct the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video.

[0040] The preset reconstruction model includes at least: a channel attention mechanism module with a gating mechanism, wherein the gating mechanism carries target feature weights corresponding to each feature information, and each target feature weight is determined through adaptive training. The channel attention mechanism module is used to extract features from the low-resolution video to be reconstructed, obtain each feature information, and reconstruct the low-resolution video to be reconstructed based on each feature information and the corresponding target feature weights to obtain the target high-resolution video.

[0041] In addition, to achieve the above objectives, this application also proposes a wearable device, which includes: a memory, a processor, and a video super-resolution reconstruction program stored in the memory and executable on the processor. When the video super-resolution reconstruction program is executed by the processor, it implements the steps of the video super-resolution reconstruction method described above.

[0042] This application provides a video super-resolution reconstruction method, apparatus, and wearable device. The method includes: acquiring a low-resolution video to be reconstructed; reconstructing the low-resolution video to be reconstructed using a preset reconstruction model to obtain a target high-resolution video; wherein the preset reconstruction model includes at least: a channel attention mechanism module with a gating mechanism, the gating mechanism carrying target feature weights corresponding to each feature information, each target feature weight being determined through adaptive training; the channel attention mechanism module is used to extract features from the low-resolution video to be reconstructed, obtain each feature information, and reconstruct the low-resolution video to be reconstructed based on each feature information according to the corresponding target feature weights to obtain the target high-resolution video. Since the channel attention mechanism module in this application may include a gating mechanism, and the gating mechanism carries target feature weights corresponding to each feature information, and the target feature weights are determined through adaptive training, reconstruction can be performed according to the target feature weights corresponding to each feature information through the gating mechanism during use, thereby reducing interference from information invalid to the current task and improving the reconstruction effect. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application;

[0046] Figure 2 This is a flowchart illustrating the first embodiment of the video super-resolution reconstruction method of this application;

[0047] Figure 3 This is a schematic diagram of the structure of the preset reconstruction model in the first embodiment of the video super-resolution reconstruction method of this application;

[0048] Figure 4 This is a schematic diagram of the structure of the dense module and the network module in the first embodiment of the video super-resolution reconstruction method of this application;

[0049] Figure 5 This is a flowchart illustrating the second embodiment of the video super-resolution reconstruction method of this application;

[0050] Figure 6This is a schematic diagram of the structure of the preset reconstruction model in the second embodiment of the video super-resolution reconstruction method of this application;

[0051] Figure 7 This is a schematic diagram of the structure of the nonlocal residual module in the second embodiment of the video super-resolution reconstruction method of this application;

[0052] Figure 8 This is a schematic diagram of the structure of the non-local attention module in the second embodiment of the video super-resolution reconstruction method of this application;

[0053] Figure 9 This is a schematic diagram of the subpixel convolution module in the second embodiment of the video super-resolution reconstruction method of this application;

[0054] Figure 10 This is a flowchart illustrating the third embodiment of the video super-resolution reconstruction method of this application;

[0055] Figure 11 This is a schematic diagram of the structure of the preset reconstruction model in the third embodiment of the video super-resolution reconstruction method of this application;

[0056] Figure 12 This is a schematic diagram of the channel attention mechanism module in the third embodiment of the video super-resolution reconstruction method of this application;

[0057] Figure 13 This is a schematic diagram of the structure of the multi-scale channel attention module in the third embodiment of the video super-resolution reconstruction method of this application;

[0058] Figure 14 This is a schematic diagram of the gating mechanism in the third embodiment of the video super-resolution reconstruction method of this application;

[0059] Figure 15 This is a structural block diagram of the first embodiment of the video super-resolution reconstruction device of this application.

[0060] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0061] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0062] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application.

[0063] like Figure 1As shown, the hardware operating environment involved in this application embodiment is a wearable device, which may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize communication between these components. The user interface 1003 may include a display screen, and optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. In this application, the wired interface of the user interface 1003 may be a USB interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0064] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0065] like Figure 1 As shown, the memory 1005, which is identified as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a video super-resolution reconstruction program.

[0066] exist Figure 1 In the device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the user equipment; the device calls the video super-resolution reconstruction program stored in the memory 1005 through the processor 1001 and executes the video super-resolution reconstruction method provided in the embodiments of this application.

[0067] It should be noted that currently, when performing super-resolution (SR) reconstruction on low-resolution (LR) videos to obtain high-resolution (HR) videos, different feature information exists during feature extraction, such as feature information related to color features, feature information related to spatial features, and feature information related to hierarchical features.

[0068] Current methods treat these features equally. However, not all features are relevant to the current task. For example, color features may be more important for color restoration, while spatial and hierarchical features may be more important for resolution reconstruction. Therefore, the current equal treatment approach may allow irrelevant information to interfere with the current task, resulting in poor reconstruction results.

[0069] Therefore, to address the aforementioned shortcomings, this embodiment provides a video super-resolution reconstruction method. The channel attention mechanism module may include a gating mechanism, which carries target feature weights corresponding to each feature information. These target feature weights are determined through adaptive training. During use, reconstruction can be performed according to the target feature weights corresponding to each feature information through the gating mechanism, thereby reducing interference from information invalid for the current task and improving the reconstruction effect.

[0070] For ease of understanding, the following is combined with Figures 2 to 15 The video super-resolution reconstruction method provided in the embodiments of this application will be described in detail.

[0071] This application provides a video super-resolution reconstruction method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the video super-resolution reconstruction method of this application.

[0072] In this embodiment, the video super-resolution reconstruction method includes the following steps:

[0073] Step S10: Obtain the low-resolution video to be reconstructed.

[0074] It is understood that the method of this embodiment can be applied to scenarios where wearable devices perform super-resolution reconstruction, such as AR glasses, or other devices. This embodiment does not limit this application. The executing entity of this embodiment can be a super-resolution reconstruction device with functions of super-resolution reconstruction, data processing, network communication, and program execution, such as AR glasses, or other devices capable of performing the same or similar functions. The following describes this embodiment and the following embodiments in detail using the aforementioned AR glasses (hereinafter referred to as the device).

[0075] It should be understood that the aforementioned low-resolution video to be reconstructed can be the video that the aforementioned device needs to display. Since AR glasses can generally be set to a display resolution at the factory, such as 1080P, and will display at that resolution in subsequent use, the resolution of the aforementioned low-resolution video to be reconstructed can be the set resolution. However, to improve the visual effect, the low-resolution video to be reconstructed can be reconstructed to increase its resolution, for example, to 2K.

[0076] Step S20: Reconstruct the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video.

[0077] It is important to emphasize that the aforementioned preset reconstruction mode can be a model used to upscale low-resolution videos to be reconstructed. However, traditional reconstruction models generally treat all feature information equally when extracting various features, such as those corresponding to color features, spatial features, and hierarchical features, resulting in poor reconstruction performance. Therefore, the preset reconstruction model in this embodiment includes at least: a channel attention mechanism module with a gating mechanism. The gating mechanism contains target feature weights corresponding to each feature information. These target feature weights are determined through adaptive training. The channel attention mechanism module is used to extract features from the low-resolution video to be reconstructed, obtain various feature information, and reconstruct the low-resolution video based on each feature information according to the corresponding target feature weights to obtain a target high-resolution video.

[0078] Understandably, the aforementioned gating mechanism allows for the setting of corresponding gates for each feature information, and the opening degree of these gates is controlled by the target feature weights corresponding to each feature information, thereby determining the degree of participation of that feature information in the current task. For example, in a color restoration task, the target feature weight of the gate corresponding to the color feature information can be increased, while the target feature weights of the gates corresponding to the spatial feature information and the hierarchical feature information can be decreased, thus ensuring the quality of color restoration.

[0079] Before going into detail, this embodiment should also explain that, traditionally, super-resolution reconstruction can be achieved through a hybrid model of convolutional neural network (CNN) and Transformer model. That is, CNN is used to extract local features from low-resolution video, and Transformer model is used to extract global features from low-resolution video. Then, the local and global features are fused and decoded to obtain high-resolution video, thus achieving super-resolution reconstruction.

[0080] However, when this approach is applied to augmented reality (AR) glasses, the limited computing resources of AR glasses, coupled with the large number of parameters required by the CNN+Transformer method, result in slow inference speed and a poor user experience for AR glasses.

[0081] Therefore, in this embodiment, the above-mentioned preset reconstruction model further includes at least: a dense module with a first network and a second network, wherein the first network is used to perform depthwise separable convolution operation on the low-resolution video to be reconstructed using an inverted residual method to obtain local features, and the second network is used to extract global features from the low-resolution video to be reconstructed using a self-attention mechanism, and reconstruct the low-resolution video to be reconstructed based on the local features and the global features.

[0082] The aforementioned first network can be a network with depthwise separable convolution and inverted residual modules. In conventional convolution operations, each convolution kernel can simultaneously process multiple input channels of the input tensor and mix them into a single output channel. However, the depthwise separable convolution in this embodiment decomposes this process into two steps: depthwise convolution and pointwise convolution. Depthwise convolution performs a convolution operation on each input channel using its own convolution kernel, without cross-channel mixing. Pointwise convolution mixes the output channels of the depthwise convolution using a 1x1 convolution kernel to generate the final output channel. This reduces the number of convolution kernels and the computational cost of convolution operations.

[0083] Traditional ResNet residual modules typically contain two convolutional layers, where the output dimension of the second convolutional layer is the same as the input dimension, allowing the input to "jump" directly to the output, forming a residual connection. In this embodiment, however, the pointwise convolutional layer in the inverted residual module is placed first, followed by the depthwise convolutional layer. That is, the number of channels is adjusted first, and then this number is maintained in the depthwise convolution, thereby further reducing the number of parameters and computational cost.

[0084] Therefore, the first network in this embodiment may include depthwise separable convolution and inverted residual modules. In practical use, after the device acquires the low-resolution video to be reconstructed, it can be input into the preset reconstruction model. The first network can extract local features from the low-resolution video to be reconstructed using depthwise separable convolution and inverted residual techniques, thereby reducing the number of parameters required. The aforementioned local features can be features within a small range of the low-resolution video to be reconstructed, which can be used to reflect details such as edges and textures. This embodiment does not impose any specific limitations on this.

[0085] The second network described above can be a network with a self-attention mechanism. This self-attention mechanism can capture the relationship between any two elements and extract information about each location in the low-resolution video to be reconstructed. Furthermore, this embodiment can learn complex feature information at each location through a multi-attention mechanism to obtain global features. These global features can be overall information, such as the overall layout, relationships between objects, and context; this embodiment does not impose any limitations on this. Since the parameters in the self-attention mechanism can be shared, the number of parameters can be further reduced. Therefore, in practical use, after the first network extracts local features, the second network can utilize the self-attention mechanism to extract global features from the low-resolution video to be reconstructed, thus obtaining global features.

[0086] After obtaining the aforementioned local and global features, the dense module can use the learned local and global features to perform super-resolution reconstruction on the low-resolution video to be reconstructed, thereby obtaining an initial high-resolution video, which is then transmitted to the aforementioned channel attention mechanism module. The channel attention mechanism module then extracts features from the initial high-resolution video to obtain the aforementioned feature information.

[0087] Therefore, the above steps of reconstructing the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video include:

[0088] The first network performs depthwise separable convolution on the low-resolution video to be reconstructed using an inverted residual approach to obtain local features. The second network uses a self-attention mechanism to extract global features from the low-resolution video to be reconstructed, and reconstructs the low-resolution video to be reconstructed based on the local and global features to obtain an initial high-resolution video. The channel attention mechanism module extracts features from the initial high-resolution video to obtain various feature information, and reconstructs the low-resolution video to be reconstructed based on each feature information according to the corresponding target feature weights to obtain the target high-resolution video.

[0089] Furthermore, to construct the aforementioned first and second networks, in this embodiment, the first network may include a MobileNet V3 network, and the second network may include a MobileVit network. That is, in this embodiment, the dense module can be constructed using a dense block network architecture, combining MobileVit's attention mechanism, a MobileNet V3 convolutional network with a residual network.

[0090] This dense module, through the aforementioned structure, fully leverages the advantages of attention mechanisms to obtain global information from keyframe images. It utilizes the self-attention mechanism of the MobileVit network to acquire location information for each frame in the low-resolution video to be reconstructed, and employs a multi-attention mechanism to learn complex feature information at each location in the image. Simultaneously, it can identify dependencies between different regions, thereby performing global feature calibration. It can learn the temporal continuity and spatial correlation in the video, completing spatiotemporal consistency calibration and ensuring smooth transitions and natural connections between frames.

[0091] This embodiment utilizes the MobileNet V3 network in the dense module to extract local features using depthwise separable convolutions and inverted residual modules, and then extracts global features using the MobileVit attention mechanism. Finally, it reconstructs the target high-resolution video using the obtained local and global features. Compared to existing methods employing CNN+Transformer, which require a large number of parameters, this embodiment reduces the number of parameters needed due to the smaller parameter count required by self-attention mechanisms, depthwise separable convolutions, and inverted residuals, thus improving inference speed and enhancing the user experience.

[0092] Meanwhile, this embodiment uses fewer parameters, thus reducing the computational load and consequently lowering the power consumption of the AR glasses. Furthermore, because this embodiment performs super-resolution reconstruction, compared to AR glasses without super-resolution reconstruction, it solves the video quality degradation problem caused by user movement while wearing the glasses, further improving the user experience.

[0093] Secondly, since this embodiment can achieve super-resolution reconstruction within AR glasses, compared to existing methods that rely on improving hardware performance (e.g., using high-definition cameras and high-performance light sensors), this embodiment can reduce the cost of hardware.

[0094] Furthermore, in order to improve the quality of the low-resolution video to be reconstructed, in this embodiment, the above-mentioned preset reconstruction model further includes: an image preprocessing module, which is connected to the first network;

[0095] The step of reconstructing the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video includes:

[0096] The image preprocessing module preprocesses the low-resolution video to be reconstructed, including at least one of keyframe extraction, bad frame removal, resolution adjustment, and data augmentation. The first network extracts local features from the preprocessed low-resolution video. The second network extracts global features from the preprocessed low-resolution video, and the video is reconstructed based on the local and global features to obtain an initial high-resolution video. The channel attention mechanism module extracts features from the initial high-resolution video to obtain feature information, and the video is reconstructed based on the corresponding target feature weights according to the feature information to obtain a target high-resolution video.

[0097] It should be noted that, for ease of understanding, refer to Figure 3 To explain, Figure 3 This is a schematic diagram of the structure of the preset reconstruction model in the first embodiment of the video super-resolution reconstruction method of this application. Figure 3 As shown, in this embodiment, after obtaining the low-resolution video to be reconstructed (i.e., Figure 3 After processing low- to medium-quality videos, they can be input into the image preprocessing module of the preset reconstruction model.

[0098] In this image preprocessing module, keyframes can be extracted from the low-resolution video to be reconstructed at preset intervals (i.e., ...). Figure 3 (Frame extraction). For example, if the preset interval is 3 frames, then images at intervals of 3 frames in the low-resolution video to be reconstructed can be used as keyframe images (i.e., ... Figure 3 The keyframes in the low-resolution video to be reconstructed are extracted. Alternatively, keyframe extraction can be omitted, and all frames in the low-resolution video to be reconstructed can be used as keyframes for further processing. Whether to extract keyframes and the preset interval frame number when extracting them can be set according to the actual situation, and this embodiment does not impose any restrictions on this.

[0099] After extracting the keyframes, some images may be blurry or of low quality. Therefore, the image preprocessing module described above can remove bad frames from the extracted low-resolution video to be reconstructed (i.e., ...). Figure 3 Remove bad frames from the video (i.e., remove frames that do not meet the requirements) to obtain the low-resolution video to be reconstructed (i.e., the video itself). Figure 3 (The final keyframe in the middle);

[0100] After removing bad frames, in order to unify all the obtained frame images, the aforementioned image preprocessing module can also adjust the resolution of each frame image in the low-resolution video to be reconstructed after removal, adjusting it to the consistent resolution required subsequently (e.g., 1080P). Figure 3 (Medium resolution adjustment);

[0101] After adjusting the resolution, data augmentation (i.e., data enhancement) can be performed on the adjusted low-resolution video to be reconstructed to enrich the data. Figure 3 Data augmentation can be performed in various ways, including but not limited to spatial transformation, color transformation, cropping and flipping, adding Gaussian noise and Gaussian blur, etc. This embodiment does not limit these methods.

[0102] After data augmentation, the augmented low-resolution video to be reconstructed can be transmitted to the dense module (i.e., Figure 3 The MobileNet V3 network (comprising the MobileVit attention mechanism and MobileNet V3 convolutional blocks in the dense module) performs local feature extraction, which is then transmitted to the MobileVit network for global feature extraction. Based on the local and global features, the low-resolution video to be reconstructed is then reconstructed. However, to improve inference speed before input, in this embodiment, a segmentation module can be set after the image preprocessing module and before the dense module. Figure 3 (not shown in the image) and a first attention mechanism layer (i.e. Figure 3 The attention mechanism convolutional layer before the dense module (hereinafter, all attention mechanism layers are attention mechanism convolutional layers) means that the preset reconstruction model also includes: a segmentation module and a first attention mechanism layer, the first attention mechanism layer contains the second network, and the first attention mechanism layer is connected to the segmentation module and the dense module respectively.

[0103] After obtaining the preprocessed low-resolution video to be reconstructed, the segmentation module can first adjust the low-resolution video to make each frame image of it a preset size. The specific preset size is not limited in this embodiment. Then, the adjusted frame images are divided into several image blocks. The size of the image block can be 3×3, or other sizes. This embodiment can refer to the Vision Transformer network architecture and uses 3×3 for illustration.

[0104] After obtaining several image blocks, they can be sorted according to their position in the adjusted frame image to flatten the two-dimensional low-resolution image blocks into a one-dimensional sequence. Each image block is treated as a separate token, and an initial feature vector, denoted as I, is obtained from the pixel values ​​of each flattened image block through a linear transformation. LRThen, position embeddings are added to each flattened image patch (i.e., each token) to provide the position information of each image patch in the original image (i.e., the frame image in the low-resolution video to be reconstructed), and the final target feature vector containing position information is obtained, denoted as I. z This process can be obtained through the following first formula (i.e., Formula 1):

[0105] I z =WI LR Formula 1;

[0106] Among them I z Then it is the vector containing location information mentioned above, I LR Then, the eigenvectors are as described above, and W is the learnable weight matrix.

[0107] After obtaining the target feature vector I containing the location information mentioned above. z Then, the target feature vector can be transmitted to the first attention mechanism layer, where the MobileVit network processes the target feature vector I. z Perform self-attention calculation to complete feature extraction, obtain the first extraction result, and further improve the subsequent results.

[0108] In this first attention mechanism layer, the query (Q), key (K), and value (V) of each token can be obtained by multiplying them with the corresponding learning weight matrix. Specifically, this can be obtained through the following second formula (i.e., Formula 2):

[0109] Q = I z W Q K = I z W K V = I z W V Formula 2;

[0110] Where Q represents the query, W... Q To query the corresponding learning weight matrix, K is the key, W K Let V be the learning weight matrix corresponding to the key, and W be the value. V The value is the learning weight matrix corresponding to the value.

[0111] After obtaining the query (Q), key (K), and value (V), an attention score can be calculated to measure the similarity between the query Q and each key K. Then, a softmax function is used for normalization, followed by a weighted summation to obtain the feature output by the first attention mechanism layer (i.e., the first extraction result mentioned above), denoted as F. s Specifically, this can be achieved through the following third formula (i.e., formula 3) and fourth formula (i.e., formula ...).

[0112] Equation 4) yields:

[0113]

[0114] F s =S w Formula 4;

[0115] Where score is the normalized result, K T d is the transpose of the key, and I is the transpose of the key. z The dimension, S w For the attention weight (i.e., score) corresponding to value V, F s This refers to the features output by the convolutional layer of this attention mechanism (i.e., the weighted summation result).

[0116] The first extraction result can then be input into the dense module for feature extraction. To facilitate dense connections, it is combined with... Figure 3 , Figure 3 The medium-density module may include at least one set of sequentially connected network modules and a first feature cascade layer (i.e., Figure 3 The network module, the first feature cascade layer, and the second attention mechanism layer are connected in a dense connection manner, that is, each first feature cascade layer is not only connected to the network module in its group, but also connected to the input of the network modules of all previous groups.

[0117] The network module can include a convolutional layer and an attention mechanism layer. The convolutional layer can use a first network, namely the MobileNet V3 convolutional network mentioned above, and the attention mechanism layer can use a second network, namely the MobileVit attention mechanism mentioned above. Therefore, local information of the input frame image can be extracted through MobileNet V3, and information of each pixel and complex feature information of each pixel can be extracted through MobileVit. To improve the performance, the network module in this embodiment can also include multiple convolutional layers and multiple attention mechanism layers.

[0118] Reference Figure 4 , Figure 4 This is a schematic diagram of the structure of the dense module and the network module in the first embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 4 As shown, in this embodiment, the dense module can be described using three sequentially connected network modules, a first feature cascade layer, and a second attention mechanism layer.

[0119] The network module may include the first network described above. The first extraction result F is obtained after passing through the first attention mechanism layer. sThen, the first network module can perform feature extraction on the first extraction result, obtain local features, and then transmit them together with the first extraction result to the first feature concatenation layer. Next, feature concatenation is performed on the global features and local features in the extracted feature map (i.e.,...). Figure 3 The first concatenation result is obtained by concatenating features along the channel dimension. This concatenation can enhance the richness of the features. The first concatenation result is then processed by the second attention mechanism layer using the MobileVit network for feature extraction to obtain global features. Since the dense modules in this implementation adopt a dense connection method, when performing feature concatenation in the subsequent first feature concatenation layer, the local features output by the network module of this layer, the global features output by the previous second attention mechanism layer, and the first extraction result F input by the first network module can be used. s The features are stitched together to further extract and refine the features; finally, video super-resolution reconstruction can be performed based on the extracted local and global features to obtain an initial high-resolution video.

[0120] The specific process described above can be obtained through the following fifth formula (i.e., formula 5):

[0121] F D =AC n (Cat(B1, B2, ..., B) n )) Formula 5;

[0122] Among them B n Here, AC represents the local features extracted by the nth network module, and Cat represents the spatial feature concatenation operation (i.e., the feature concatenation mentioned above). n For the feature extraction operation of the second attention mechanism layer mentioned above, F D These are the global features mentioned above. This operation, which is easy to implement in deep learning, can complement the extracted deep features and combine them to improve the generalization performance of the model.

[0123] In order to obtain the above local feature B n , continue as Figure 4 As shown, in this embodiment, the network module includes: an extraction module, a second feature cascade layer, and a third convolutional layer. The third convolutional layer contains the first network. The second feature cascade layer is connected to the extraction module and the third convolutional layer, respectively. The third convolutional layer is connected to the first feature cascade layer.

[0124] In the process of obtaining local features by using the network module to extract features from the first extraction result or the global features output by the previous second attention mechanism layer through the first network, the extraction module can first extract features from the first extraction result or the global features output by the previous second attention mechanism layer to obtain the second extraction result. Then, the second feature concatenation layer can concatenate the features of the second extraction result to obtain the second concatenation result. Finally, the third convolutional layer can use the first network to extract features from the second concatenation result to obtain local features.

[0125] To further improve reasoning speed, such as Figure 4 As shown, in this embodiment, the extraction module includes at least two extraction units, each of which is connected sequentially and then connected to the second feature cascade layer in a skip connection manner;

[0126] It should be noted that in this embodiment, each extraction unit is connected to the second feature cascade layer using a jump connection method, such as... Figure 4 As shown, this embodiment uses two extraction units for illustration, referred to as the first extraction unit and the second extraction unit respectively. The input of the first extraction unit is connected not only to the second attention mechanism layer of the previous group, but also to the second feature cascade layer in the network module. When it is the first network module, the first extraction unit can also be connected to the first attention mechanism layer. The output of the first extraction unit can be connected to the input of the second extraction unit and the second feature cascade layer. The output of the second feature extraction unit can also be connected to the second feature cascade layer.

[0127] In practical use, the first extraction unit can perform feature extraction on the first extraction result or the global features output by the previous second attention mechanism layer to obtain the second extraction result corresponding to the first extraction unit, and transmit the second extraction result to the second extraction unit and the second feature cascade layer. The second extraction unit can then perform feature extraction on the second extraction result to obtain the second extraction result corresponding to the second extraction unit, and transmit it to the second feature cascade layer. The second feature cascade layer then processes the second extraction result output by the first extraction unit and the first extraction result F. s Alternatively, the global features output by the previous second attention mechanism layer can be concatenated to obtain the second concatenation result.

[0128] If we denote the local features output by the network module as B n Then, for the local feature B output by the above network module n The feature extraction process can be obtained through the following sixth formula (i.e., formula 6):

[0129]

[0130] The above For the 3rd MobileNet V3 convolutional layer in the nth network module (i.e. Figure 4 The third convolutional layer), Cat is the spatial feature concatenation operation (i.e., the second feature cascade layer), X n The input to the nth network module (i.e., the first extraction result or the global features output by the previous second attention mechanism layer) is used. Since this embodiment has two extraction units and one third convolutional layer, there are three feature extraction processes. For X n The second extraction result output after the first feature extraction (i.e., the first extraction unit) is as follows: for The second extraction result is output after the second feature extraction (i.e., the second extraction unit).

[0131] To further improve speed, continue to refer to Figure 4 In this embodiment, the extraction unit may include a third attention mechanism layer with a second network and a fourth convolutional layer with a first network. The third attention mechanism layer and the fourth convolutional layer are connected in sequence, and the second convolutional layer is also connected to the second feature cascade layer.

[0132] It should be noted that when the third attention mechanism layer is the first extraction unit of the first network module, feature extraction can be performed on the first extraction result output by the first attention mechanism layer. When the third attention mechanism layer is not the first extraction unit in the first network module, feature extraction can be performed on the global features obtained by the previous second attention mechanism layer. When the third attention mechanism layer is not the first extraction unit, feature extraction can be performed on the second extraction result output by the previous extraction unit. Furthermore, in this embodiment, the feature extraction result of the third attention mechanism layer is referred to as the third extraction result.

[0133] After obtaining the third extraction result, the first network is used to extract features from the third extraction result through the fourth convolutional layer to obtain the second extraction result of the extraction unit. This can be specifically expressed by the following seventh formula (i.e., formula 7) and eighth formula (i.e., formula 8):

[0134]

[0135]

[0136] Among the above For the third attention mechanism layer within the first extraction unit of the nth network module, the above For the fourth convolutional layer within the first extraction unit of the nth network module, the above For the third attention mechanism layer within the second extraction unit of the nth network module, the above This is the fourth convolutional layer within the second extraction unit of the nth network module. F can be obtained using the first to eighth formulas described above. D .

[0137] Secondly, continue to refer to Figure 3 After the dense module, upsampling and depth sampling can be added. Upsampling can include a deconvolution layer, which can also be composed of MobileNet V3 convolutional layers. Upsampling can further extract and refine the enlarged features. Depth sampling can be used to restore the feature map to the desired size. These two operations can enlarge the residual image before image reconstruction and linear feature addition (i.e.,...) Figure 3 (The linear features are added together) to obtain the initial high-resolution video.

[0138] When adding linear features, bicubic interpolation can be used, which can be expressed by the following ninth formula (i.e., formula 9):

[0139] I SR =Bic(I SR )+OC 0 (DeC(F D )) Formula 9;

[0140] Among them I SR For the initial high-resolution video, Bic is bicubic interpolation, and OC is... 0 For deep sampling operations, DeC is the aforementioned deconvolution layer.

[0141] It needs to be emphasized that, such as Figure 4 As shown, in this embodiment, the output of each fourth convolutional layer in the network module is directly connected to the subsequent second feature cascade layer. In the dense module, the input of each network module is directly connected to all subsequent first feature cascade layers. This means that the input of each first feature cascade layer is the union of the outputs of all previous layers. This design allows the network to reuse the features of previous layers, instead of simply passing the features to the next layer as in traditional networks. This connection method can be called dense connection, which further reduces the amount of computation.

[0142] It should also be emphasized that when the preset reconstruction model is in the learning phase, its input includes not only the low-resolution video to be reconstructed, but also its corresponding high-resolution video. This video undergoes preprocessing operations by the aforementioned image preprocessing module before being transmitted to the dense module for feature extraction and learning. This preprocessing process corresponds one-to-one with the preprocessing process for the low-resolution video described above, and will not be elaborated upon in this embodiment. When adjusting the resolution, the high-resolution video can be adjusted to 2K, but other resolutions are also possible; this embodiment does not impose any limitations on this.

[0143] During the learning phase, the dense module learns image detail information to perform detailed super-resolution reconstruction of low-resolution keyframe images. Furthermore, this dense module can better leverage the long-term dependency modeling capabilities of the attention mechanism, helping it better capture dependencies between different locations in the image and learn more inter-block information. This avoids gradient propagation problems, better models long-term dependencies, and effectively improves the network's reconstruction performance by learning more inter-block information. In summary, this dense block network learns the mapping relationship and feature map information from low-resolution keyframes to high-resolution keyframes.

[0144] Furthermore, continue as Figure 4 As shown, in this embodiment, skip connections are used between each extraction unit and the second feature cascade layer. Since edge contour information and high-frequency texture information are particularly important during image restoration, treating the features extracted by the last feature extraction unit and the inter-layer features transmitted by the skip connections equally might lead the model to focus only on low-frequency information (such as color and lighting) while ignoring high-frequency information, resulting in suboptimal high-resolution video restoration. Therefore, in this embodiment, the second feature cascade layer may include a target loss function. This target loss function is constructed by setting a first target weight for the first loss function of the extraction unit and a second target weight for the second loss function of the skip connections. The first and second target weights are determined through adaptive training.

[0145] It should be noted that the first loss function of the extraction unit mentioned above can be the loss function used by the extraction unit during training. For ease of explanation later, since the extraction unit may include a third attention mechanism layer and a fourth convolutional layer, the first loss function corresponding to the extraction unit can be denoted as L. MobileNetV3 +L MobileVit L MibileNetV3 L is the loss function of the third attention mechanism layer. MobileVit This is the loss function for the fourth convolutional layer.

[0146] The second loss function for the aforementioned skip connections can be the loss function corresponding to the skip connections between each extraction unit and the second feature cascade layer, and can be denoted as L. Skip .

[0147] This embodiment can incorporate the aforementioned first target weight and second target weight to treat the image features extracted by the convolutional layer and the attention mechanism layer equally, while enriching the inter-layer information extracted by the residual network and the convolutional layer. The first target weight can be denoted as α, and the second target weight as β. Both weights are independent and learnable parameters, and they borrow from the task-oriented attention mechanism concept, adaptively fluctuating between [0, 1] in conjunction with the following target loss function. If the target loss function is denoted as L, then the specific target loss function is: L = α(L MobileNetV3 +L MobileVit )+βL Skip .

[0148] Therefore, during training, training data can be acquired first. This training data may include the low-resolution video to be reconstructed and its corresponding high-resolution video. The data is transmitted to the dense module in the initial reconstruction model for adaptive training. In the dense module, the second feature cascade layer can continuously adjust the first initial weight α and the second initial weight β according to the training results to obtain the first target weight α and the second target weight β. Thus, the target loss function mentioned above can be obtained, and the initial reconstruction model can also be transformed into the preset reconstruction model mentioned above.

[0149] During adaptive training, when the detailed and contextual information learned by skip connections at different levels is less important, while the image feature information transmitted by the attention mechanism and convolution is more important, β can be decreased and α increased; conversely, α can be decreased and β increased. This task-oriented mechanism ensures that in the MobileVit attention-based convolutional network, the hierarchical feature information learned by each residual connection positively impacts the performance of the video super-resolution network in learning low-resolution image features, further improving reconstruction quality.

[0150] In this embodiment, the channel attention mechanism module may include a gating mechanism. This gating mechanism contains target feature weights corresponding to each feature information, and these target feature weights are determined through adaptive training. During use, the gating mechanism can be used to reconstruct the data according to the target feature weights corresponding to each feature information, thereby reducing interference from information invalid for the current task and improving the reconstruction effect.

[0151] refer to Figure 5 , Figure 5 This is a flowchart illustrating the second embodiment of the video super-resolution reconstruction method of this application.

[0152] To further capture global information from frame images, learn internal structure and texture information, thereby ensuring that the high-resolution image recovered from the low-resolution frame image has more natural brightness, clearer image, and more realistic edge contours, such as... Figure 5As shown, in this embodiment, the preset reconstruction model mentioned above further includes: a nonlocal residual module, which is connected to the second network in the dense module;

[0153] The step of extracting features from the initial high-resolution video using the channel attention mechanism module to obtain feature information includes:

[0154] Step S21: Perform detail adjustments on the initial high-resolution video using the non-local residual module;

[0155] Step S22: Extract features from the adjusted initial high-resolution video using the channel attention mechanism module to obtain various feature information.

[0156] For ease of understanding, please refer to Figure 6 To explain, Figure 6 This is a schematic diagram of the structure of the preset reconstruction model in the second embodiment of the video super-resolution reconstruction method of this application. Figure 6 As shown, after reconstructing local and global features in the second network of the dense module, an initial high-resolution video can be obtained, which is then transmitted to the non-local residual module (i.e., Figure 6 In the nonlocal residual network based on the MobileVit attention mechanism, the aforementioned nonlocal residual module can be a nonlocal residual network based on the MobileVit attention mechanism. Specifically, it can be located after the linear summation of the features. It can capture the global information of the feature map and use structural self-similarity loss to determine the similarity between different positions in the feature map, learn the internal structure and texture information of the image, and at the same time learn the temporal continuity and spatial correlation between video frames. It focuses on the global information in the feature map, and then adaptively adjusts the feature map of the key frame, performs detailed adjustments and optimizations to the feature map, and performs global calibration, thereby ensuring that the high-resolution image recovered from the low-resolution key frame image has more natural brightness, clearer image, and more realistic edge contours.

[0157] Furthermore, in order to achieve the above-mentioned detailed adjustments, in this embodiment, the nonlocal residual module includes: a nonlocal attention module and a hierarchical residual module, wherein the nonlocal attention module is connected to the second network and the hierarchical residual module;

[0158] The step of performing detail adjustments on the initial high-resolution video using the non-local residual module includes:

[0159] The initial high-resolution video is globally calibrated using the non-local attention module; the calibrated initial high-resolution video is then hierarchically fused using the hierarchical residual module.

[0160] like Figure 6As shown, the nonlocal residual module may include a nonlocal attention module based on an attention mechanism (i.e., Figure 6 Non-local modules) and hierarchical residual modules (i.e. Figure 6 The non-local attention module can be located after the linear feature addition and before the hierarchical residual module.

[0161] The aforementioned nonlocal attention module can be a module built based on the nonlocal residual module of MobileVit, and the aforementioned hierarchical residual module can be a module composed of multiple residual connections. In this embodiment, the nonlocal residual module based on MobileVit can be used as the basic unit and integrated into the MobileNet V3 convolutional neural network to form a nonlocal residual module.

[0162] This is because reconstructing a high-quality image requires accurately restoring the structural and textural relationships between different regions.

[0163] By introducing a nonlocal attention mechanism, the network can better learn the semantic information of the input image and the relationships between pixels. The hierarchical residual module utilizes residual connections to optimize feature propagation and gradient flow. By establishing residual connections between different layers, the network can more effectively reuse and fuse feature information from different layers, thereby enhancing the expressive power of features.

[0164] It should be noted that in this embodiment, the non-local attention module and the hierarchical residual module can be iterated multiple times and then integrated with the deep network, so that the network can give full play to the effect of the self-attention mechanism.

[0165] For ease of understanding, please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of the nonlocal residual module in the second embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 7 As shown, the nonlocal residual module in this embodiment may include several nonlocal residual blocks. This embodiment uses four nonlocal residual blocks for illustration, and each nonlocal residual block may include: a nonlocal attention module and a hierarchical residual module.

[0166] In practical use, it can first be achieved through a MobileNet V3 convolutional neural network (i.e., Figure 7 The process of extracting frame images from the reconstructed initial high-resolution video using MobileNet V3 convolutional blocks can be represented by the following formula 10:

[0167] F s1 =f 0 (I LR1 ) Formula 10;

[0168] Among them I LR1For the initial high-resolution video obtained after reconstruction, f 0 For MobileNet V3 convolutional blocks,

[0169] F s1 The extracted frame image;

[0170] Next, a non-local attention module is used for feature extraction to obtain global information and dependencies, completing global calibration. This ensures that the high-resolution image reconstructed from the low-resolution keyframe image has more natural brightness, clearer image, and more realistic edge contours. Then, a hierarchical residual module is used to perform hierarchical fusion of the extracted features. Low-level features help restore local details of the image, while high-level features help maintain the overall structure and semantic information of the image. Finally, the features before and after the convolutional network can be linearly stacked through global skip connections, thereby realizing the processing of the non-local residual network based on the attention mechanism. The multi-layer stacked hierarchical residual module can linearly align the low-dimensional and high-dimensional feature maps of the keyframe, realizing linear mapping of keyframe feature map pairs, which can reduce the speed of reconstructing low-resolution images into high-resolution images and improve the network's generalization performance.

[0171] Furthermore, continue as Figure 7 As shown, in this embodiment, the non-local attention module may include a second network. This second network employs a self-attention mechanism to correlate the extracted frames of the initial high-resolution video, and based on this correlation, global calibration of the initial high-resolution video can be performed. The specific process is described in [reference needed]. Figure 8 , Figure 8 This is a schematic diagram of the structure of the non-local attention module in the second embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 8 As shown, the nonlocal residual module in this embodiment may include several nonlocal residual blocks. This embodiment uses four blocks for illustration, and they are connected in sequence.

[0172] In practical applications, the extracted initial high-resolution video frame images are first subjected to dimensionality reduction using a non-local attention module. Let's assume the input frame image size to the non-local attention module is M×N, and the number of channels is C, denoted as X. M ×N×C (Right now Figure 8 In the middle (X), dimensionality reduction is performed using 1×1 convolution, reducing its feature dimension to X. M×N×C / 2 This effectively reduces the computational burden;

[0173] Next, the self-similarity of the dimensionality-reduced feature maps is calculated using a structural similarity function, which can be...

[0174]

[0175] In this process, a 3×3 MobileNet V3 convolutional layer is first used to extract features from the dimensionality-reduced frame image to obtain... as well as

[0176] To facilitate computation and enable the model to learn the dependencies between different regions within the feature map, an attention-weighted aggregation strategy is used to convert the obtained feature map information into a matrix. and The self-similarity matrix of the frame image is then obtained by transposing and multiplying. Then, with the nonlocal transformation coefficients w(x) j Multiply;

[0177] Simultaneously, a 1×1 MobileNet V3 convolutional network (i.e., ...) is used in the non-local attention module. Figure 8 The algorithm performs calculations on each pixel block in the frame image to achieve cross-channel information integration and interaction, enhance the non-linear characteristics of the network, and generate a weight matrix containing information on the relationships between pixels at different locations in the frame image. N represents the number of pixels in the frame image used by the non-local attention module. Next, let z... s With z w Multiplying two matrices yields a nonlocal matrix.

[0178] After converting the nonlocal matrix into a 3D frame image, it is then subjected to a 1×1 convolution (i.e., Figure 8 (h)

[0179] Obtain frame image Z containing self-similarity information M×N×C / 2 Finally, let X M×N×C / 2 With Z M×N×C / 2 Adding corresponding elements, corresponding to the initial channel descent, achieves global calibration. Then, using a 1×1 upscaling method, the feature map channels are increased to obtain the final result Y. M×N×C The convolution process in this step integrates the MobileNet V3 convolutional modules using grouped convolution. Grouped convolution significantly reduces the number of parameters by dividing the input feature map into multiple groups and performing convolution operations independently within each group. This makes the model more lightweight, reduces computational resource consumption, and helps prevent overfitting. Grouped convolution is a sparse operation, making it easier for the model to learn important feature representations during training, providing a certain degree of regularization, reducing computational complexity, and improving the model's inference speed.

[0180] After obtaining the initial high-resolution video Y after calibration M×N×CThen, it can be transmitted to the hierarchical residual module, which can be configured with several residual modules. In this embodiment, eight residual modules are used for illustration, that is, each residual module is connected to the non-local attention module in a network cascade manner; the residual module can include the first network and the second network mentioned above.

[0181] In practical use, the calibrated initial high-resolution video can be grouped first. In this embodiment, it can be grouped according to paired frame images and divided into 4 groups for explanation. Then, it is transmitted to the first network and the second network for feature extraction, and hierarchical fusion is performed based on the corresponding extraction results. The non-local residual module can adopt a global residual connection method, that is, the input of the first non-local attention module is connected to the output of the last non-local residual block. The above hierarchical fusion process can be expressed by the following formula 11:

[0182] F D1 =B 4 (B 3 (B 2 (B 1 (F s1 )))) Formula 11;

[0183] Among them B i (i = 1, 2, 3, 4) represents the i-th nonlocal residual block, F D1 This is the result of hierarchical fusion.

[0184] It needs to be emphasized that, continuing as Figure 6 as well as Figure 7 As shown, in order to generate the final super-resolution image, the nonlocal residual module in this embodiment does not use the traditional deconvolution method to enlarge the image.

[0185] While deconvolution can magnify images, its computation process can be complex and inefficient.

[0186] Therefore, this embodiment uses the subpixel convolution method in ESPCN to enlarge the image. That is, after the hierarchical fusion, a subpixel convolution module can be set up. This subpixel convolution module can be connected to the hierarchical residual module of the last non-local residual block. After obtaining the fusion result, the fusion result can be enlarged by the above-mentioned subpixel convolution module. Then, the enlarged initial high-resolution video is transmitted to the channel attention mechanism module. The channel attention mechanism module performs feature extraction on the enlarged initial high-resolution video to obtain various feature information.

[0187] The subpixel convolutional layer in this subpixel convolution module upsamples the low-resolution image, restoring image details, rearranging the output of the convolutional layer, and reducing image blurring. The enlarged initial high-resolution video can then be used as the target high-resolution video.

[0188] Specifically, refer to Figure 9 , Figure 9 This is a schematic diagram of the subpixel convolution module process in the second embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 9 As shown, the sub-pixel convolution module described above can rearrange the fusion result into an r×r matrix (i.e., Figure 9 W×H×r 2 C is processed by subpixel convolution to obtain rW×rH×C), where r is the parameter of the subpixel convolution layer, representing the number of subpixels occupied by each pixel. Each element in this matrix represents a pixel in the high-resolution image. The fusion result is upsampled through the subpixel convolution layer. Simultaneously, the subpixel convolution layer can recover the edge loss caused by the convolution operation, restoring edge information to the higher-resolution image. Furthermore, the subpixel convolution layer increases the expressive power of features by expanding the number of channels in the input feature map, allowing the model to learn on higher-resolution feature maps, thereby improving the model's performance and accuracy. In this embodiment, eight subpixel convolution modules can be set, but other numbers are also possible; this embodiment does not impose any limitations on this.

[0189] It should also be emphasized that the optimization function of the nonlocal residual module in this embodiment can be represented by the mean absolute error, which can be specifically expressed by the following formula 12:

[0190]

[0191] Among them, the above L SR For the target high-resolution video, N is the number of pixels in the fused frame image. For the i-th pixel in the frame image of the initial high-resolution video obtained after reconstruction, Let be the i-th pixel in the frame image of the fused result, and 1 be the L1 loss function.

[0192] Furthermore, considering that AR glasses generally display 3D video with a certain depth information, therefore, continuing as follows... Figure 6 and Figure 7 As shown, in this embodiment, the dense module further includes a disparity map generation module, and the nonlocal residual module further includes a disparity map refinement and regression module. The disparity map generation module is connected to the second network and the nonlocal attention module, and the disparity map refinement and regression module is connected to the hierarchical residual module.

[0193] After the step of extracting global features from the low-resolution video to be reconstructed using the self-attention mechanism through the second network, the method further includes:

[0194] The disparity map generation module extracts depth information from the low-resolution video to be reconstructed to obtain a disparity map.

[0195] After the step of performing hierarchical fusion on the calibrated initial high-resolution video using the hierarchical residual module, the method further includes:

[0196] Based on the disparity map, the initial high-resolution video is linearly summed using the disparity map refinement regression module.

[0197] It should be noted that the disparity map generation module described above can be connected after the last second attention mechanism layer in the dense module and before upsampling. The disparity map refinement and regression module can be connected after the last non-local residual block.

[0198] The disparity map generation module mentioned above may include a regularization extraction layer (i.e. Figure 6 (Regularized extraction) and window matching computation layer (i.e.) Figure 6 (MSD window matching calculation), wherein the above regularization extraction layer can be connected to the last second attention mechanism layer and the window matching calculation layer in the dense module, respectively;

[0199] The regularization extraction layer can use the L1 regularization loss function to extract the regularization information of the frame image output by the last second attention mechanism layer, providing a positive reference for learning and processing the depth information of AR videos. This layer can constrain the complexity of the model and encourage parameter sparsity by adding the sum of the absolute values ​​of the model parameters as a regularization term to the loss function. L1 regularization imposes constraints on the weight parameters of the convolutional layer. Since the L1 regularization term is the sum of the absolute values ​​of the weights, its derivative will jump when the weight is zero (from -1 to 1 or from 1 to -1). This property makes it easier for some weight values ​​to be pushed to zero during the optimization process, thereby achieving weight sparsity.

[0200] The regularization extraction layer can transmit the obtained regularization information to the window matching calculation layer. The window matching calculation layer can include MSD window matching calculation. Based on the regularization information, the network using MSD window matching calculation can generate a low-resolution disparity map (i.e., the aforementioned disparity map). This disparity map can be a difference map of binocular imaging. In the human visual system, due to the distance between the two eyes, the image position of the same object on the retinas of the two eyes is slightly different; this difference is called disparity. Specifically, the window matching calculation layer can first determine a reference window and candidate windows for the frame image based on the regularization information. The reference window can be a window used for reference, selected from a small rectangular region in the frame image of the low-resolution video to be reconstructed before input to the regularization extraction layer. This reference window can contain the features or patterns to be matched. The candidate window can be a window used to compare with the reference window to determine depth information, and can be a window corresponding to a larger search region defined for the received frame image.

[0201] After determining the reference window and candidate windows, depth information can be extracted to generate a disparity map. The specific process is as follows: determine the number of pixels in the window, and determine the pixel values ​​of the pixels in the reference window and the pixel values ​​of the pixels in the candidate windows; determine the similarity score based on the number of pixels in the window, the pixel values ​​of the pixels in the reference window, and the pixel values ​​of the pixels in the candidate windows; generate a disparity map based on the similarity score.

[0202] The number of pixels in the aforementioned window can be the number of pixels contained in the reference window or the candidate window. The pixel value of the pixel in the aforementioned reference window can be the grayscale value or intensity value corresponding to the pixel in the reference window. The pixel value of the pixel in the aforementioned candidate window can be the grayscale value or intensity value corresponding to the pixel in the candidate window.

[0203] The similarity score mentioned above can be considered as the similarity score between the reference window and the candidate window. The specific calculation can be expressed by the following formula 13:

[0204]

[0205] Where MSD is the similarity score, N is the number of pixels in the window, l1(i) is the pixel value of the i-th pixel in the reference window, and l2(i) is the pixel value of the i-th pixel in the candidate window.

[0206] After obtaining the similarity score, the similarity scores of all candidate windows are compared. The window with the smallest MSD value can be selected from the candidate windows. This candidate window can be the window most similar to the reference window and is used as the best matching window.

[0207] After determining the optimal matching window, its position information in the frame image can be determined. The position information and the similarity score of the optimal matching window are output together to generate a disparity map and send it to the disparity refinement regression module. The disparity map generated by the disparity map generation module is a lightweight feature matching disparity map. This disparity map mainly contains the depth information in the keyframes, which prepares the input for the disparity refinement regression module in the next part of the non-local residual network and provides depth-level information reference for subsequent reconstruction of high-resolution video.

[0208] After obtaining the disparity map, it can be used as input to the disparity map refinement regression module. In the disparity map refinement regression module of this embodiment, the unsupervised structural similarity index (SSIM) loss function can be used to determine the image residual between the disparity map and the frame image of the low-resolution video to be reconstructed before the input regularization layer through an upsampling network. Then, the image residual is added to the disparity map to obtain the accurate disparity map. Finally, the obtained accurate disparity map is linearly superimposed with the frame image of the low-resolution video to be reconstructed before the input regularization layer using a global connection. The resulting feature map not only contains the local and global feature information of the key frame image, but also contains the feature information at the depth level. The obtained superimposed initial high-resolution video is then transmitted to the channel attention mechanism module for feature extraction to obtain the feature information.

[0209] Simultaneously refer to Figure 7 In this embodiment, the disparity thinning regression module can also use a global residual connection to directly transmit the output to the subsequent linear addition. This allows the network to learn the residual between the input and output, focusing on the difference between the input and output images, rather than learning the entire image from scratch. This connection method helps prevent the network from losing key information during training, helps ensure the stability of training, and improves the quality of the reconstructed image. Furthermore, the amplification process of the sub-pixel convolution module can be represented by the following formula 14:

[0210] I SR =S UP (F s2 +F D2 ) Formula 14;

[0211] F s2 For the frame image of the first non-local residual block as input, F D2 S represents the image residual output by the disparity refinement regression module. UP For the scaling operation of the subpixel convolution module, I SR This is the initial high-resolution video after overlay.

[0212] Secondly, considering both the speed and quality of the generated image, this embodiment does not use deconvolution to enlarge the image, but instead uses subpixel convolution in ESPCN, which has the advantage of being fast.

[0213] This embodiment can capture global information of the frame image through a nonlocal residual module, learn its internal structure and texture information, thereby ensuring that the high-resolution image recovered from the low-resolution frame image has more natural brightness, clearer image, and more realistic edge contours, further improving image quality. At the same time, through the disparity map generation module and the disparity map refinement regression module, the disparity map is linearly superimposed with the obtained initial high-resolution video to obtain the target high-resolution video, thereby realizing AR video super-resolution reconstruction.

[0214] refer to Figure 10 , Figure 10 This is a flowchart illustrating the third embodiment of the video super-resolution reconstruction method of this application.

[0215] In order to extract various feature information, in this embodiment, the channel attention mechanism module includes: a feature information extraction module and a linear addition layer with a gating mechanism, wherein the linear addition layer is connected to the feature information extraction module;

[0216] The step of reconstructing the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video includes:

[0217] Step S23: Extract features from the low-resolution video to be reconstructed using the feature information extraction module to obtain various feature information;

[0218] Step S24: The low-resolution video to be reconstructed is reconstructed by the linear addition layer based on each feature information and according to the corresponding target feature weights to obtain the target high-resolution video.

[0219] Reference Figure 11 as well as Figure 12 , Figure 11 This is a schematic diagram of the structure of the preset reconstruction model in the third embodiment of the video super-resolution reconstruction method of this application. Figure 12 This is a schematic diagram of the channel attention mechanism module in the third embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 11 and Figure 12As shown, the preset reconstruction model in this embodiment may further include a low-resolution super-resolution module. In this embodiment, the low-resolution video super-resolution module can be connected after the sub-pixel convolution module in the non-local residual module. The low-resolution video super-resolution module can be a module of a deep network formed by continuously stacking a multi-channel attention mechanism module and MobileNetV3 convolutional blocks. In this embodiment, the low-resolution super-resolution module can improve and optimize the channel attention mechanism, which can color grade low-quality videos and improve resolution. The channel attention mechanism can make the network pay more attention to the channels with high information content in the input keyframe image, which can help the network better understand and utilize the feature information in the video frame, thereby improving the quality of super-resolution reconstruction.

[0220] This embodiment can construct a channel attention mechanism module using a channel attention mechanism, which in turn forms a color restoration module for color recovery and a resolution enhancement module for resolution enhancement (super-resolution). For the color restoration part, this module can focus on features such as color, brightness, and contrast; for the resolution enhancement part, this module can focus on details such as texture and edges. Therefore, the two modules differ only in parameters and depth to adapt to different task requirements.

[0221] In this embodiment, the multi-scale channel attention module can effectively distinguish between the feature maps extracted by the MobileVit attention mechanism and the MobileNet V3 convolutional network. It can differentiate and process different feature maps based on their importance and relevance, thereby extracting more accurate information useful for super-resolution reconstruction. Different feature maps contain different information, and some feature maps may be more important and require more attention. Therefore, when fitting nonlinear mappings, each feature map is processed in different ways to better utilize its information, learn richer feature map channel information, and effectively utilize the interdependencies between feature maps to provide prior guidance for network inference.

[0222] In order to extract various feature information, such as Figure 12 As shown, a feature information extraction module can be set in this low-resolution super-resolution module. Figure 12 The feature information extraction module and the linear addition layer with a gating mechanism ( Figure 12 The linear addition layer can be set after the feature information extraction module. The feature information extraction module can extract features from the input frame image to obtain feature information, which may include, but is not limited to, color feature information, spatial feature information and hierarchical feature information. The linear addition layer can perform color restoration or resolution enhancement on the low-resolution video to be reconstructed according to the target feature weights corresponding to each feature information, so as to obtain the target high-resolution video.

[0223] To extract various feature information, the feature information extraction module includes a multi-scale channel attention module, a large residual module, and a small residual module. The multi-scale channel attention module, the large residual module, and the small residual module are all connected to the linear addition layer.

[0224] like Figure 12 As shown, the feature information extraction module can include a multi-scale channel attention module and a small residual module (i.e., ... Figure 12 Small residual modules 1 to small residual modules M) and large residual modules (i.e. Figure 12 The medium-to-large residual modules 1 to large residual modules M, the multi-scale channel attention module, the small residual module, and the large residual module are set in parallel. That is, the inputs of the multi-scale channel attention module, the small residual module, and the large residual module are all connected to the sub-pixel convolution module, and the outputs are all connected to the linear additive layer.

[0225] The steps described above for extracting features from the low-resolution video to be reconstructed using the feature information extraction module to obtain various feature information include:

[0226] The color features of the low-resolution video to be reconstructed are extracted by the multi-scale channel attention module to obtain color feature information; the hierarchical features of the low-resolution video to be reconstructed are extracted by the small residual module to obtain hierarchical feature information; and the spatial features of the low-resolution video to be reconstructed are extracted by the large residual module to obtain spatial feature information.

[0227] The step of reconstructing the low-resolution video to be reconstructed based on each feature information according to the corresponding target feature weights through the linear summation layer to obtain the target high-resolution video includes:

[0228] The linear additive layer reconstructs the low-resolution video to be reconstructed based on the color feature information, the hierarchical feature information, and the spatial feature information according to the corresponding target feature weights, thereby obtaining the target high-resolution video.

[0229] Understandably, the aforementioned multi-scale channel attention module can be a module for color feature extraction, the aforementioned small residual module can be a module for hierarchical feature extraction, and the aforementioned large residual module can be a module for spatial feature extraction. Furthermore, these three modules can be connected in parallel with sub-pixel convolution modules and linear additive layers.

[0230] In practical use, color feature information, hierarchical feature information, and spatial feature information can be obtained through the three modules mentioned above, and then transmitted to the linear addition layer. The linear addition layer then performs color restoration or resolution enhancement on the low-resolution video to be reconstructed according to the target feature weights corresponding to each feature information, thereby obtaining the target high-resolution video.

[0231] Further, the step of extracting color features from the low-resolution video to be reconstructed using the multi-scale channel attention module includes:

[0232] The low-resolution video to be reconstructed is globally pooled using the multi-scale channel attention module, and the pooling result is mapped; the mapped result is segmented, and the color features of the segmentation result are extracted.

[0233] Reference Figure 13 , Figure 13 This is a schematic diagram of the multi-scale channel attention module in the third embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 13 As shown, this embodiment improves upon traditional channel attention by assuming that the features input to the multi-scale channel attention module are... Figure X The number of layers is C, and the size of each layer is W×H. They can be initially processed using two different sizes of MobileNet V3 convolutional layers (i.e., 3×3 and 5×5).

[0234] Figure 13 The network performs convolution operations using 3×3 and 5×5 convolutions. These two types of convolution kernels can capture local details and broader spatial context information, respectively, thereby generating two new sets of features, U and V. The channel dimensions of U and V are the same as X, namely C×W×H. Subsequently, the multi-scale channel attention module merges these two sets of features U and V together by adding corresponding matrix elements. This merging strategy ensures the effective fusion of feature information at different scales, enabling the network to more comprehensively understand and process image content. The fusion process can be represented by the following formula 15:

[0235] M = U + V (Formula 15)

[0236] Where M represents the feature after adding the matrix elements.

[0237] The feature maps processed by the multi-scale channel attention module require further processing to extract deeper features. This is followed by global pooling (i.e., ...). Figure 13 Global pooling (GSP) is applied to the feature maps, compressing each feature map of size W×H into a single value (i.e., ...). Figure 13 (C×1) (i.e., the pooling result above). This step helps to aggregate the global information of each feature map, providing a more representative feature representation for subsequent operations.

[0238] After global pooling, the resulting features are fed into two fully connected layers (i.e., Figure 13 Fully Connected 1

[0239] And the fully connected layers 2) perform mapping. The role of these two fully connected layers is to further transform the pooled features and finally map them to a 2C×1 dimension (i.e., the mapping result mentioned above). This process can be represented by the following formula 16:

[0240] w = FC2(FC1(GP(M))) Formula 16;

[0241] Here, GP represents the global pooling operation, FC1 represents the first fully connected layer, FC2 represents the second fully connected layer, and w represents the mapping result. After obtaining the mapping result, segmentation can be performed to extract color features.

[0242] Furthermore, the step of segmenting the mapping result and extracting color features from the segmented result includes:

[0243] The mapping result is expanded, and the expanded result is transformed by a preset activation function; the transformed result is segmented, and the color features of the segmented result are extracted.

[0244] In practical use, after obtaining the mapping result w, a reshape operation is used to expand it from two dimensions to three dimensions. Then, a preset activation function (such as the sigmoid function) is applied to obtain an output with the same dimensions as the transformed feature map. This output can be interpreted as the importance weight for each channel, which is a feature vector containing channel attention information with dimensions 2C×1×1. Next, the obtained feature vector is divided into two sub-vectors u and v corresponding to the two sets of features U and V obtained earlier, and these sub-vectors are then mapped to U and V respectively.

[0245] By performing multiplication and then addition, the entire process of calibrating the feature map using the multi-scale attention mechanism is completed, obtaining the relevant feature information of the color channels (i.e., the red, green, and blue (RGB) channels), which is the color feature information, denoted as Y. The specific process can be expressed by the following formula 17:

[0246] Y = uU + vV (Formula 17)

[0247] Furthermore, the feature information extraction module also includes: a first convolutional layer, an activation function layer, and a second convolutional layer connected in sequence, wherein the second convolutional layer is also connected to the multi-scale channel attention module;

[0248] The step of extracting color features from the low-resolution video to be reconstructed using the multi-scale channel attention module to obtain color feature information includes:

[0249] The first convolutional layer performs channel downsampling on the low-resolution video to be reconstructed, and the activation function layer activates the downsized low-resolution video to be reconstructed. The second convolutional layer restores the channels of the activated low-resolution video to be reconstructed, and the multi-scale channel attention module extracts color features from the restored low-resolution video to obtain color feature information.

[0250] Continue as Figure 12 As shown, a convolutional layer (i.e., ...) can be sequentially connected before the aforementioned multi-scale channel attention module. Figure 12 The MobileNet V3 convolutional block, i.e., the first convolutional layer mentioned above), and an activation function layer (i.e. Figure 12 In the middle ReLU, that is, the activation function layer mentioned above) and a convolutional layer (i.e. Figure 12

[0251] In the MobileNet V3 convolutional block (i.e., the second convolutional layer mentioned above), all convolutional layers can be 1×1 convolutions. The first convolutional layer (the first convolutional layer mentioned above) reduces the number of channels in the input feature map to 1 / r of the original number of channels, where r is the number of channels in the input feature map. This helps reduce computational cost and the number of parameters. Then, an activation function layer (which can be the ReLU activation function) is used to activate the reduced feature map. The second convolutional layer (the second convolutional layer mentioned above) restores the number of channels to the original dimension to generate attention coefficients with the same number of channels as the original feature map. The restored feature map is then passed to the multi-scale channel attention module for the above processing to obtain a feature map with channel attention coefficients. Finally, an element-wise multiplication operation is performed with the original feature map. The multiplication process can recalibrate the original feature map, adjusting the feature values ​​of the corresponding channels according to the importance of each channel. In this way, the model can pay more attention to the channel information that has an important impact on super-resolution reconstruction, while suppressing the information of unimportant channels, thereby improving the performance and efficiency of video super-resolution reconstruction.

[0252] It should be emphasized that the aforementioned color feature information can be information about color channel features in the image, the aforementioned hierarchical feature information can be feature information between different layers of images, and the aforementioned spatial feature information can be feature information of key regions in the image.

[0253] For small residual modules, hierarchical features of the input feature map can be extracted to obtain hierarchical feature information. For large residual modules, spatial features of the input feature map can be extracted to obtain spatial feature information. Finally, color adjustment or super-resolution tasks are completed based on color feature information, hierarchical feature information, and spatial feature information.

[0254] In both small and large residual modules, a convolutional layer (i.e., ...) can be used first. Figure 12 The shallow feature extraction is performed using MobileNet V3 convolutional blocks, and this process can be represented by the following formula 18:

[0255] F s3 =C SF (I LR ) Formula 18;

[0256] Among them I LR C is the initial high-resolution video output by the subpixel convolution module. SF For convolution operations, F s3 This represents the shallow features obtained from the convolutional layer. These shallow features are then passed to the small residual module and the large residual module for deep feature extraction. This process can be represented by the following formula 19:

[0257] F D3 =R DF (F s3 ) Formula 19;

[0258] Where R DF For the extraction of the aforementioned deep features, F D3 The results are extracted.

[0259] It is important to emphasize that the small residual module and the large residual module can be multi-level residual structures. To balance the complexity of the network and the stability of training, the low-quality video super-resolution network adopts a design strategy of nesting small residual modules within large residual modules (i.e., Figure 12 (As shown). This nested structure not only deepens the network but also effectively prevents the gradient explosion problem during training. Simultaneously, by increasing the number of feature extraction units, the network can expand its receptive field, enhance its ability to capture image information, and further improve the accuracy of feature extraction.

[0260] It should also be emphasized that, if Figure 12 The structure shown can be used for color correction tasks without needing a subpixel convolutional module after the linear summation layer. However, if used for super-resolution tasks, a subpixel convolutional module can be placed after the linear summation layer (i.e., ...). Figure 12 (Subpixel convolution) This subpixel convolution module may contain subpixel convolution layers. After performing super-resolution reconstruction through linear additive layers, the obtained results can be upsampled and amplified through this subpixel convolution module to further improve the resolution.

[0261] That is, the feature information extraction module further includes: a sub-pixel convolution module, which is connected to the linear addition layer;

[0262] The step of reconstructing the low-resolution video to be reconstructed using the linear additive layer based on the color feature information, the hierarchical feature information, and the spatial feature information according to the corresponding target feature weights, to obtain the target high-resolution video, includes:

[0263] The low-resolution video to be reconstructed is reconstructed by the linear additive layer based on the color feature information, the hierarchical feature information, and the spatial feature information according to the corresponding target feature weights to obtain an initial high-resolution video; the initial high-resolution video is then magnified by the subpixel convolution module to obtain a target high-resolution video.

[0264] After being magnified by the subpixel convolution module, the final target high-resolution video is generated by a MobileNet V3 convolution operation. This process can be represented by the following formula 20:

[0265] I LR =C RE (S UP (F D3 )) Formula 20.

[0266] Where S UP For the upsampling operation of this subpixel convolution module, C RE For the convolution operation of this convolutional layer, I LR For high-resolution video.

[0267] Furthermore, to differentiate between color grading tasks and super-resolution tasks, this embodiment can set different loss functions for different tasks. For the channel attention mechanism module in the color grading task, the corresponding color grading stage loss function can be composed of two loss functions: mean absolute error and structural similarity error (SSIM), which can be expressed by the following formula 21:

[0268] L Color =αL1+βL SSIM Formula 21;

[0269] Where L Color L1 is the loss function for the color grading stage, and L2 is the L1 loss function. SSIM Let α be the structural similarity error loss function, and let β be the weight coefficients. These can be obtained through adaptive training. In color correction tasks, since structural similarity error can focus on the structural information of the image, it tends to retain more details such as edges and textures during the optimization process, making the reconstructed video more natural and realistic. Therefore, β will be smaller.

[0270] For the channel attention mechanism module in the super-resolution task, the corresponding super-resolution stage loss function can be constructed using the mean absolute error, and can be expressed by the following formula 22:

[0271]

[0272] Where L SR For the super-resolution stage loss function, Let be the frame image of the target super-resolution video after super-resolution, and 1 be the L1 loss function.

[0273] To further differentiate between color correction and super-resolution tasks, this embodiment can also incorporate a gating mechanism into the linear addition layer. Specifically, corresponding gates are set for the multi-scale channel attention module, the large residual module, and the small residual module, respectively, as shown in the reference. Figure 14 , Figure 14 This is a schematic diagram of the gating mechanism in the third embodiment of the video super-resolution reconstruction method of this application, as shown below. Figure 14 As shown, a channel gate can be set for the multi-scale channel attention module, a spatial gate can be set for the large residual module, and a hierarchical gate can be set for the small residual module. A target feature weight (i.e., ...) is set for each gate. Figure 14 The target feature weights for the channel gate (h), spatial gate (σ), and hierarchical gate (tanh) are obtained through adaptive training of the initial feature weights using training data. The channel gate controls the flow of information between channels by calculating the weight of each channel. This helps the model focus on important channel features while ignoring irrelevant channel information. The spatial gate controls the flow of spatial information by calculating the weight at each location, which helps the model focus on key regions in the image while suppressing the influence of background or noise regions. The hierarchical gate controls the interaction between information at different levels.

[0274] Furthermore, in order to obtain the aforementioned target feature weights, the step of obtaining the low-resolution video to be reconstructed includes, prior to:

[0275] Acquire training data and adaptively train the initial reconstruction model using the training data; adjust the initial feature weights corresponding to each feature information in the gating mechanism based on the training results to obtain a preset reconstruction model with target feature weights.

[0276] Before use, training data containing the target high-resolution video and the low-resolution video to be reconstructed can be obtained, and the initial feature weights set for each gate in the gating mechanism can be trained. For color grading tasks, color feature information may be more important, and the target feature weights obtained by the channel gate after self-learning may be higher. For super-resolution tasks, the target feature weights obtained by the hierarchical gate and spatial gate lock after self-learning may be higher.

[0277] Therefore, the aforementioned gating mechanism can learn the importance or weight of each channel and adjust the output of the feature map based on these weights. This helps improve the quality of super-resolution reconstruction, resulting in clearer and more detailed images. The gating mechanism can process all channels uniformly, meaning that regardless of the number of channels in the input feature map, the gating mechanism can use the same structure and parameters for computation. This uniform approach not only simplifies the model's complexity but also makes the model more flexible and scalable.

[0278] In summary, in this embodiment, the low-resolution video super-resolution module is sequentially connected to a color restoration module and a resolution enhancement module, and the color restoration module is also connected to the non-local residual module.

[0279] The specific process of color restoration and resolution enhancement is as follows: the color restoration module restores the color of the adjusted initial high-resolution video; the resolution enhancement module enhances the resolution of the restored initial high-resolution video to obtain the target high-resolution video.

[0280] It should be noted that the aforementioned color restoration module can be connected after the sub-pixel convolution module in the non-local residual module. Furthermore, the aforementioned color restoration module may include the aforementioned channel attention mechanism module. For ease of subsequent explanation, the channel attention mechanism module in the color restoration module will be referred to as the first channel attention mechanism module, and the first channel attention mechanism module may include the aforementioned gating mechanism and color grading stage loss function set for the color grading task.

[0281] The multi-scale channel attention module in the first channel attention mechanism module is denoted as the first multi-scale channel attention module, the small residual module is denoted as the first small residual module, the large residual module is denoted as the first large residual module, the convolutional layer and activation function layer before the first multi-scale channel attention module can be denoted as the third convolutional layer, the first activation function layer and the fourth convolutional layer, respectively, and the linear addition layer is denoted as the first linear addition layer.

[0282] In practical use, the input initial low-resolution video (i.e., the initial low-resolution video after being amplified by the sub-pixel convolution module) is down-channeled through the third convolutional layer, and the down-channeled initial low-resolution video is activated through the first activation function layer; the activated initial low-resolution video is restored through the fourth convolutional layer, and the color features of the restored initial low-resolution video are extracted through the first multi-scale channel attention module to obtain color feature information; the hierarchical features of the input initial low-resolution video are extracted through the first small residual module to obtain hierarchical feature information; the spatial features of the input initial low-resolution video are extracted through the first large residual module to obtain spatial feature information; and finally, the color is restored through the gating mechanism in the first linear addition layer based on the color feature information, hierarchical feature information, and spatial feature information according to the corresponding target feature weights to obtain the restored initial high-resolution video.

[0283] Similarly, the resolution enhancement module described above can be connected after the color restoration module. The resolution enhancement module can also include the aforementioned channel attention mechanism module, and its depth can be greater than that of the color restoration module (i.e.,...). Figure 11 (7 layers in the middle). For ease of explanation, the channel attention mechanism module in the resolution enhancement module will be referred to as the second channel attention mechanism module. The second channel attention mechanism module may include the gating mechanism and the loss function set for the super-resolution task.

[0284] The multi-scale channel attention module in the second channel attention mechanism module is denoted as the second multi-scale channel attention module, the small residual module is denoted as the second small residual module, the large residual module is denoted as the second large residual module, the convolutional layer and activation function layer before the second multi-scale channel attention module can be denoted as the fifth convolutional layer, the second activation function layer and the sixth convolutional layer, respectively, and the linear addition layer is denoted as the second linear addition layer.

[0285] In practical application, the fifth convolutional layer performs channel downscaling on the restored initial high-resolution video, and the second activation function layer activates the downscaled initial low-resolution video. The sixth convolutional layer restores the channels of the activated initial low-resolution video, and the second multi-scale channel attention module extracts color features from the restored initial low-resolution video to obtain color feature information. The second small residual module extracts hierarchical features from the restored initial high-resolution video to obtain hierarchical feature information. The second large residual module extracts spatial features from the restored initial high-resolution video to obtain spatial feature information. Then, the gating mechanism in the second linear addition layer performs resolution upscaling based on color, hierarchical, and spatial feature information according to the corresponding target feature weights. Finally, a sub-pixel convolution is used for magnification, allowing for the processing of each magnified result (i.e.,...) Figure 11 By encoding high-quality keyframes (in medium to high quality) into the video, the aforementioned high-resolution video can be obtained.

[0286] It should be emphasized that the above process can be referred to as [the previous one]. Figures 12 to 14 The description process is not elaborated here in this embodiment.

[0287] This embodiment can perform color restoration and resolution enhancement through the aforementioned low-resolution video super-resolution module, thereby further improving video quality.

[0288] Reference Figure 15 , Figure 15 This is a structural block diagram of the first embodiment of the video super-resolution reconstruction device of this application.

[0289] like Figure 15 As shown, the video super-resolution reconstruction apparatus proposed in this application includes:

[0290] The video acquisition unit 151 is used to acquire the low-resolution video to be reconstructed.

[0291] The resolution reconstruction unit 152 is used to reconstruct the low-resolution video to be reconstructed using a preset reconstruction model to obtain the target high-resolution video.

[0292] The preset reconstruction model includes at least: a channel attention mechanism module with a gating mechanism, wherein the gating mechanism carries target feature weights corresponding to each feature information, and each target feature weight is determined through adaptive training. The channel attention mechanism module is used to extract features from the low-resolution video to be reconstructed, obtain each feature information, and reconstruct the low-resolution video to be reconstructed based on each feature information and the corresponding target feature weights to obtain the target high-resolution video.

[0293] In this embodiment, the channel attention mechanism module may include a gating mechanism. This gating mechanism contains target feature weights corresponding to each feature information, and these target feature weights are determined through adaptive training. During use, the gating mechanism can be used to reconstruct the data according to the target feature weights corresponding to each feature information, thereby reducing interference from information invalid for the current task and improving the reconstruction effect.

[0294] As one implementation, the video acquisition unit 151 is also used to acquire training data and adaptively train the initial reconstruction model using the training data; and adjust the initial feature weights corresponding to each feature information in the gating mechanism based on the training results to obtain a preset reconstruction model with target feature weights.

[0295] In one implementation, the channel attention mechanism module includes: a feature information extraction module and a linear addition layer with a gating mechanism, wherein the linear addition layer is connected to the feature information extraction module;

[0296] The resolution reconstruction unit 152 is further configured to extract features from the low-resolution video to be reconstructed through the feature information extraction module to obtain various feature information; and to reconstruct the low-resolution video to be reconstructed through the linear addition layer based on each feature information and according to the corresponding target feature weights to obtain the target high-resolution video.

[0297] In one implementation, the feature information extraction module includes a multi-scale channel attention module, a large residual module, and a small residual module, all of which are connected to the linear addition layer.

[0298] The resolution reconstruction unit 152 is further configured to extract color features from the low-resolution video to be reconstructed through the multi-scale channel attention module to obtain color feature information; extract hierarchical features from the low-resolution video to be reconstructed through the small residual module to obtain hierarchical feature information; and extract spatial features from the low-resolution video to be reconstructed through the large residual module to obtain spatial feature information.

[0299] The resolution reconstruction unit 152 is further configured to reconstruct the low-resolution video to be reconstructed based on the color feature information, the hierarchical feature information and the spatial feature information according to the corresponding target feature weights through the linear additive layer, thereby obtaining the target high-resolution video.

[0300] In one implementation, the resolution reconstruction unit 152 is also used to perform global pooling on the low-resolution video to be reconstructed through the multi-scale channel attention module, and to map the pooling result; to segment the mapping result, and to extract the color features of the segmentation result.

[0301] In one implementation, the resolution reconstruction unit 152 is also used to expand the mapping result and transform the expanded result through a preset activation function; to segment the transformed result and to extract the color features of the segmented result.

[0302] In one implementation, the feature information extraction module further includes: a first convolutional layer, an activation function layer, and a second convolutional layer connected in sequence, wherein the second convolutional layer is also connected to the multi-scale channel attention module;

[0303] The resolution reconstruction unit 152 is further configured to perform channel reduction on the low-resolution video to be reconstructed through the first convolutional layer, and activate the reduced low-resolution video to be reconstructed through the activation function layer; perform channel restoration on the activated low-resolution video to be reconstructed through the second convolutional layer, and extract color features from the restored low-resolution video to be reconstructed through the multi-scale channel attention module to obtain color feature information.

[0304] In one implementation, the feature information extraction module further includes a sub-pixel convolution module, which is connected to the linear addition layer;

[0305] The resolution reconstruction unit 152 is further configured to reconstruct the low-resolution video to be reconstructed based on the color feature information, the hierarchical feature information, and the spatial feature information according to the corresponding target feature weights through the linear additive layer to obtain an initial high-resolution video; and to enlarge the initial high-resolution video through the subpixel convolution module to obtain a target high-resolution video.

[0306] Other embodiments or specific implementations of the video super-resolution reconstruction apparatus of this application can be found in the above-described method embodiments, and will not be repeated here.

[0307] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0308] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0309] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0310] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for video super-resolution reconstruction, characterized in that, The method comprises: acquiring a low-resolution video to be reconstructed; reconstructing the low-resolution video to be reconstructed by a preset reconstruction model to obtain a target high-resolution video; wherein the preset reconstruction model at least comprises a channel attention mechanism module provided with a gating mechanism, the gating mechanism being provided with target feature weights corresponding to each feature information, each target feature weight being determined through adaptive training, the channel attention mechanism module being used for feature extraction of the low-resolution video to be reconstructed to obtain each feature information, and the low-resolution video to be reconstructed being reconstructed based on each feature information according to the corresponding target feature weight to obtain a target high-resolution video.

2. The method of claim 1, wherein, Before the step of acquiring the low-resolution video to be reconstructed, the method further comprises: acquiring training data and performing adaptive training on an initial reconstruction model through the training data; adjusting initial feature weights corresponding to each feature information in the gating mechanism based on a training result to obtain the preset reconstruction model provided with target feature weights.

3. The method of claim 1, wherein, The channel attention mechanism module comprises a feature information extraction module and a linear addition layer provided with a gating mechanism, and the linear addition layer is connected with the feature information extraction module; The step of reconstructing the low-resolution video to be reconstructed by the preset reconstruction model to obtain a target high-resolution video comprises: extracting features of the low-resolution video to be reconstructed by the feature information extraction module to obtain each feature information; reconstructing the low-resolution video to be reconstructed based on each feature information according to the corresponding target feature weight by the linear addition layer to obtain a target high-resolution video.

4. The method of claim 3, wherein, The feature information extraction module comprises a multi-scale channel attention module, a large residual module and a small residual module, and the multi-scale channel attention module, the large residual module and the small residual module are all connected with the linear addition layer; The step of extracting features of the low-resolution video to be reconstructed by the feature information extraction module to obtain each feature information comprises: extracting color features of the low-resolution video to be reconstructed by the multi-scale channel attention module to obtain color feature information; extracting hierarchical features of the low-resolution video to be reconstructed by the small residual module to obtain hierarchical feature information; extracting spatial features of the low-resolution video to be reconstructed by the large residual module to obtain spatial feature information; The step of reconstructing the low-resolution video to be reconstructed based on each feature information according to the corresponding target feature weight by the linear addition layer to obtain a target high-resolution video comprises: reconstructing the low-resolution video to be reconstructed based on the color feature information, the hierarchical feature information and the spatial feature information according to the corresponding target feature weight by the linear addition layer to obtain a target high-resolution video.

5. The method of claim 4, wherein, The step of extracting color features of the low-resolution video to be reconstructed by the multi-scale channel attention module comprises: The global pooling is performed on the low-resolution video to be reconstructed by the multi-scale channel attention module, and the mapping result is obtained. The mapping result is segmented, and the color features of the segmented result are extracted.

6. The method of claim 5, wherein, The step of segmenting the mapping result and extracting the color features of the segmented result comprises: The mapping result is augmented, and the augmented result is converted by a preset activation function; The conversion result is segmented, and the color features of the segmented result are extracted.

7. The method of claim 4, wherein, The feature information extraction module further comprises a first convolutional layer, an activation function layer and a second convolutional layer connected in sequence, and the second convolutional layer is further connected with the multi-scale channel attention module. The step of extracting the color feature information of the low-resolution video to be reconstructed by the multi-scale channel attention module comprises: The channel of the low-resolution video to be reconstructed is reduced by the first convolutional layer, and the reduced low-resolution video to be reconstructed is activated by the activation function layer; The channel of the activated low-resolution video to be reconstructed is restored by the second convolutional layer, and the color features of the restored low-resolution video to be reconstructed are extracted by the multi-scale channel attention module to obtain the color feature information.

8. The method of claim 4, wherein, The feature information extraction module further comprises a sub-pixel convolution module connected with the linear addition layer. The step of reconstructing the low-resolution video to be reconstructed by the linear addition layer based on the color feature information, the hierarchical feature information and the spatial feature information according to the corresponding target feature weight to obtain the target high-resolution video comprises: The low-resolution video to be reconstructed is reconstructed by the linear addition layer based on the color feature information, the hierarchical feature information and the spatial feature information according to the corresponding target feature weight to obtain the initial high-resolution video; The initial high-resolution video is enlarged by the sub-pixel convolution module to obtain the target high-resolution video.

9. An apparatus for video super-resolution reconstruction, the apparatus comprising: a video super-resolution reconstruction module configured to perform the method of any one of claims 1-8. The device comprises: A video acquisition unit is configured to acquire a low-resolution video to be reconstructed; A resolution reconstruction unit is configured to reconstruct the low-resolution video to be reconstructed by a preset reconstruction model to obtain a target high-resolution video. The preset reconstruction model at least comprises a channel attention mechanism module provided with a gating mechanism, the gating mechanism has target feature weights corresponding to each feature information, each target feature weight is determined by adaptive training, the channel attention mechanism module is used for feature extraction of the low-resolution video to be reconstructed, obtaining each feature information, and reconstructing the low-resolution video to be reconstructed based on each feature information according to the corresponding target feature weight to obtain a target high-resolution video.

10. A wearable device, comprising: The wearable device comprises a memory, a processor, and a video super-resolution reconstruction program stored on the memory and executable on the processor, the video super-resolution reconstruction program, when executed by the processor, implements the steps of the video super-resolution reconstruction method according to any one of claims 1 to 8.

Images