A teaching video super-resolution reconstruction method and system based on cross-modal feature fusion
By employing a cross-modal feature fusion method that combines various physical information and the Mamba network, the problems of low resolution and privacy concerns in teaching videos are addressed, enabling efficient and low-latency reconstruction of teaching videos and improving the clarity and temporal consistency of key details.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANHUI POLYTECHNIC UNIV MECHANICAL & ELECTRICAL COLLEGE
- Filing Date
- 2026-04-30
- Publication Date
- 2026-07-24
AI Technical Summary
Existing teaching videos have low resolution and significant loss of detail. Traditional super-resolution methods have a large number of parameters and high latency, and fail to effectively combine multi-source physical information, resulting in poor reconstruction results. Furthermore, high-definition video acquisition is prone to privacy concerns.
A cross-modal feature fusion method is adopted. By acquiring normal, BRDF, depth and motion vector information, global spatiotemporal modeling is performed in combination with the SS2D module of the Mamba network. High-resolution video frames are generated by cross-channel feature interaction rearrangement and point-by-point product fusion, which reduces computational complexity and suppresses inter-frame jitter.
It improves the clarity of key elements in instructional videos, reduces computational load and latency, is suitable for real-time augmentation scenarios, has privacy protection capabilities, and provides a high-quality data foundation for instructional behavior analysis.
Smart Images
Figure CN122453609A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of teaching technology, and in particular relates to a method and system for super-resolution reconstruction of teaching videos based on cross-modal feature fusion. Background Technology
[0002] Currently, classroom teaching videos generally suffer from low resolution and significant loss of detail, making it difficult to accurately identify and analyze key teaching behaviors such as teachers' blackboard writing, operation of teaching aids, and micro-expressions.
[0003] Traditional super-resolution methods, mostly based on interpolation, reconstruction, or deep learning models, while improving image quality, suffer from problems such as large parameter counts, high latency, and poor temporal consistency, making it difficult to meet the demands of detailed restoration and real-time processing in educational videos. Furthermore, existing methods often rely on a single visual modality, failing to effectively integrate multi-source physical information (such as geometry, material, and motion), resulting in poor reconstruction performance in complex teaching scenarios. Simultaneously, high-definition video capture raises privacy concerns, and current technologies fail to simultaneously improve image quality and protect privacy. Therefore, there is an urgent need for a super-resolution reconstruction method for educational videos that can integrate multimodal information and possess efficient feature extraction and spatiotemporally consistent modeling capabilities.
[0004] Based on the above reasons, this invention designs a method and system for super-resolution reconstruction of teaching videos based on cross-modal feature fusion. Summary of the Invention
[0005] The purpose of this invention is to solve the problems in the prior art, and to propose a method and system for super-resolution reconstruction of teaching videos based on cross-modal feature fusion.
[0006] This invention first discloses a method for super-resolution reconstruction of teaching videos based on cross-modal feature fusion, comprising the following steps: Step S1: Obtain the low-resolution teaching video frame sequence and its corresponding G-buffer information sequence; the G-buffer information includes normal information, BRDF information, depth information and motion vector information; Step S2: Input the current video frame into the encoding network. The encoding network first extracts initial features from the current video frame through convolutional layers to obtain initial features. Then, it performs separable convolution on the initial features and separates them into first branch features and second branch features. The first branch features are rearranged across channels. The rearranged features are concatenated with the initial features and then multiplied point-by-point with the second branch features to obtain the encoded features. Step S3: The encoded features and the G-buffer information are used as dual-branch inputs and input into the decoding network. In the decoding network, the two-dimensional selective scanning module SS2D of the Mamba network is used to perform global feature extraction and spatiotemporal feature fusion on the encoded features and the G-buffer information respectively. The decoded features output by the decoding network of the previous frame image are fused with the SS2D output features of the current frame to obtain the decoded features of the current frame. Step S4: Upsample the decoded features through subpixel convolution to generate a high-resolution video frame for the current frame.
[0007] In the above method, the cross-channel feature interaction rearrangement in step S2 specifically includes: For the first branch feature, let its spatial height be H, width be W, and number of channels be C. The interaction is represented as follows:
[0008] in, This indicates that after cross-channel feature interaction, in The feature sampled values of the location, where , , These represent the output height, width, and number of channels, respectively. The change in channels is represented as follows:
[0009]
[0010] In the formula, The number of channels for the output features. This represents the scaling factor.
[0011] In the above method, the high-resolution features output by the decoding network of the previous frame are fused with the SS2D output features of the current frame by performing a linear transformation and activation function processing on the decoding output of the previous frame, and then multiplying it point by point with the SS2D output features of the current frame.
[0012] In the above method, step S3 specifically includes the following processing of the encoded features and G-buffer information by the decoding network: performing layer normalization and linear transformation on the encoded features and G-buffer information respectively, and then inputting them into their respective SS2D modules after passing through depthwise separable convolution and SiLU activation function; fusing the outputs of the two SS2D modules with the decoded output of the previous frame after linear transformation and activation to obtain the decoded features of the current frame.
[0013] The SS2D module includes a scan expansion operation and a scan merging operation; the scan expansion operation is to expand the input features into multiple sub-images along at least two different directions; the scan merging operation is to merge each sub-image into an output feature with the same size as the input feature after processing by a selective state-space model.
[0014] In the above method, the scan expansion operation is performed along four symmetrical directions, namely from top to bottom, from bottom to top, from left to right, and from right to left.
[0015] In the above method, in step S2, the convolutional layer is a 3×3 convolutional layer.
[0016] In the above method, the normal information, BRDF information, depth information and motion vector information in the G-buffer information are input into the decoding network with the same spatiotemporal resolution as the low-resolution video frame, and are used as conditional information to participate in the global feature extraction of the SS2D module after being aligned with the encoded features in the channel dimension.
[0017] In the above method, after the decoding network outputs the decoding features of the current frame, it also buffers them for temporal fusion of the next frame.
[0018] Secondly, the present invention provides a teaching video super-resolution reconstruction system based on cross-modal feature fusion, used in the above method, comprising: Input module: used to acquire low-resolution teaching video frame sequences and their corresponding G-buffer information sequences, wherein the G-buffer information includes normal information, BRDF information, depth information and motion vector information; Encoding module: used to perform the operation of step S2 as described in claim 1 on the current video frame to obtain encoded features; Decoding module: used to take the encoded features and the G-buffer information as dual-branch inputs, execute the operation of step S3 as described in claim 1, and obtain the decoded features of the current frame; Reconstruction module: used to perform the operation of step S4 of claim 1 on the decoded features to generate a high-resolution video frame of the current frame.
[0019] The beneficial effects of this invention are as follows: This scheme introduces G-buffer information containing normals, BRDFs, depth, and motion vectors as additional input. In the encoding stage, it employs cross-channel feature interactive rearrangement and pointwise product fusion. In the decoding stage, it utilizes the SS2D module of the Mamba network to perform global spatiotemporal modeling of the bi-branch features and incorporates the pointwise product of the previous frame's decoding output into the current frame. Compared to existing super-resolution methods that rely solely on RGB, this scheme uses physical information to guide reconstruction, improving the clarity of key teaching elements such as blackboard writing and teaching aids. Simultaneously, the temporal fusion mechanism suppresses inter-frame jitter, making continuous motion reconstruction smoother.
[0020] This scheme uses depthwise separable convolution instead of ordinary convolution in the decoding network and uses the SS2D module as the core feature extraction unit, which reduces the overall number of parameters. Combined with subpixel convolution upsampling, it can achieve fast processing capabilities at high resolution, making it suitable for real-time enhancement scenarios in classroom teaching. It solves the problems of large number of parameters and high latency in existing methods.
[0021] This solution reconstructs high-resolution images based on low-resolution video and estimated G-buffer information, eliminating the need to collect original high-definition video and reducing privacy sensitivity during data acquisition. Simultaneously, the reconstructed high-resolution teaching video provides a high-quality data foundation for subsequent tasks such as teaching behavior analysis, teacher-student interaction recognition, and blackboard content comprehension, demonstrating adaptability to digital education scenarios. Attached Figure Description
[0022] Figure 1 This is a flowchart of a teaching video super-resolution reconstruction method based on cross-modal feature fusion disclosed in this invention.
[0023] Figure 2 This is an architecture diagram of a teaching video super-resolution reconstruction method based on cross-modal feature fusion disclosed in this invention.
[0024] Figure 3 This is a schematic diagram of the cross-modal feature fusion module in a teaching video super-resolution reconstruction method based on cross-modal feature fusion disclosed in this invention.
[0025] Figure 4 This is a schematic diagram illustrating the visualization of teaching video reconstruction in a teaching video super-resolution reconstruction method based on cross-modal feature fusion disclosed in this invention. Detailed Implementation
[0026] To facilitate understanding of this application and to make the aforementioned objectives, features, and advantages of this application more apparent, a detailed description of specific embodiments of this application is provided below in conjunction with the accompanying drawings. Numerous specific details are set forth in the following description to provide a thorough understanding of this application, and preferred embodiments are shown in the accompanying drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application. This application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified. In the description of this application, "several" means at least one, such as one, two, etc., unless otherwise explicitly specified. It should be noted that when an element is referred to as being "fixed to" another element, it can be directly attached to the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementations. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is only for describing particular implementations and is not intended to limit the scope of this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0027] Reference Figure 1-4 A method for super-resolution reconstruction of teaching videos based on cross-modal feature fusion is proposed. The method includes the following steps: First, a low-resolution teaching video frame sequence and the corresponding G-buffer information sequence are obtained. The G-buffer information includes normal information, BRDF information, depth information, and motion vector information.
[0028] Then, the current video frame is input into the encoding network: the encoding network first extracts initial features from the current frame through convolutional layers to obtain initial features; The initial feature is then subjected to a separable convolution, and the convolution result is separated into first branch features and second branch features. Cross-channel feature interaction rearrangement is performed on the first branch features, and the rearranged features are concatenated with the original initial features. The concatenated result is then multiplied pointwise with the second branch features to obtain the encoded features.
[0029] Next, the encoded features and G-buffer information are used as dual branches to input into the decoding network: In the decoding network, the two-dimensional selective scanning module SS2D contained in the Mamba network is used to perform global feature extraction and spatiotemporal feature fusion on the encoded features and G-buffer information respectively. At the same time, the high-resolution features output by the decoding network of the previous frame are fused with the SS2D output features of the current frame to obtain the decoding features of the current frame.
[0030] Finally, the decoded features are upsampled through subpixel convolution to generate a high-resolution video frame for the current frame. This method, by introducing various physical information, including normals, BRDFs, depth, and motion vectors, and combining cross-modal feature fusion, can recover key details such as blackboard writing and teaching aids in teaching videos with low computational cost. At the same time, it effectively suppresses inter-frame jitter by fusing information from previous and subsequent frames, improving the temporal stability of the reconstructed video.
[0031] Furthermore, the specific implementation method of the above cross-channel feature interaction rearrangement is as follows: For the input first branch feature tensor, its spatial height is set to H, width to W, and number of channels to C. The value of the output rearranged feature at position (i,j,k) is taken from the value of the input feature at position (h,w,c). The output position indices i,j,k, the input position indices h,w,c, and a preset scaling factor r satisfy the following relationship:
[0032]
[0033] In the formula, , , Let r represent the height, width, and number of channels of the output, respectively, where r is a positive integer; c is the channel index of the output feature; and h and w are the spatial position indices of the input features. This mapping relationship ensures that each position of the output feature is provided by a specific, cross-channel rearranged position of the input features. This rearrangement mechanism enhances the spatial structure modeling ability of features without introducing additional learnable parameters, promotes the interaction of nonlocal information, and reduces redundant features, thereby improving the efficiency of representing geometric and texture information in the teaching video during the encoding stage.
[0034] In one feasible embodiment, the fusion method between the decoded output of the previous frame and the SS2D output features of the current frame is as follows: The high-resolution features output by the decoding network of the previous frame are subjected to linear transformation and activation function processing. Then, the processed features are multiplied point-by-point with the SS2D output features of the current frame. This point-by-point multiplication fusion can adaptively modulate the feature response of the current frame according to the semantic information of the previous frame, ensuring the continuity and consistency of the reconstruction results in the time dimension, and effectively eliminating flickering and jitter that may occur due to independent reconstruction of a single frame.
[0035] More preferably, the specific processing flow of the decoding network for encoded features and G-buffer information is as follows: the encoded features and G-buffer information are sequentially subjected to layer normalization and linear transformation, then passed through depthwise separable convolution and SiLU activation function, and then fed into their respective SS2D modules. The outputs of the two SS2D modules are fused with the decoded output of the previous frame after linear transformation and activation to obtain the decoded features of the current frame. In this process, depthwise separable convolution significantly reduces the amount of computation and parameters, while the dual-branch SS2D structure allows visual features and physical information (normals, BRDF, depth, motion vectors) to interact deeply in the state space model, thereby more accurately reconstructing complex details such as material boundaries and motion trajectories.
[0036] The internal structure of the SS2D module includes a scan expansion operation and a scan merging operation. The scan expansion operation expands the input features into multiple sub-images along at least two different directions; the scan merging operation processes each sub-image through a selective state-space model (i.e., the S6 module) and then merges them into an output feature of the same size as the input features. In a preferred embodiment, the scan expansion operation is performed along four symmetrical directions, specifically from top to bottom, bottom to top, left to right, and right to left. This multi-directional scan layout can comprehensively cover all regions of the input features, providing a rich multi-dimensional information foundation for subsequent feature extraction through systematic directional transformations, thereby improving the model's ability to capture edges and motion patterns with different orientations in the teaching video.
[0037] Furthermore, the initial feature extraction convolutional layer used in the encoding network can employ a 3×3 convolutional kernel. This size of convolution achieves a good balance between receptive field and computational cost, making it suitable for real-time processing requirements.
[0038] Regarding the use of G-buffer information in the decoding network, each piece of physical information is input into the decoding network at the same spatiotemporal resolution as the low-resolution video frames. It is then aligned with the encoded features along the channel dimension as conditional information, and together they participate in the global feature extraction of the SS2D module. This alignment and conditional injection method ensures a pixel-by-pixel correspondence between physical information and visual features, enabling the network to fully utilize geometric and material priors to guide super-resolution reconstruction, achieving superior recovery results, especially in key teaching areas such as handwriting on the blackboard and highlights on teaching aids.
[0039] After the decoding network outputs the decoding features of the current frame, this method also caches these features for temporal fusion during the decoding process of the next frame. In this way, the reconstruction results of the entire video sequence can be correlated in time, further enhancing long-term consistency.
[0040] Corresponding to the above method, this invention also provides a teaching video super-resolution reconstruction system based on cross-modal feature fusion. The system includes an input module, an encoding module, a decoding module, and a reconstruction module. The input module is used to acquire a low-resolution teaching video frame sequence and its corresponding G-buffer information sequence, wherein the G-buffer information includes normals, BRDFs, depth, and motion vectors.
[0041] The encoding module is used to perform the aforementioned encoding operations on the current video frame, including shallow convolution, separable convolution and splitting, cross-channel rearrangement, concatenation and pointwise multiplication, and then obtain the encoded features.
[0042] The decoding module takes the encoded features and G-buffer information as dual-branch inputs and performs the aforementioned decoding operations, including layer normalization, linear transformation, depthwise separable convolution, SS2D processing, and fusion of the previous frame output to obtain the decoded features of the current frame.
[0043] The reconstruction module performs subpixel convolutional upsampling on the decoded features to generate a high-resolution video frame for the current frame. This system employs a modular structure that fully corresponds to the method, enabling efficient real-time super-resolution reconstruction of teaching videos while providing robust privacy protection.
[0044] In the specific implementation of this scheme, during the encoding stage, low-resolution teaching video frames and their corresponding G-buffer information, including normals, BRDF, depth, and motion vectors, are input. Then, through a dynamic feature selection mechanism, the input features are interacted and rearranged across channels to enhance the spatial structure modeling capability and reduce redundant features.
[0045] In specific implementation, this invention can be carried out as follows: During the encoding stage, for a low-resolution teaching video, it is decomposed into a series of video frames. and G-buffer information Collection, which contains G-buffer (Normal information) (BRDF information) (Depth information), and (Motion vector information).
[0046] Then, the video frames are input into the encoding network, for Video frames input from the network at any given time , These represent the height, width, and number of channels of the feature, respectively. The network first passes through a 3×3 convolutional layer... Initial feature extraction, used as the initial feature input for the encoding part, can be represented as:
[0047] like Figure 3 As shown in a. The above separable convolution operation will then... Separation yields the first path feature Second branch characteristics , can be represented as:
[0048] In the formula, This indicates a feature separation operation.
[0049] Meanwhile, this invention designs a dynamic feature selection mechanism for the first branch features. To improve the information exchange of non-local features, cross-channel feature interaction is performed, specifically including: set up This indicates that after cross-channel feature interaction, in The feature sample value of the location is the feature. exist The interactive rearrangement can be represented as follows:
[0050] in:
[0051]
[0052]
[0053] These three represent the output height, width, and channel, respectively. The change process across channels is represented as follows:
[0054]
[0055] In the formula, The number of channels for the output features. This represents the scaling factor.
[0056] The cross-channel interaction features are then fused with the initial features, which can be represented as:
[0057] In the formula, This indicates a feature concatenation operation. The fused feature... By combining the characteristics of the first branch after separation After performing pointwise multiplication, the network output during the encoding stage can be represented as:
[0058] In the formula, This indicates a pointwise multiplication operation. This represents the GELU activation function.
[0059] In the decoding stage, the encoded output and G-buffer information are used as conditional inputs. A Mamba network and SS2D module are combined for global feature extraction and spatiotemporal feature fusion. Depthwise separable convolutions reduce computational complexity, and features from preceding and following frames are introduced to enhance temporal consistency and reduce inter-frame jitter. For the network output in the encoding stage... First, the data is normalized through layers and then processed through linear layers. The G-buffer information is used as network input during the decoding stage, along with the initial features. Performing the same operation can be represented as:
[0060] In the formula Indicates a linear operation. The representation layer is normalized. Then, the features are first subjected to depthwise separable convolutions, which operate independently on each input channel without mixing channel information. This effectively extracts the local spatial information of the input feature map while reducing computational cost and the number of parameters. The features are then input into the SS2D module through an activation function. This process can be represented as:
[0061] In the formula This is represented as a depthwise separable convolution operation. It is represented as the SiLU activation function.
[0062] The SS2D (2D-Selective-Scan) module includes two main steps: scan expansion and scan merging, such as... Figure 3As shown in c. In the scan expansion operation, the input image is expanded into a series of sub-images, each corresponding to a specific direction. When viewed from a diagonal perspective, the scan expansion operation proceeds along four symmetrical directions: top-down, bottom-up, left-right, and right-left. This layout not only comprehensively covers all areas of the input image but also provides a rich multidimensional information foundation for subsequent feature extraction through system orientation transformation, thereby improving the efficiency and comprehensiveness of multidimensional image feature capture. Subsequently, during the scan merging process, the sub-images undergo feature extraction under the operation of the selective state-space model S6 module. Finally, these sub-images are merged through the scan merging operation to form an output image of the same size as the input image, represented as:
[0063] To enhance the fusion of spatiotemporal information between consecutive frames, this invention uses the decoded features output by the decoding network from the previous frame image. It also serves as the network input for the current image. After linear operations and activation, it is then combined with... The fusion process can be represented as:
[0064] Finally, the multi-scale and multi-modal features are fused to obtain the final decoding features for the decoding stage, which can be represented as:
[0065] For the specific architecture, please refer to... Figure 2 Design diagram of the cross-modal feature fusion module.
[0066] Finally, the reconstructed output is performed, and subpixel convolution technology is used to expand the receptive field of the model to generate high-resolution video frames with rich details and stable temporal sequence.
[0067] This invention integrates physical information and visual features to construct a dual-branch cross-modal fusion architecture, achieving high-fidelity reconstruction of key behavioral features in teaching videos. It also has the advantages of fewer parameters, lower latency, and privacy protection.
[0068] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative and not exhaustive. All modifications within the scope of this invention or its equivalents are included in this invention.
Claims
1. A method for super-resolution reconstruction of teaching videos based on cross-modal feature fusion, characterized in that, Includes the following steps: Step S1: Obtain the low-resolution teaching video frame sequence and its corresponding G-buffer information sequence; the G-buffer information includes normal information, BRDF information, depth information and motion vector information; Step S2: Input the current video frame into the encoding network. The encoding network first extracts initial features from the current video frame through convolutional layers to obtain initial features. Then, it performs separable convolution on the initial features and separates them into first branch features and second branch features. The first branch features are rearranged across channels. The rearranged features are concatenated with the initial features and then multiplied point-by-point with the second branch features to obtain the encoded features. Step S3: The encoded features and the G-buffer information are used as dual-branch inputs and input into the decoding network. In the decoding network, the two-dimensional selective scanning module SS2D of the Mamba network is used to perform global feature extraction and spatiotemporal feature fusion on the encoded features and the G-buffer information respectively. The decoded features output by the decoding network of the previous frame image are fused with the SS2D output features of the current frame to obtain the decoded features of the current frame. Step S4: Upsample the decoded features through subpixel convolution to generate a high-resolution video frame for the current frame.
2. The method for super-resolution reconstruction of teaching videos based on cross-modal feature fusion according to claim 1, characterized in that, The cross-channel feature interaction rearrangement in step S2 specifically includes: For the first branch feature, let its spatial height be H, width be W, and number of channels be C. The interaction is represented as follows: in, This indicates that after cross-channel feature interaction, in The feature sampled values of the location, where , , These represent the output height, width, and number of channels, respectively. The change in channels is represented as follows: In the formula, The number of channels for the output features. This represents the scaling factor.
3. The method for super-resolution reconstruction of teaching videos based on cross-modal feature fusion according to claim 1, characterized in that, The method for fusing the high-resolution features output by the decoding network of the previous frame with the SS2D output features of the current frame is as follows: after performing a linear transformation and activation function processing on the decoding output of the previous frame, the feature is multiplied point by point with the SS2D output features of the current frame.
4. The method for super-resolution reconstruction of teaching videos based on cross-modal feature fusion according to claim 1, characterized in that, In step S3, the decoding network's processing of the encoded features and G-buffer information specifically includes: performing layer normalization and linear transformation on the encoded features and G-buffer information respectively, and then inputting them into their respective SS2D modules after passing through depthwise separable convolution and SiLU activation function; fusing the outputs of the two SS2D modules with the decoded output of the previous frame after linear transformation and activation to obtain the decoded features of the current frame.
5. The method for super-resolution reconstruction of teaching videos based on cross-modal feature fusion according to claim 1, characterized in that, The SS2D module includes a scan expansion operation and a scan merging operation; the scan expansion operation is to expand the input features into multiple sub-images along at least two different directions; the scan merging operation is to merge each sub-image into an output feature with the same size as the input feature after processing by a selective state-space model.
6. The method for super-resolution reconstruction of teaching videos based on cross-modal feature fusion according to claim 5, characterized in that, The scan expansion operation is performed along four symmetrical directions, namely from top to bottom, from bottom to top, from left to right, and from right to left.
7. The method for super-resolution reconstruction of teaching videos based on cross-modal feature fusion according to claim 2, characterized in that, In step S2, the convolutional layer is a 3×3 convolutional layer.
8. The method according to claim 1, characterized in that, The normal information, BRDF information, depth information, and motion vector information in the G-buffer information are input into the decoding network at the same spatiotemporal resolution as the low-resolution video frame. They are then aligned with the encoded features in the channel dimension as conditional information and participate in the global feature extraction of the SS2D module.
9. The method according to claim 1, characterized in that, After the decoding network outputs the decoding features of the current frame, it also buffers them for temporal fusion of the next frame.
10. A super-resolution reconstruction system for teaching videos based on cross-modal feature fusion, characterized in that, For implementing any one of the methods as claimed in claims 1 to 8, comprising: Input module: used to acquire low-resolution teaching video frame sequences and their corresponding G-buffer information sequences, wherein the G-buffer information includes normal information, BRDF information, depth information and motion vector information; Encoding module: used to perform the operation of step S2 as described in claim 1 on the current video frame to obtain encoded features; Decoding module: used to take the encoded features and the G-buffer information as dual-branch inputs, execute the operation of step S3 as described in claim 1, and obtain the decoded features of the current frame; Reconstruction module: used to perform the operation of step S4 of claim 1 on the decoded features to generate a high-resolution video frame of the current frame.