A binocular frame generation method and system based on central feature flow

Abstract

The application discloses a binocular frame generation method and system based on central feature flow, comprising: extracting multi-source features from a current frame monocular low-resolution color image, current frame geometry buffer data and historical prediction image respectively; performing disparity offset prediction and consistency adjustment on left and right eye generation results through a centering operator to align the multi-source features to a central feature coordinate system, forming disparity-consistent central feature flow; inputting the aligned central feature flow of the current frame and the refined central feature flow cached from the previous frame into a recurrent feature network for time series modeling and feature fusion, and outputting the refined central feature flow of the current frame; performing inverse transformation on the refined central feature flow through the centering operator to generate high-resolution left and right eye output frames respectively, and caching the central feature flow of the current frame for next frame processing. The application can significantly reduce rendering calculation amount, improve stereoscopic vision quality and immersion, and is especially suitable for real-time stereoscopic display scenes such as VR / AR.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, specifically relating to a method and system for generating binocular frames based on a central feature flow. Background Technology

[0002] In recent years, neural network-based image super-resolution (SR) and video frame interpolation (FE) technologies have received widespread attention in computer graphics and game rendering, especially in improving real-time rendering efficiency, and have become an important direction for reducing the computational overhead of high-resolution and high-frame-rate rendering. In binocular display scenarios such as Virtual Reality (VR) and Augmented Reality (AR), rendering engines typically need to generate two images for the left and right eyes simultaneously to achieve stereoscopic vision. In traditional rendering workflows, the left and right eye frames are often rendered and post-processed separately. This means that even in acceleration methods based on super-resolution and frame interpolation, inference needs to be performed on both views separately, resulting in redundant calculations and accumulated latency.

[0003] In existing technologies, the paper "FuseSR: Super Resolution for Real-time Renderingthrough Efficient Multi-resolution Fusion" (Zhong et al., 2023) proposes an efficient super-resolution method based on multi-scale feature fusion, which can restore high-quality output on the basis of low-resolution rendering, thereby reducing the computational cost of pixel shading. Specifically, FuseSR fuses high-resolution geometric and material features generated in the rendering stage with image features of low-resolution frames at multiple levels, and uses residual prediction to compensate for high-frequency details. This method can significantly reduce the computational cost of pixel shading in the rendering stage at 4K output resolution and achieve near real-time inference speed on GPUs. However, while this method achieves a high balance between efficiency and quality in monocular rendering, in binocular scenes, the left and right eye images still need to be extracted and reconstructed separately, which cannot fully utilize the high correlation between binocular frames, resulting in inference redundancy and increased bandwidth consumption.

[0004] Another related work, the paper "Deep Fourier-based Arbitrary-scale Super-resolution for Real-time Rendering" (Zhang et al., 2024), proposes an arbitrary-magnification super-resolution method based on Fourier domain / frequency domain modules. This aims to break the dependence of traditional SR models on a fixed scale, enabling a single trained model to support arbitrary magnification, thereby improving its applicability and practical value at different output resolutions. DFASR makes design choices specifically for the needs of real-time rendering scenarios, achieving a balance between speed and scale flexibility.

[0005] While the above methods have achieved good results in accelerating single-view (monocular) rendering, significant technical bottlenecks still exist in application scenarios that require the simultaneous generation of left and right eye images, such as binocular or stereoscopic displays (e.g., virtual reality (VR), augmented reality (AR)).

[0006] (1) Computational redundancy (repeated inference): Current solutions often perform feature extraction, super-resolution, or frame interpolation inference separately for the left and right eye views, failing to fully utilize the highly overlapping spatial information between binocular views, resulting in a large number of binocular shared regions being repeatedly calculated. This not only increases the burden on video memory and bandwidth consumption, but also significantly reduces the overall computational efficiency, limiting the real-time performance and energy efficiency of the system.

[0007] (2) Risk of binocular consistency: Independent generation of left and right eye frames may lead to subtle differences in structure, texture or motion estimation, resulting in inconsistencies between binocular images, which in turn affect the stereoscopic visual experience and may cause discomfort or visual fatigue in viewers.

[0008] (3) Reduced latency and responsiveness: Since the reasoning processes for the two views are independent of each other, the computation time and data transmission latency will be superimposed. Especially in interactive scenarios with high frame rate and low latency requirements, this cumulative latency will seriously affect the system's response speed and user experience.

[0009] Therefore, in response to the actual needs of binocular rendering, there is an urgent need for a new technical solution that can jointly model the spatiotemporal feature information of the left and right viewpoints, realize shared inference and generate binocular frames for the left and right eyes at one time, instead of processing them independently, so as to significantly reduce redundant calculations, improve binocular consistency, reduce overall system latency, and further improve rendering speed and interactive response performance while ensuring image visual quality and stereoscopic effect. Summary of the Invention

[0010] In view of the above, the purpose of this invention is to provide a binocular frame generation method and system based on a central feature flow, aiming to solve the problems of computational redundancy and poor binocular consistency in existing neural network super-resolution and frame interpolation techniques in binocular rendering scenarios. With alternating input of low-resolution monocular rendering results, binocular frame synthesis is achieved based on disparity unification and a central feature flow, generating stable and high-fidelity high-resolution binocular rendering output. This method is particularly suitable for real-time rendering systems, such as virtual reality (VR) and augmented reality (AR) scenes. Under conditions of limited computing resources, it achieves high-quality binocular composite output through spatiotemporal information reuse and disparity unification strategies, effectively improving overall rendering speed and stability.

[0011] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0012] In a first aspect, the present invention provides a binocular frame generation method based on a central feature stream, comprising the following steps:

[0013] Multi-source features extracted from the current frame and historical frames are fused into a consistent central feature stream;

[0014] The central feature stream of the current frame and the central feature stream of the previous frame are input into the neural network for temporal modeling and feature fusion, and the refined central feature stream of the current frame is output.

[0015] High-resolution left and right eye output frames are generated based on the refined central feature stream.

[0016] Preferably, the step of fusing the multi-source features extracted from the current frame and historical frames into a consistent central feature stream includes:

[0017] By using a centering operator to predict disparity shift in the generated results for both the left and right eyes, the predicted binocular disparity is obtained. And obtain the pixel-wise importance weights of the geometry buffer prediction. As an eye identifier, calculate the alignment displacement. The centering operator utilizes alignment shift. A forward deformation operation is performed to uniformly align and map the multi-source features to the central feature coordinate system, forming a parallax-consistent central feature flow.

[0018] Preferably, the extraction of multi-source features includes:

[0019] Monocular features, geometric buffer features, and historical prediction features are extracted from the current frame's monocular low-resolution shading image, the current frame's geometric buffer data, and the historical prediction image, respectively. Specifically, the current frame's monocular low-resolution shading image is encoded to extract monocular features including texture and lighting features; the current frame's geometric buffer data obtained from the rendering pipeline is encoded to extract geometric buffer features including spatial and geometric features; and the previous frame's historical prediction image is encoded to extract historical prediction features including cross-frame texture residuals and temporal consistency features.

[0020] Preferably, the central feature stream is a multi-channel feature representation located in the central virtual viewpoint coordinate system, formed by performing parallax consistency processing on the generation results of the left and right eyes through a centering operator. It eliminates the coordinate offset caused by the difference in viewing angle of the original multi-source features.

[0021] Preferably, the step of inputting the central feature stream of the current frame and the central feature stream buffered in the previous frame into the neural network for temporal modeling and feature fusion, and outputting the refined central feature stream of the current frame, includes:

[0022] The central feature stream aligned to the current frame and the refined central feature stream buffered from the previous frame are input into the neural network. The neural network recursively reuses historical central feature information, i.e., the refined central feature stream buffered from the previous frame, in the time dimension, and fuses it with the central feature stream aligned to the current frame to compensate for the information loss caused by monocular alternating sampling, thereby obtaining the refined central feature stream of the current frame, thus maintaining the stability of the binocular output and the integrity of high-frequency details.

[0023] Preferably, the neural network adopts a recurrent neural network structure including ConvLSTM units and ResBlocks to achieve cyclic reuse of the central feature stream and gradual accumulation and optimization of cross-frame features.

[0024] Preferably, the step of generating high-resolution left and right eye output frames based on the refined central feature stream includes:

[0025] Based on alignment displacement The inverse transformation operation of the inverse deformation is performed by the centering operator, and the refined central feature stream is mapped back to the left eye coordinate system and the right eye coordinate system respectively to obtain the refined features of the left and right eyes. The refined features of the left and right eyes are upsampled and reconstructed by the dual-channel decoder to generate the final high-resolution left eye output frame and high-resolution right eye output frame, and the central feature stream of the current frame is cached for processing of the next frame.

[0026] Secondly, embodiments of the present invention also provide a binocular frame generation system based on a central feature flow, which is implemented using the above-mentioned binocular frame generation method based on a central feature flow, including: a central feature flow alignment module, a neural network fusion module, and a binocular frame generation module;

[0027] The central feature stream alignment module is used to fuse multi-source features extracted from the current frame and historical frames into a consistent central feature stream.

[0028] The neural network fusion module is used to input the central feature stream of the current frame and the central feature stream cached in the previous frame into the neural network for temporal modeling and feature fusion, and output the refined central feature stream of the current frame.

[0029] The binocular frame generation module is used to generate high-resolution left and right eye output frames based on the refined central feature stream.

[0030] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described binocular frame generation method based on a central feature stream when executing the computer program.

[0031] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described binocular frame generation method based on a central feature stream.

[0032] This invention proposes a binocular frame synthesis method based on disparity uniformity and central feature flow, applicable to real-time rendering scenarios requiring binocular stereoscopic display, such as virtual reality (VR) and augmented reality (AR). It is particularly suitable for generating high-fidelity, stable, and stereoscopically consistent binocular images under conditions of alternating input of low-resolution monocular rendering results. Compared with existing technologies, the beneficial effects of this invention include at least:

[0033] (1) Significantly reduces computational redundancy:

[0034] Existing binocular rendering methods typically perform complete feature extraction and super-resolution / frame interpolation inference on the left and right eye images separately, resulting in almost double the computational load and GPU memory usage. This invention uses a central feature flow mechanism to map and align the features of the left and right eyes to a unified central feature coordinate system, achieving feature-level cross-viewpoint reuse, significantly reducing redundant computation, and effectively reducing GPU memory usage and bandwidth pressure while ensuring image quality.

[0035] (2) Improve binocular consistency and stereo comfort:

[0036] By using parallax unification and centering operator, the features of the left and right eyes are compensated for parallax and geometrically aligned, thereby achieving high-quality fusion and refinement of binocular information. This significantly reduces structural differences and texture misalignment between the left and right eyes, fundamentally reducing stereoscopic visual discomfort caused by binocular inconsistency and improving comfort during long-term viewing.

[0037] (3) Enhance timing stability and dynamic response performance:

[0038] When only alternating low-resolution monocular frames are input, existing methods are prone to temporal instability phenomena such as flickering and drift. This invention introduces a Cyclic Network to recursively reuse and temporally model the central feature stream, achieving frame-by-frame optimization and seamless transfer of historical features, thereby maintaining the temporal smoothness and response stability of the rendering results even in dynamic scenes and high frame rate conditions.

[0039] (4) Balancing high-frequency details with low-frequency structure reproduction:

[0040] This method integrates multi-source features, including monocular features, geometric buffer features, and historical prediction features, during the construction of the central feature flow. This preserves the clarity of high-frequency detail textures while maintaining the consistency of low-frequency structure and illumination distribution, making the generated binocular images close to the real rendering effect in terms of both overall feel and detail representation.

[0041] (5) High adaptability and real-time performance:

[0042] The method of this invention can be directly embedded into existing real-time rendering pipelines. Due to the introduction of feature reuse, parallax uniformity and central feature flow mechanism, this method can significantly improve the frame rate while ensuring image quality, meeting the strict requirements of low latency and high refresh rate in highly interactive applications such as VR / AR.

[0043] In summary, this invention, through unified alignment of the central feature stream, reuse of cyclic features, and disparity-consistent generation strategies, not only solves the shortcomings of traditional binocular rendering methods in terms of computational efficiency, binocular consistency, and temporal stability, but also demonstrates good adaptability and efficiency across various hardware platforms and rendering scenarios, possessing significant practical value and promotion potential. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the binocular frame generation method based on central feature flow provided in an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram of the network framework of the binocular frame generation method based on the central feature flow provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the structure of a binocular frame generation system based on a central feature stream provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0049] Existing methods typically extract and process features separately for the left and right eye views, failing to fully utilize the high correlation between them. This leads to redundant computation and wasted resources. Furthermore, independently generated binocular frames are prone to structural and texture differences, affecting stereoscopic visual effects and user experience, especially in high-frame-rate interactive environments where latency accumulation is a significant issue. Therefore, this invention addresses the bottlenecks in existing binocular rendering acceleration technologies, such as redundant inference, insufficient visual consistency, and high system latency. This invention provides a binocular frame generation method and system based on a central feature flow. By constructing a central virtual viewpoint coordinate system, monocular features, geometric buffer features, and historical prediction features are uniformly aligned to the central feature flow. Combined with a recurrent feature network, cross-frame feature reuse is achieved, significantly reducing redundant computation. Simultaneously, the central feature flow and recurrent reuse mechanism effectively solve the temporal instability problem when alternating monocular frame inputs. Through binocular frame synthesis and disparity consistency strategies, this method significantly improves stereoscopic visual quality and overall immersion while maintaining rendering speed, making it particularly suitable for stereoscopic display scenarios with stringent real-time requirements, such as VR / AR.

[0050] like Figure 1 and Figure 2 As shown, the embodiment provides a method for generating stereo frames based on a central feature stream, including the following steps:

[0051] S1 fuses the multi-source features extracted from the current frame and historical frames into a consistent central feature stream.

[0052] S1.1 Extract monocular features, geometric buffer features, and historical prediction features from the current frame monocular low-resolution colored image, the current frame geometric buffer data, and the historical prediction image respectively to obtain multi-source features.

[0053] In this embodiment, the system input and scene configuration are such that at any given time, only a monocular low-resolution shaded image and a binocular geometry and material buffer (GBuffer), including depth, normals, material identifiers, etc., are received, while the high-resolution prediction frame from the previous moment is retained as a historical reference. The scene can be a dynamic or static 3D environment, containing complex geometry and various material distributions (such as diffuse, semi-gloss, specular, etc.), and supports real-time changes in camera position and orientation.

[0054] The current frame's monocular low-resolution shaded image is encoded using a dedicated image encoder to extract monocular features, including texture and lighting characteristics. Simultaneously, the current frame's geometry buffer (GBuffer) data is obtained from the rendering pipeline and encoded using a separate encoding network to extract geometry buffer features, including spatial and geometric characteristics. Furthermore, the historical prediction image from the previous frame is encoded to extract historical prediction features, including cross-frame texture residuals and temporal consistency features. These three types of features constitute multi-source features, containing complementary information from the texture, geometry, and temporal domains, respectively.

[0055] S1.2 uses a centering operator to predict and adjust the disparity offset of the generated results for the left and right eyes, so as to align the multi-source features to the central feature coordinate system and form a disparity-consistent central feature flow.

[0056] In this embodiment, a Central Feature Stream is introduced as the core of global alignment and fusion. As a common representation for subsequent cross-temporal and cross-viewpoint information fusion, it effectively eliminates geometric misalignment caused by time or viewpoint differences and significantly improves computational efficiency. Disparity offset prediction and consistency adjustment are performed on the left and right eye generation results to reduce stereo matching errors and geometric distortions. Monocular features, geometric buffer features, and historical prediction features are mapped and aligned to the central feature coordinate system, thereby eliminating disparity differences and coordinate offsets between different sources in the feature space. Specifically, the disparity consistency operation first involves a small convolutional neural network (UC) receiving the predicted binocular disparity obtained by the Centering Operator through disparity offset prediction of the left and right eye generation results. Pixel-wise importance weights predicted by the geometry buffer And calculate the alignment displacement. The left eye right eye The CenteringOperator operates through forward warping. Multi-source features Aligning the geometric buffer features and historical prediction features to the central feature coordinate system, the same method is used to align them, resulting in the aligned central feature flow. , The importance weights are assigned pixel-wise. The resulting central feature stream is a multi-channel feature representation located in the central virtual viewpoint coordinate system, formed by applying a centering operator to the disparity consistency of the left and right eye generation results. This eliminates the coordinate offset caused by viewpoint differences in the original multi-source features. This process not only achieves spatial alignment but also enhances the ability to express complex lighting and detailed regions through a multi-scale feature fusion mechanism.

[0057] S2 inputs the central feature stream of the current frame and the central feature stream of the previous frame into the neural network for temporal modeling and feature fusion, and outputs the refined central feature stream of the current frame.

[0058] In this embodiment, a cyclic feature network composed of ConvLSTM and ResBlock is used to perform temporal modeling and recursive reuse of the central feature stream. Specifically, the cyclic feature network takes as input the central feature of the current frame aligned by the Centering Operator and the refined central feature output from the cyclic feature network of the previous frame, and outputs the refined central feature of the current frame, thereby realizing the cyclic reuse and updating of the central feature stream and achieving the gradual accumulation and optimization of features across frames. This process can significantly reduce the temporal discontinuity problem caused by the alternation of monocular inputs and ensure the stability of binocular frame generation under long sequences.

[0059] S3 generates high-resolution left and right eye output frames based on the refined central feature stream.

[0060] In the embodiment, due to the reversibility of the Centering Operator, the central feature flow in the central feature coordinate system... It can be done through backward warping. Inverse transformation back to the left and right eye coordinate systems yields the refined left and right eye features. The system then uses a dual-channel decoder to upsample and reconstruct the refined features of the left and right eyes separately, generating the final high-resolution output frames for the left and right eyes. The rendering results for the left and right eyes are generated separately using the Centering Operator inverse transform and the dual-channel decoder. The binocular frames generated from a unified central feature stream not only improve binocular consistency and stereoscopic visual comfort but also ensure accuracy in edge details and high-frequency texture areas. Finally, the generated high-resolution frames for both eyes are output to the display device. The current central feature stream is buffered and used as input for recursive reuse of historical features in the next frame, achieving efficient cross-frame transfer of feature information.

[0061] Through the above steps, this embodiment of the invention achieves the generation of high-resolution, high-fidelity, and geometrically consistent binocular frames under the premise of only alternately receiving low-resolution monocular rendering results. It solves the problems of texture flicker, parallax drift, and lack of detail that exist in traditional methods in alternating sampling scenarios. It is especially suitable for deployment on hardware platforms with limited computing resources, achieving a balance between performance and image quality.

[0062] Furthermore, to meet real-time rendering requirements, this embodiment employs lightweight convolution and feature caching mechanisms during central feature alignment and cyclic feature reuse to reduce redundant computation. During training, L1 Loss is used as the color consistency loss and SSIM as the geometric constraint loss to supervise the reconstructed frames and ground truth, ensuring the stability of the generated results in the disparity and temporal dimensions during inference. This design enables the method of this invention to maintain high image quality while possessing low latency characteristics.

[0063] Based on the same inventive concept, such as Figure 3 As shown, this embodiment of the invention also provides a binocular frame generation system 300 based on a central feature stream, including: a central feature stream alignment module 310, a neural network fusion module 320, and a binocular frame generation module 330.

[0064] The central feature flow alignment module 310 is used to fuse multi-source features extracted from the current frame and historical frames into a consistent central feature flow.

[0065] The network fusion module 320 is used to input the central feature stream of the current frame and the central feature stream of the previous frame into the neural network for temporal modeling and feature fusion, and output the refined central feature stream of the current frame.

[0066] The binocular frame generation module 330 is used to generate high-resolution left and right eye output frames based on the refined central feature stream.

[0067] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described binocular frame generation method based on a central feature stream when executing the computer program.

[0068] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described binocular frame generation method based on a central feature stream.

[0069] It should be noted that the binocular frame generation system, electronic device, and computer-readable storage medium based on the central feature stream provided in the above embodiments all belong to the same inventive concept as the binocular frame generation method based on the central feature stream. For details of their specific implementation process, please refer to the embodiments of the binocular frame generation method based on the central feature stream, which will not be repeated here.

[0070] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating binocular frames based on a central feature stream, characterized in that, Includes the following steps: The multi-source features extracted from the current frame and historical frames are fused into a consistent central feature stream. The central feature stream is a multi-channel feature representation located in the central virtual viewpoint coordinate system, which is formed after the disparity consistency processing of the left and right eye generation results by the centering operator. It eliminates the coordinate offset of the original multi-source features caused by the difference in viewing angle. The central feature stream of the current frame and the central feature stream of the previous frame are input into the neural network for temporal modeling and feature fusion, and the refined central feature stream of the current frame is output. High-resolution left and right eye output frames are generated based on the refined central feature stream.

2. The binocular frame generation method based on central feature flow according to claim 1, characterized in that, The process of fusing multi-source features extracted from the current frame and historical frames into a consistent central feature stream includes: By using a centering operator to predict disparity shift in the generated results for both the left and right eyes, the predicted binocular disparity is obtained. And obtain the pixel-wise importance weights of the geometry buffer prediction. As an eye identifier, calculate the alignment displacement. The centering operator utilizes alignment shift. A forward deformation operation is performed to uniformly align and map the multi-source features to the central feature coordinate system, forming a parallax-consistent central feature flow.

3. The binocular frame generation method based on central feature flow according to claim 1 or 2, characterized in that, Multi-source feature extraction includes: Monocular features, geometric buffer features, and historical prediction features are extracted from the current frame's monocular low-resolution shading image, the current frame's geometric buffer data, and the historical prediction image, respectively. Specifically, the current frame's monocular low-resolution shading image is encoded to extract monocular features including texture and lighting features; the current frame's geometric buffer data obtained from the rendering pipeline is encoded to extract geometric buffer features including spatial and geometric features; and the previous frame's historical prediction image is encoded to extract historical prediction features including cross-frame texture residuals and temporal consistency features.

4. The binocular frame generation method based on central feature flow according to claim 1, characterized in that, The step of inputting the central feature stream of the current frame and the central feature stream buffered from the previous frame into a neural network for temporal modeling and feature fusion, and outputting the refined central feature stream of the current frame, includes: The central feature stream aligned to the current frame and the refined central feature stream buffered from the previous frame are input into the neural network. The neural network recursively reuses historical central feature information, i.e., the refined central feature stream buffered from the previous frame, in the time dimension, and fuses it with the central feature stream aligned to the current frame to compensate for the information loss caused by monocular alternating sampling, thereby obtaining the refined central feature stream of the current frame, thus maintaining the stability of the binocular output and the integrity of high-frequency details.

5. The binocular frame generation method based on central feature flow according to claim 1 or 4, characterized in that, The neural network employs a recurrent neural network structure including ConvLSTM units and ResBlocks to achieve cyclic reuse of the central feature stream and gradual accumulation and optimization of cross-frame features.

6. The binocular frame generation method based on central feature flow according to claim 2, characterized in that, The generation of high-resolution left and right eye output frames based on the refined central feature stream includes: Based on alignment displacement The inverse transformation operation of the inverse deformation is performed by the centering operator, and the refined central feature stream is mapped back to the left eye coordinate system and the right eye coordinate system respectively to obtain the refined features of the left and right eyes. The refined features of the left and right eyes are upsampled and reconstructed by the dual-channel decoder to generate the final high-resolution left eye output frame and high-resolution right eye output frame, and the central feature stream of the current frame is cached for processing of the next frame.

7. A stereo frame generation system based on a central feature flow, implemented using the stereo frame generation method based on a central feature flow as described in any one of claims 1 to 6, characterized in that, include: The module includes a central feature flow alignment module, a neural network fusion module, and a binocular frame generation module. The central feature stream alignment module is used to fuse multi-source features extracted from the current frame and historical frames into a consistent central feature stream. The neural network fusion module is used to input the central feature stream of the current frame and the central feature stream cached in the previous frame into the neural network for temporal modeling and feature fusion, and output the refined central feature stream of the current frame. The binocular frame generation module is used to generate high-resolution left and right eye output frames based on the refined central feature stream.

8. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is used to implement the binocular frame generation method based on the central feature stream as described in any one of claims 1 to 6 when executing a computer program.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the binocular frame generation method based on the central feature stream as described in any one of claims 1 to 6.

Images