Offline interleaving coding methods, equipment and media for multi-view video

By using a device mask to rearrange multi-view video into interleaved frames and encode them into a standard format during the offline stage, the computational load and stability issues of multi-view video interleaving on high-resolution screens or low-power terminals are solved, achieving efficient video playback.

CN123123574APending Publication Date: 2026-11-17SHENZHEN IMAGETRU3D TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611614784.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-19
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

When performing multi-view video interleaving on high-resolution screens or low-power embedded terminals, existing technologies consume a lot of resources for real-time graphics processing, resulting in dropped frames and black frames, and the data copying from video memory to RAM severely reduces processing efficiency.

Method used

By using a device mask in the offline stage to rearrange the multi-viewpoint information in the source video frame into interlaced frames that conform to the optical display rules, and directly encoding them into a standard video format to generate pre-interlaced video, the terminal does not need to perform complex real-time interlacing calculations during playback, but only needs to call the hardware decoder for output.

Benefits of technology

It reduces the real-time computing load on the terminal, improves processing efficiency, avoids data copying overhead, and maintains timeline consistency before and after interleaving, ensuring the stability and smoothness of video playback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123574A_ABST
    Figure CN123123574A_ABST
Patent Text Reader

Abstract

This application relates to the field of video processing technology, and provides an offline interleaving encoding method, apparatus, and computer-readable storage medium for multi-viewpoint video. To reduce the burden of real-time processing on the terminal, this application moves the device-related viewpoint interleaving from the terminal runtime to the offline stage. That is, the interleaved video is pre-generated during the content production stage, so that the target terminal only needs to perform video decoding and display during playback. Furthermore, by using a device mask in the offline stage to rearrange the multi-viewpoint information in the source video frames into interleaved frames conforming to optical display principles, and directly encoding them into a standard video format, the terminal no longer needs to perform complex real-time interleaving calculations during playback; it only needs to call the hardware decoder for output. This reduces the real-time computing load on the terminal and improves processing efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to an offline interleaving coding method for multi-view video, a computer device, and a computer-readable storage medium. Background Technology

[0002] In glasses-free 3D display technology, directional optical display devices such as lenticular lenses or parallax barriers need to interleave multiple viewpoint images into the same output image according to the correspondence between optical components and display sub-pixels. Currently, interleaving and colorization are typically performed frame-by-frame in real-time during video playback on the terminal, requiring the terminal to run multiple high-load tasks simultaneously. For high-resolution screens or low-power embedded terminals, this real-time graphics processing method not only consumes a large amount of graphics processor resources but is also prone to frame drops and black frames due to thread scheduling or device differences. Furthermore, if the frames output by the graphics processor are read back to the central processing unit's memory frame by frame before encoding, a large amount of data copying from video memory to main memory will occur, severely reducing processing efficiency. Therefore, how to provide a multi-viewpoint video interleaving encoding scheme that can reduce the real-time computing load of the terminal and improve processing efficiency is a technical problem that urgently needs to be solved. Summary of the Invention

[0003] The main objective of this invention is to provide an offline interleaving encoding method, apparatus, device, and computer-readable storage medium for multi-view video, aiming to reduce the real-time computing load on the terminal and improve processing efficiency.

[0004] To achieve the above objectives, the present invention provides an offline interleaving encoding method for multi-view video, wherein the premium pricing method includes the following steps: A source multi-view video is acquired, and a video decoder and an external texture are created. The video decoder has an output surface. The output surface of the video decoder is bound to the external texture. The source multi-view video is input into the video decoder for decoding. The video decoder outputs each decoded source video frame and writes each decoded source video frame into the external texture via the output surface to obtain the source video frame located in the external texture. The source video frame includes multiple viewpoint images arranged in a preset manner. Determine the interleaving frame to be constructed corresponding to each of the source video frames, wherein the interleaving frame to be constructed includes a plurality of target output pixels, and each of the target output pixels includes a plurality of color components; Obtain a device mask corresponding to the target directional optical display device; the device mask is used to characterize the target viewpoint corresponding to each color component of the target output pixel; For each color component of each target output pixel, the target viewpoint corresponding to each color component is determined according to the device mask, and the corresponding viewpoint image region is determined in the source video frame according to the target viewpoint and the preset arrangement. From the viewpoint image region, sample at the relative position corresponding to the position of the target output pixel in the interlaced frame to be constructed to obtain the component value of the color component, which is used as the interlacing value of the color component of the target output pixel; Based on the interleaving values ​​of the color components of each of the target output pixels, a target interleaved frame is formed, and multiple target interleaved frames are video encoded and encapsulated to generate a pre-interleaved video.

[0005] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a processor, a memory, and an offline interleaving encoding program for multi-view video stored in the memory and executable by the processor, wherein when the offline interleaving encoding program for multi-view video is executed by the processor, the steps of the offline interleaving encoding method for multi-view video as described above are implemented.

[0006] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an offline interleaving encoding program for multi-view video, wherein when the offline interleaving encoding program for multi-view video is executed by a processor, it implements the steps of the offline interleaving encoding method for multi-view video as described above.

[0007] This application provides an offline interleaving encoding method for multi-view video. The method involves acquiring source multi-view video and creating a video decoder and an external texture. The video decoder has an output surface. The output surface of the video decoder is bound to the external texture. The source multi-view video is input into the video decoder for decoding. The video decoder outputs each decoded source video frame and writes each decoded source video frame to the external texture via the output surface, resulting in a source video frame located in the external texture. The source video frame includes multiple viewpoint images arranged in a preset manner. An interleaving frame to be constructed is determined corresponding to each source video frame. The interleaving frame to be constructed includes multiple target output pixels, and each target output pixel includes multiple color components. The method also involves acquiring the target... A device mask corresponding to a directional optical display device is provided. The device mask is used to characterize the target viewpoint corresponding to each color component of the target output pixel. For each color component of each target output pixel, the target viewpoint corresponding to each color component is determined according to the device mask, and the corresponding viewpoint image region is determined in the source video frame according to the target viewpoint and the preset arrangement. From the viewpoint image region, the component value of the color component is sampled at the relative position corresponding to the position of the target output pixel in the interlaced frame to be constructed, and the component value of the color component is used as the interlacing value of the color component of the target output pixel. Based on the interlacing values ​​of each color component of each target output pixel, a target interlaced frame is constructed, and multiple target interlaced frames are video encoded and encapsulated to generate a pre-interlaced video. In this way, this application aims to reduce the burden of real-time processing of the terminal by moving the device-related viewpoint interlacing from the terminal runtime to the offline stage, that is, pre-generating the interlaced video in the content production stage, so that the target terminal only needs to perform video decoding and display when playing it. Furthermore, by using a device mask in the offline stage to rearrange the multi-viewpoint information in the source video frame into interlaced frames that conform to the optical display rules, and directly encode them into a standard video format, the terminal no longer needs to perform complex real-time interlacing calculations during playback. It only needs to call the hardware decoder for output, thereby reducing the real-time computing load of the terminal and improving processing efficiency. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the first embodiment of the offline interleaving encoding method for multi-view video according to the present invention; Figure 2 This is a schematic diagram illustrating the selection and sampling of viewpoint regions for each color component according to the corresponding mask channel in this invention; Figure 3 This is a schematic diagram of the processing timing of the offline interleaving coding method for multi-view video of the present invention; Figure 4This is a schematic diagram of the hardware structure of the computer device involved in the embodiment of the present invention.

[0009] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0012] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0013] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0014] In glasses-free 3D display technology, directional optical display devices such as lenticular lenses or parallax barriers need to interweave multiple viewpoint images into the same output image according to the correspondence between optical devices and display sub-pixels. Currently, interweaving and colorization are typically performed frame-by-frame in real-time when playing video on the terminal, which requires the terminal to run multiple high-load tasks simultaneously. For high-resolution screens or low-power embedded terminals, this real-time graphics processing method not only consumes a large amount of graphics processor resources but is also prone to frame drops and black frames due to thread scheduling or device differences. Furthermore, if the frames output by the graphics processor are read back to the central processing unit's memory frame by frame for encoding, it will generate a large amount of data copying from video memory to main memory, severely reducing processing efficiency.

[0015] To address the aforementioned issues, this embodiment reduces the burden of real-time processing on the terminal by moving the device-related viewpoint interleaving from the terminal runtime stage to the offline stage. That is, the interleaved video is pre-generated during the content production stage, so that the target terminal only needs to perform video decoding and display when playing it.

[0016] Furthermore, to further avoid the practice of reading the output of the graphics processing unit (GPU) frame by frame back into memory before sending it to the encoder for offline preprocessing, which results in data copying between the GPU and memory—an unacceptable overhead at high resolutions—a new approach is adopted. This involves using a device mask to rearrange the multi-viewpoint information in the source video frames into interlaced frames that conform to optical display principles during the offline stage, and directly encoding them into a standard video format. This eliminates the need for complex real-time interlacing calculations during playback; the terminal only needs to call the hardware decoder for output.

[0017] The offline interleaving coding method for multi-view video involved in the embodiments of the present invention is mainly applied to computer devices, which can be PCs, laptops, mobile terminals, or other devices with display and processing functions.

[0018] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0019] Reference Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the offline interleaving encoding method for multi-view video according to the present invention.

[0020] like Figure 1 As shown, this embodiment of the invention provides an offline interleaving coding method for multi-view video, which includes steps S10 to S40.

[0021] Step S10: Acquire source multi-view video and create a video decoder and an external texture. The video decoder has an output surface. Bind the output surface of the video decoder to the external texture. Input the source multi-view video into the video decoder for decoding. The video decoder outputs each decoded source video frame and writes each decoded source video frame into the external texture via the output surface to obtain the source video frame located in the external texture. The source video frame includes multiple viewpoint images arranged in a preset arrangement. The resolution of the device mask is the same as the output resolution of the target orientation optical display device, or it can be mapped to the output resolution according to a preset sampling method.

[0022] Currently, multi-view video is typically presented through real-time playback on the terminal. During playback, the target terminal needs to sequentially perform video decoding, external texture conversion, mask sampling interleaving, and screen output for each video frame. This method requires the terminal to continuously perform video and graphics processing while playing the video. For high-resolution screens or low-power embedded terminals, real-time graphics processing consumes a significant amount of graphics processing resources; furthermore, it is susceptible to frame drops, black frames, and unstable frame times due to factors such as thread scheduling and texture synchronization.

[0023] Therefore, to reduce the burden of real-time processing on the terminal, this embodiment moves the device-related viewpoint interleaving from the terminal runtime to the offline stage. That is, the interleaved video is pre-generated during the content production stage, so that the target terminal only needs to perform video decoding and display during playback. However, if offline preprocessing is simply implemented by reading the output of the graphics processing unit frame by frame back to memory and then sending it to the encoder, data copying from the graphics processing unit to memory will occur, and this copying overhead is unacceptable under high-resolution conditions. Furthermore, if the readback path does not correctly transmit the display timestamp of the decoded frame, it will cause deviations in the preprocessed audio-visual rhythm and video segment duration. To further improve data processing efficiency and reduce system resource overhead, a zero-copy transmission mechanism is used to decode the source multi-view video. This involves creating a video decoder and an external texture, where the video decoder has an output surface; binding the output surface of the video decoder to the external texture; inputting the source multi-view video into the video decoder for decoding; outputting each decoded source video frame from the video decoder; and writing each decoded source video frame to the external texture via the output surface to obtain the source video frame located in the external texture.

[0024] Therefore, this embodiment provides a multi-viewpoint video offline interleaving encoding method for directional optical displays, which at least solves the problems of computational load and playback stability caused by real-time viewpoint interleaving on the terminal. On this basis, it avoids the data copying overhead caused by frame-by-frame readback during offline interleaving encoding and maintains the consistency of the time axis before and after interleaving, thereby generating a pre-interleaved video that can be directly played by the target terminal.

[0025] Directional optical display devices, such as those employing optical films like lenticular lens gratings or parallax barriers, can project images from different viewpoints onto different viewing directions, allowing viewers at different locations to see the corresponding viewpoint image. To ensure correct display of multi-viewpoint content on these devices, multiple viewpoint images need to be interleaved into the same output image according to the correspondence between the color sub-pixels of the output pixels and the viewpoints. These devices, after factory manufacturing or installation calibration, provide a device mask for mask sampling. The channel values ​​of this device mask characterize the viewpoint corresponding to the output color sub-pixels. Multi-viewpoint images can be sampled and interleaved into the output image based on this device mask.

[0026] Specifically, source multiview video refers to an original video stream containing information from multiple perspectives, where each frame (i.e., the source video frame) integrates multiple viewpoint images in a grid or other preset arrangement.

[0027] First, the source multi-view video is decapsulated to extract the compressed frame data stream. This stream is then fed into a hardware-accelerated decoder or software decoder, where decompression is performed according to the corresponding video encoding standard, restoring the compressed frame data to the original pixel data, such as a pixel matrix in YUV or RGB format. Taking the source multi-view video using the H.265 encoding standard as an example, the decoder sequentially performs inter-frame prediction, inverse transform, and pixel reconstruction, outputting a complete pixel matrix for each frame. The decoded source video frames are written to the graphics processor's video memory or shared memory, maintaining the complete structure of their multi-view composite frames for subsequent steps of viewpoint image localization and sampling.

[0028] A media extractor can also be used to determine the video track, source resolution, frame rate, and encoding type of the source multiview video, and create a corresponding video decoder accordingly. The source multiview video can then be decapsulated based on the video decoder.

[0029] The source video frame output after decoding the multi-view video source is not a typical single-view 2D image, but a composite frame structure integrating data from multiple viewpoints. The preset arrangement can be a grid-like topology, for example, tightly stitching N viewpoint images together in an R-row, C-column configuration within the same frame, where N is no greater than the product of R and C. This grid arrangement allows a single frame to carry complete light field or multi-viewpoint information, facilitating batch addressing by subsequent processing units.

[0030] It should be understood that although this embodiment uses a rectangular grid as an example for illustration, in other embodiments, the preset arrangement can also be an irregular arrangement, hexagonal tiling, or other custom spatial layout, as long as each independent viewpoint image region can be located in the decoded frame data.

[0031] The process includes executing a first rendering pass after obtaining the source video frame located in the external texture and before determining the viewpoint image region. A two-dimensional texture is created as a two-dimensional frame buffer texture, the resolution of which is the same as the resolution of the source video frame; The source video frame in the external texture is drawn onto the two-dimensional frame buffer texture according to the texture transformation matrix to obtain the two-dimensional texture of the source video frame. The texture transformation matrix only flips the vertical component of the texture coordinates, while the horizontal direction of the texture coordinates remains unchanged.

[0032] In this embodiment, a video decoder object and an external texture object are first instantiated. The video decoder can be a hardware-accelerated media codec component, configured with an output surface for outputting decoded image data. The external texture is a special type of graphics texture object, such as a GL_TEXTURE_EXTERNAL_OES type texture created in an OpenGL ES environment, which receives image streams from external sources such as the video decoder. Subsequently, a binding operation is performed, passing the output surface of the video decoder as a parameter to the external texture, or associating the two through an intermediate surface container object. This binding action establishes a direct data link from the video decoding unit to the graphics processing unit at the underlying level.

[0033] When the multi-view video source is fed into the video decoder, the decoder does not copy the pixel data to the application memory of the central processing unit when it completes the decoding of a frame. Instead, it directly writes the decoded YUV or RGB data to the graphics memory area corresponding to the external texture bound to the output surface. When the decoded output buffer is released and rendering is specified, the decoded image is written to the external texture.

[0034] Because the memory blocks pointed to by external textures are directly addressable and sampled by the graphics processor, the entire data transfer process is completed at the hardware level, without the central processing unit (CPU) involved in any pixel-level data movement or format conversion. This allows the decoded source video frames to reside in the graphics processor's video memory or shared memory with extremely low latency, ready to be read by subsequent rendering pipelines at any time.

[0035] Step S20: Determine the interleaving frame to be constructed corresponding to each of the source video frames. The interleaving frame to be constructed includes multiple target output pixels, and each target output pixel includes multiple color components. Specifically, the interleaved frame to be constructed refers to a logical data object reserved in memory or video memory for storing the final output result. It is the output frame adapted to the target-oriented optical display device, and the different color components of each target output pixel may originate from different viewpoint images. The resolution and format of the interleaved frame to be constructed are matched with the physical screen parameters of the target-oriented optical display device. The target output pixel refers to the smallest addressable unit in the interleaved frame to be constructed, corresponding to a physical pixel position on the screen of the target-oriented optical display device. Color components refer to the sub-pixel channel data that constitute the target output pixel, such as the red, green, and blue components.

[0036] Based on the screen resolution parameters of the target directional optical display device, a frame buffer with the same screen resolution is allocated in memory or video memory as the storage space for the interleaved frame to be constructed, and the frame buffer is initialized to a blank state, waiting for subsequent steps to fill the interleaved value pixel by pixel.

[0037] Unlike regular video frames, each target output pixel in the interleaved frame to be constructed does not directly correspond to a single pixel in the source video, but needs to be recombined according to the characteristics of the optical devices. Each target output pixel contains multiple color components (such as red, green, and blue components), which may originate from different viewpoint images in the source video frame during the subsequent interleaving process. Through the above steps, the target dimension of data processing, i.e., the multi-viewpoint set at the source end, is logically mapped to the single-frame interleaved graph at the terminal.

[0038] Wherein, the position of the target output pixel in the interleaving frame to be constructed is represented by normalized horizontal coordinates and normalized vertical coordinates, which are coordinates obtained by normalizing the position of the target output pixel in the interleaving frame to be constructed to between zero and one; The relative position is represented by a relative horizontal coordinate and a relative vertical coordinate; the relative horizontal coordinate is equal to the normalized horizontal coordinate divided by the preset number of columns, plus the value obtained by dividing the column number by the preset number of columns; the relative vertical coordinate is equal to the normalized vertical coordinate divided by the preset number of rows, plus the value obtained by dividing the row number by the preset number of rows; The component values ​​of the color components are sampled from the viewpoint image regions according to the relative horizontal and vertical coordinates, wherein each viewpoint image region is sampled at the same relative position.

[0039] Step S30: Obtain the device mask corresponding to the target directional optical display device; the device mask is used to characterize the target viewpoint corresponding to each color component of the target output pixel; Specifically, a device mask is an index map describing the optical characteristics of a target display device, whose channel values ​​indicate the viewpoint source from which the output sub-pixels should sample. A device mask is an index map of optical characteristics, not a visual image used for display. The resolution of the device mask is typically consistent with or proportional to the interleaved frame being constructed, and the channel values ​​(such as the values ​​of the RGB channels) at each pixel location encode the viewpoint source information that each color sub-pixel at that location should sample. For example, the red channel value of a pixel in the mask might indicate that the red sub-pixel at that location should display information from the 5th viewpoint, while the green channel value might indicate that information from the 6th viewpoint should be displayed. The target viewpoint refers to the specific viewpoint image assigned to a particular color component by the device mask among multiple viewpoint images of the source multi-view video.

[0040] Specifically, the mask image data can be read from the device configuration file and uploaded to the texture memory of the graphics processor as the input texture for subsequent shader sampling. Simultaneously, the mask's metadata information, including mask resolution, channel bit depth, and subpixel arrangement identifier, is also read to provide necessary parameter support for viewpoint mapping in subsequent steps.

[0041] After completing the resource preparations described above, the rendering process in this embodiment includes two channels. The first rendering channel converts the external texture into a standard two-dimensional texture according to the texture transformation matrix to adapt to the sampling characteristics of the graphics rendering pipeline and correct coordinate system differences. The specific implementation of this process will be described in detail in subsequent embodiments. The second rendering channel uses a device mask and a two-dimensional texture to perform viewpoint determination and sampling operations on each color component of each target output pixel, generating interleaved frames and directly drawing them onto the input surface of the video encoder. Steps S40 and S50 correspond to the specific processing procedures of the second rendering channel.

[0042] This embodiment introduces a device mask to convert the physical parameters of complex optical lenses or parallax barriers into a digital lookup table, enabling the interleaving algorithm to complete the correct viewpoint allocation based solely on the mask data.

[0043] Step S40: For each color component of each target output pixel, determine the target viewpoint corresponding to each color component according to the device mask, and determine the corresponding viewpoint image region in the source video frame according to the target viewpoint and the preset arrangement. Specifically, the viewpoint image region refers to the rectangular area occupied by the viewpoint image corresponding to the target viewpoint in the source video frame, and its position is determined by a preset arrangement. First, the channel value corresponding to the current color component in the device mask is analyzed, and this channel value is converted into a discrete viewpoint index to determine the target viewpoint. Subsequently, based on the known preset arrangement (such as the number of rows R and columns C mentioned above), mathematical mapping methods such as modular arithmetic or lookup tables are used to convert the one-dimensional viewpoint index into two-dimensional row and column numbers, thereby accurately locating the rectangular area occupied by the target viewpoint in the grid structure of the source video frame.

[0044] Using the pixel coordinates of the target output pixel in the interlaced frame to be constructed as an index, the channel value at the corresponding position is read from the device mask texture. Then, the channel to be read for the current color component is determined according to the sub-pixel arrangement identifier. The read normalized floating-point value is converted into an integer viewpoint number through quantization calculation. Then, a two-dimensional coordinate mapping is performed according to the grid parameters of the source video frame to finally obtain the row and column numbers of the viewpoint image region corresponding to the color component in the source video frame.

[0045] By using the above method, precise addressing from the mask index to the source data address is achieved, ensuring that each color component can find the correct data source.

[0046] Step S50: Sample the color component values ​​from the viewpoint image region at the relative position corresponding to the position of the target output pixel in the interleaved frame to be constructed, and use them as the interleaving values ​​of the color components of the target output pixel. Specifically, the interleaving value refers to the color component value sampled from the viewpoint image region of the source video frame, which serves as the final output value of the corresponding color component of the target output pixel. The sampling operation follows the spatial alignment principle. Assuming that the target output pixel is located at the normalized coordinates (u, v) of the frame to be interleaved, and a certain color component of this pixel is assigned to the viewpoint image region of the i-th column and j-th row in the source video frame, then the local relative coordinates of the sampling point within this viewpoint image region are also (u, v) or its linear transformation form.

[0047] The frame to be interleaved is treated as a frame buffer object. The full-screen quadrilateral is rasterized using the graphics rendering pipeline. In the fragment shader, device mask channel value reading, viewpoint index quantization, 2D mesh coordinate mapping, and texture sampling within the viewpoint image region are performed sequentially. Finally, the sampled color component values ​​are written to the corresponding pixel positions in the frame buffer. Once the fragment shaders for all pixels have finished executing, a complete target interleaved frame is stored in the frame buffer.

[0048] If adjacent pixels in an interleaved frame point to the same viewpoint, then the sampling points of those adjacent pixels in the source viewpoint image are also adjacent, thus ensuring the geometric continuity and correctness of the reconstructed viewpoint view.

[0049] Sampling can be achieved through texture samplers in the graphics rendering pipeline, direct memory access of compute shaders, or CPU-based software loops.

[0050] Specifically, such as Figure 2 As shown, each color component of the target output pixel determines the target viewpoint based on its corresponding device mask channel value. This determines the corresponding viewpoint image region within the R×C source frame on the left, and sampling is performed from the relative position within that region corresponding to the target output pixel's location. The three components can be sampled from different viewpoint images, thus merging to form a sub-pixel-level directional light output pixel. The left side of the figure shows an R×C multi-pixel source frame, i.e., a source video frame. This source frame is obtained by decoding the source multi-viewpoint video using a video decoder and then converting it to 2D using an external texture. It includes multiple viewpoint images arranged in an array according to a preset number of rows R and a preset number of columns C. The total number of viewpoints N is not greater than the product of the preset number of rows and columns. The upper right side of the figure shows the three channels of the device mask, with channel values ​​of normalized viewpoint values ​​m1, m2, and m3. Each channel value characterizes the target viewpoint corresponding to the color component of the target output pixel. The lower right side of the figure shows the three color components of the target output pixel, i.e., component 1, component 2, and component 3. The middle part represents the target output pixel.

[0051] Step S60: Based on the interleaving values ​​of the color components of each of the target output pixels, a target interleaved frame is formed, and multiple target interleaved frames are video encoded and encapsulated to generate a pre-interleaved video.

[0052] Specifically, a target interleaved frame refers to a complete image frame composed of the interleaved values ​​of all color components of all target output pixels, adapted to the optical characteristics of a target-oriented optical display device. Pre-interleaved video refers to a standard format video file generated by encoding multiple target interleaved frames in chronological order. It already contains all the optical display information required to adapt to a target-oriented optical display device, and the terminal device can directly decode and play it without performing additional interleaving calculations.

[0053] After traversing all pixels and color components of the interleaved frame to be constructed, a complete interleaved image conforming to the optical characteristics of the target device is obtained. The video encoding and encapsulation module then compresses and encapsulates these interleaved frame sequences according to standard video encoding protocols (such as H.264, H.265, or AV1) into a common video container format. All device-related viewpoint rearrangement and sub-pixel mixing calculations in the pre-interleaved video are completed offline, eliminating the need for any further interleaving operations during terminal playback. Furthermore, the pre-interleaved video can be played directly using a mature hardware video decoder without requiring customized drivers or high-performance graphics processor support. Thus, the offline processing mechanism frees up the terminal's real-time computing power, enabling low-power embedded devices to smoothly present high-resolution naked-eye 3D video content.

[0054] First, a surface-input video encoder is created, such as an H.264 encoder, and the encoder input surface is obtained. A recordable window surface is created using this input surface, allowing the rendering results of the default frame buffer to directly enter the video encoder without calling pixel read operations. When the decoder outputs each valid video frame, its display timestamp is read. After interleaving rendering is complete, this display timestamp is converted from microseconds to the time unit required by the graphics window surface, set to the encoder input surface, and the graphics buffer is swapped. Thus, the interleaved frame uses the timeline of the source frame. The encoding resolution and bitrate are determined based on the target screen and hardware decoding capabilities. For example, approximately 80 Mbit / s bitrate can be used for 1440×2560 output, and a higher bitrate can be used for 2160×3840 output. H.265, AV1, or other encoding formats supported by the target terminal can also be used. When decoding input ends, an input end flag is sent to the decoder; after the decoder has completely output, an input end signal is sent to the video encoder; after the encoder output format is determined, a container track is created and encoded data is written until an encoding output end flag is received, ultimately resulting in the pre-interleaved video in a normal video container.

[0055] In the above manner, the mask index interleaving calculation, which was originally performed frame by frame during terminal playback, is moved to the offline stage, so that the target terminal no longer performs the interleaving process during playback, reducing the real-time computing load of the terminal; based on the encoding and encapsulation as pre-interleaved video as the interleaving result, the terminal playback link is shortened, reducing the occupation of graphics processing resources and texture synchronization links, thereby improving the processing efficiency and stability of terminal playback.

[0056] Further, determining the target viewpoint corresponding to each color component based on the device mask includes: Obtain the sub-pixel arrangement identifier in the device mask, the sub-pixel arrangement identifier being used to characterize the correspondence between each channel of the device mask and each color component of the target output pixel; Obtain the channel value of the channel in the device mask that corresponds to the color component arranged according to the sub-pixel identifier, wherein the channel value is a normalized viewpoint value; Obtain the total number of viewpoints in the multiple viewpoint images, and determine the viewpoint number of the target viewpoint of the color component in the multiple viewpoint images based on the channel value and the total number of viewpoints; The viewpoint image corresponding to the viewpoint number is determined as the target viewpoint corresponding to the color component.

[0057] Specifically, the total number of viewpoints in the plurality of viewpoint images is obtained; the normalized viewpoint value is multiplied by a difference, the difference being equal to the total number of viewpoints minus one; and the integer part of the resulting product is taken to obtain the viewpoint index of the target viewpoint of the color component in the plurality of viewpoint images; the viewpoint image with the viewpoint index is determined as the target viewpoint corresponding to the color component.

[0058] In this embodiment, the device mask does not directly store discrete viewpoint index integers. Instead, it encodes viewpoint information using normalized floating-point values ​​(i.e., normalized viewpoint values ​​m, typically ranging from 0 to 1). This design allows mask data to be stored and transmitted in a standard image texture format, while also reserving space for smooth interpolation or high-precision quantization of viewpoints. When determining the target viewpoint, the continuous normalized viewpoint values ​​m need to be discretized into specific viewpoint indices. .

[0059] In a preferred embodiment, the quantization calculation formula is as follows: Where N is the total number of viewpoints, This represents the floor function. The reason for multiplying by (N-1) instead of N is that viewpoint numbers usually start from 0 (i.e., 0 to N-1), and when m takes the maximum value of 1, it should be mapped to the last viewpoint (number N-1).

[0060] Specifically, the subpixel arrangement identifier is first read from the device configuration parameters and passed as a global variable to the shader program. Then, the fragment shader uses the normalized coordinates of the current target output pixel in the frame to be interleaved as texture sampling coordinates, performs sampling on the device mask texture, and extracts the channel value corresponding to the current color component from the sampling result, i.e., the normalized viewpoint value m. Next, the shader takes the normalized viewpoint value m and the total number of viewpoints N as input parameters, performs multiplication and rounding down operations, and outputs an integer viewpoint index. This viewpoint number serves as an index for subsequent 2D coordinate mapping and texture sampling, and is passed between different shader calculation stages as a register variable throughout the rendering pipeline.

[0061] This quantization process precisely converts abstract optical mask data into addressable data indexes, ensuring that each sub-pixel is assigned to the correct viewpoint source.

[0062] Furthermore, to accommodate the differences in physical subpixel arrangements of different display panels, this embodiment introduces a subpixel arrangement identifier as a control parameter for channel mapping. The target output pixel typically includes a first color component, a second color component, and a third color component (e.g., red, green, and blue). When obtaining the channel value in the device mask corresponding to the color component according to the subpixel arrangement identifier, the specific steps include: When the sub-pixel arrangement is identified as the first arrangement, the first channel, the second channel, and the third channel of the device mask are made to correspond to the first color component, the second color component, and the third color component in sequence. When the sub-pixel arrangement is identified as the second arrangement, the channel values ​​read by the first color component and the third color component are transformed so that the first color component reads the channel value of the third channel, the third color component reads the channel value of the first channel, and the channel value read by the second color component remains unchanged.

[0063] In this embodiment, a configurable channel remapping layer is established at the software level. Because directional optical display screens from different manufacturers or batches may have differences in the physical arrangement of their subpixels (RGB or BGR), if the interleaving algorithm fixes the reading order, severe color misalignment or stereoscopic visual anomalies will occur on mismatched screens.

[0064] Specifically, channel remapping is implemented through conditional branching logic within the fragment shader. The shader receives the subpixel arrangement identifier as a unified variable. When the identifier value is the first arrangement, the shader directly reads the sampled value of the corresponding channel from the device mask texture according to the channel number for the current color component. When the identifier value is the second arrangement, before performing texture sampling, the shader replaces the sampling channel index of the first color component with the third channel index, replaces the sampling channel index of the third color component with the first channel index, and leaves the sampling channel index of the second color component unchanged.

[0065] Specifically, such as Figure 2 As shown, for component 1, the channel value of the corresponding channel in the device mask is read. The channel read is determined by the sub-pixel arrangement identifier: When the sub-pixel arrangement is identified as the first arrangement, component 1 reads the channel value m1 of the first channel; When the sub-pixel arrangement is identified as the second arrangement, the channel read by component 1 is transformed so that component 1 reads the channel value m3 of the third channel.

[0066] The normalized viewpoint value is multiplied by a difference value, which is equal to the total number of viewpoints N minus one. The integer part of the product is then taken to obtain the viewpoint index of the target viewpoint with component 1 in the multiple viewpoint images. .

[0067] Viewpoint numbers are assigned to each viewpoint image in the order in which the multiple viewpoint images are arranged in the source frame, and the arrangement order is row priority and the viewpoint numbers start from zero. By viewpoint number Divide by a divisor to obtain a quotient and a remainder, where the divisor is the preset column number C. The integer part of the quotient is used as the row number j of the viewpoint image in the source frame, and the remainder is used as the column number of the viewpoint image in the source frame. Thus, the j-th row and the first position in the source frame arranged in the R×C pattern are located. The viewpoint image region of the column.

[0068] Then, the position of the target output pixel in the interlaced frame to be constructed is normalized to obtain normalized horizontal and vertical coordinates, and the relative position within the viewpoint image region is obtained accordingly. The component value of component 1 is then sampled from this relative position. The relative position is represented by the relative horizontal and vertical coordinates, where the relative horizontal coordinate is equal to the normalized horizontal coordinate divided by the preset column number C, plus the column number. The value obtained by dividing by the preset number of columns C; the relative vertical coordinate is equal to the normalized vertical coordinate divided by the preset number of rows R, plus the value obtained by dividing the row number j by the preset number of rows R.

[0069] For components 2 and 3, respectively, according to their corresponding mask channel values ​​m2 and m3 (when the sub-pixel arrangement is identified as the second arrangement, the channel read by component 3 is transformed so that component 3 reads the channel value m1 of the first channel, while the channel read by component 2 remains unchanged), the same processing as that of component 1 is performed independently. The target viewpoint of each component is determined, the viewpoint image region of each component is located in the source frame, and the component values ​​of component 2 and component 3 are obtained from the same relative position in their respective regions.

[0070] When the source multi-view video adopts a reverse viewpoint arrangement, then the viewpoint sequence number is obtained. Then, subtract the viewpoint number from the difference. The reverse viewpoint number is obtained, and the difference is equal to the total number of viewpoints N minus one. The reverse viewpoint number is then used to replace the original viewpoint number. Perform the determination of the row number j and the column number. The steps.

[0071] Since each color component independently determines the target viewpoint based on its respective mask channel value, the viewpoint image regions corresponding to components 1, 2, and 3 can be different. This means that different color components of the same target output pixel can be sampled from different viewpoint images. Simultaneously, each viewpoint image region is sampled at the same relative position, ensuring that the sampling points of each viewpoint image geometrically correspond one-to-one with the position of the target output pixel. Therefore, merging the component values ​​sampled from components 1, 2, and 3 yields the interleaved color components of the target output pixel. Each of these three color components carries image information from its corresponding viewpoint, forming an output pixel that adapts to sub-pixel-level directional light emission. Traversing each target output pixel in the frame to be interleaved constitutes the target interleaved frame corresponding to the source video frame.

[0072] By employing the above method, in order to avoid altering the data content of the mask texture itself and affecting the subsequent quantization calculation and coordinate mapping execution logic, the conditional replacement operation based on channel indexes only replaces the channel selection parameters of the texture sampling function in the shader. Furthermore, by identifying sub-pixel arrangement identifiers and dynamically adjusting the channel reading order, the same interleaving code can be compatible with multiple hardware specifications without modification, thereby improving the versatility and deployment efficiency of the solution.

[0073] After determining the viewpoint number corresponding to the color component, it is necessary to map it back into the two-dimensional grid structure of the source video frame. When the preset arrangement is arranged according to a preset number of rows and columns, the total number of viewpoints in the multiple viewpoint images is not greater than the product of the preset number of rows and the preset number of columns. The step of determining the corresponding viewpoint image region in the source video frame according to the target viewpoint and the preset arrangement specifically includes: According to the arrangement order of the multiple viewpoint images in the source video frame, the viewpoint sequence number is assigned to each viewpoint image in sequence; Based on the viewpoint number and the preset number of columns, the row number and column number of the viewpoint image in the source video frame are determined; The viewpoint image region located at the row number and column number in the source video frame is determined as the viewpoint image region corresponding to the target viewpoint.

[0074] The arrangement order is row priority order, that is, the multiple viewpoint images are first allocated in ascending order of row number, and then in ascending order of column number within the same row number, and the viewpoint number starts from zero. For the viewpoint number corresponding to the target viewpoint, divide the viewpoint number by a divisor to obtain a quotient and a remainder, where the divisor is the preset column number. Use the integer part of the quotient as the row number of the viewpoint image in the source video frame, and use the remainder as the column number of the viewpoint image in the source video frame. The viewpoint image region located at the row number and column number in the source video frame is determined as the viewpoint image region corresponding to the target viewpoint.

[0075] In this embodiment, the shader will use the viewpoint number The preset column number C is taken as an integer input. The row number j is obtained through a single integer division operation, and the column number is obtained through a single modulo operation. Then, the shader sets the row number j and column number j. The pixel offset of the viewpoint image region in the source video frame is calculated by multiplying it by the pixel height and pixel width of a single viewpoint image, respectively. This pixel offset is then added to the local sampling coordinates corresponding to the normalized coordinates of the target output pixel in the interlaced frame to be constructed, thus obtaining the final sampling coordinates in the texture of the source video frame.

[0076] Specifically, if the normalized coordinates of the target output pixel in the interleaved frame to be constructed are The preset number of rows is R, the preset number of columns is C, and the row number and column number of the viewpoint image region are j and j, respectively. Then the sampling coordinates in the source texture can be represented as:

[0077]

[0078] The shader performs texture sampling based on these coordinates and reads the color component values ​​at the corresponding positions.

[0079] This describes the inverse transformation process from one-dimensional linear indexes to two-dimensional spatial coordinates. Assume the preset number of columns is C, and the viewpoint number is... Then the row number j and column number It can be calculated using the following formula: , .

[0080] in, This represents the modulo operation. Based on low-computational-complexity addressing algorithms such as division and modulo, it is executed in parallel within the shader of the graphics processor, thereby establishing a deterministic mathematical relationship between the logical index of the viewpoint and the physical position of the source frame. This ensures that no matter how the total number of viewpoints changes, as long as the grid topology parameters (number of rows R, number of columns C) are known, each viewpoint image block can be accurately located.

[0081] Furthermore, considering the diversity of multi-view content production processes and the potential inconsistency between the viewpoint arrangement order of the source video and the expected light field direction of the target display device, the method also includes a viewpoint inversion correction mechanism: When the source multi-view video adopts a reverse viewpoint arrangement, the reverse viewpoint number is obtained according to the viewpoint number corresponding to the target viewpoint, and the reverse viewpoint number replaces the viewpoint number corresponding to the target viewpoint.

[0082] Specifically, when the source multi-viewpoint video adopts a reverse viewpoint arrangement, the reverse viewpoint number is first obtained based on a difference and the viewpoint number corresponding to the target viewpoint. The difference is equal to the total number of viewpoints minus one, and the reverse viewpoint number is equal to the difference obtained by subtracting the viewpoint number from the difference. Then, the reverse viewpoint number is used to replace the viewpoint number corresponding to the target viewpoint, and the steps of determining the row number and the column number are performed.

[0083] In this embodiment, the original viewpoint number is obtained through quantization. Afterwards, before performing two-dimensional coordinate mapping, based on the calculation formula Obtain the reverse viewpoint number .

[0084] When the viewpoint order of the source content is reversed compared to the device's expected order, the reconstructed 3D image will exhibit depth inversion, meaning that objects that were originally protruding will appear sunken, severely impacting the viewing experience. By introducing a correction step that flips the left-right order of the viewpoint, it can automatically accommodate both forward and reverse order source content formats, enhancing its compatibility with external materials.

[0085] Specifically, after quantizing to obtain the original viewpoint number Afterwards, before performing the 2D coordinate mapping, a conditional judgment step is inserted. The shader reads the viewpoint arrangement direction identifier of the source video. When the identifier indicates a reverse arrangement, a subtraction operation is performed, subtracting the current viewpoint number from the total number of viewpoints N. Obtain the reverse viewpoint number And the input parameters in the subsequent coordinate mapping steps will be changed from Replace with When the identifier indicates an ascending order, skip the subtraction operation and directly use the original viewpoint number. Pass in the coordinate mapping steps that follow.

[0086] Furthermore, to avoid callbacks failing to execute due to worker threads with graphics contexts waiting for frame notifications, the frame availability listener for surface textures can run in a separate message thread. The worker thread updates the surface texture and obtains the texture transformation matrix after waiting for the frame availability signal. This thread separation design ensures the smooth operation of the graphics rendering pipeline, avoiding pipeline blockage caused by waiting for frame notifications.

[0087] In traditional solutions, to use decoded video frames for graphics rendering, it is typically necessary to call interfaces such as glReadPixels to read data back from video memory to central processing unit (CPU) memory, and then upload it back to video memory via interfaces such as glTexImage2D for use by the shader. This round-trip copying method not only consumes a huge amount of memory bandwidth but also causes pipeline synchronization blockage between the CPU and the GPU, reducing the frame rate of high-resolution video processing.

[0088] Compared to traditional video processing workflows, this embodiment eliminates intermediate data readback and retransmission steps by directly binding the output surface to external textures, achieving zero-copy transmission and reducing memory bandwidth usage and CPU load.

[0089] It should be understood that although this embodiment is described using a specific graphics interface and hardware decoder as an example, in other embodiments, as long as direct memory mapping or zero-copy transmission between the decoding output and the graphics texture can be achieved, they are all within the scope of protection of this application.

[0090] Correspondingly, the method also includes: Create a surface-input type video encoder and obtain the input surface of the video encoder; create a graphics rendering window surface using the input surface.

[0091] The process of determining the target viewpoint, determining the viewpoint image region, and sampling the interlacing value based on the device mask is executed through a second rendering channel, which includes: Using the aforementioned two-dimensional texture as the multi-viewpoint source texture and the device mask as the index texture, rendering is performed to obtain the rendering result; The rendering result is drawn onto the surface of the graphics rendering window, so that it can directly enter the input surface of the video encoder through the surface of the graphics rendering window, and the target interlaced frame is sent to the input surface of the video encoder. The target interlaced frame is read back to memory without pixel reading operation.

[0092] Alternatively, after obtaining the target interleaved frame corresponding to the source video frame, the display timestamp of the source video frame read above can be updated to the time unit required by the surface of the graphics rendering window, and the converted display timestamp can be added to the input surface of the video encoder before swapping the graphics buffer.

[0093] The step of video encoding and encapsulating the multiple target interleaved frames includes: When the source multi-view video decoding input ends, a decoding input end flag is sent to the video decoder; After the video decoder has finished outputting, an encoding input end signal is sent to the video encoder. After the output format of the video encoder is determined, a container track is created and encoded data is written until the encoding output end marker is received, thus obtaining the pre-interleaved video.

[0094] The resolution of the target interlaced frame is consistent with the output resolution of the target directional optical display device, and the bitrate of the video encoding is set according to the resolution. To adapt to the sampling characteristics of the graphics rendering pipeline and ensure the accuracy of stereo vision, the step of sampling the component values ​​of the color components from the viewpoint image region at a relative position corresponding to the position of the target output pixel in the interlaced frame to be constructed includes: In a two-dimensional texture, a corresponding viewpoint image region is determined, and the component values ​​of the color components are sampled from the viewpoint image region in the two-dimensional texture at the relative positions corresponding to the positions of the target output pixel in the interlaced frame to be constructed, and used as the interlacing values ​​of the color components of the target output pixel.

[0095] In this embodiment, under certain graphics processing interfaces or hardware architectures, external textures, as a special type of texture object, may have limitations in sampling functionality. For example, some mobile GPUs' external textures do not support non-normalized coordinate sampling, do not support specific texture wrapping modes, or cannot perform high-precision random access in the fragment shader.

[0096] To further address the aforementioned issues, this embodiment introduces an intermediate conversion step to convert the data in the external texture into a standard two-dimensional texture, enabling accurate reading using a common texture sampling method. A two-dimensional texture refers to a 2D Texture object conforming to the standard graphics API specification, which supports complete filtering, addressing, and mipmap functions, ensuring sampling stability under different resolution scaling.

[0097] To illustrate the data processing procedure more clearly, a specific numerical example is provided below. Assume the source video frame contains 45 viewpoint images (N=45), arranged in 5 rows and 9 columns (R=5, C=9). For a target output pixel in the frame to be interleaved, the device mask channel value (normalized viewpoint value) corresponding to its red component is m=0.5, and the current device is in RGB arrangement (first arrangement), while the source video is in ascending order. First, the viewpoint index is calculated according to the quantization formula: .

[0098] Since the order is ascending, there's no need to calculate the reverse viewpoint index; coordinate mapping can be performed directly using v=22. Next, the position of this viewpoint in the source frame grid is calculated: row number Column number .

[0099] The red component of the target output pixel should be sampled from the viewpoint image region in the 2nd row and 4th column of the source video frame. If the device is now in BGR arrangement (second arrangement), the red component will be calculated by reading the value of the blue channel of the mask, while the green component will still read the green channel value, thus achieving hardware adaptation without changing the source data.

[0100] It should be understood that the above values ​​are merely illustrative examples. In actual applications, the number of viewpoints, grid size, and mask accuracy can be flexibly set according to the specific equipment capabilities, and all of these fall within the scope of protection of this application.

[0101] To generate this two-dimensional texture, the method further includes the following preprocessing steps: The two-dimensional texture is created, the resolution of which is the same as that of the source video frame, and the two-dimensional texture is used as the drawing target; Using the source video frame located in the outer texture as the texture source, the texture coordinates of the outer texture are transformed based on the texture transformation matrix so that the source video frame written in the two-dimensional texture completes the vertical flip. The pixel data of the source video frame in the external texture is written into the two-dimensional texture to obtain the two-dimensional texture carrying the source video frame.

[0102] In this embodiment, to further address the inherent coordinate system differences between video data and graphics rendering data, the following measures are taken. Typically, video encoding standards and video decoders use the top-left corner as the origin (0,0) and the Y-axis pointing downwards as the positive direction; while mainstream graphics rendering pipelines such as OpenGL use the bottom-left corner as the origin (0,0) and the Y-axis pointing upwards as the positive direction. If external textures are directly drawn onto 2D textures without coordinate correction, the generated image will be inverted in the vertical direction, causing the subsequently interwoven image to be upside down.

[0103] To this end, this application designs a specific texture transformation matrix to perform coordinate correction. In a preferred embodiment, the texture transformation matrix is ​​configured to flip only the vertical component of the texture coordinates, while keeping the horizontal component unchanged. The specific transformation formula can be expressed as:

[0104]

[0105] in, Original texture coordinates, These are the transformed texture coordinates.

[0106] In the source video frames of multi-viewpoint videos, the horizontal arrangement of the viewpoint images directly corresponds to the spatial displacement sequence of the viewing angle (i.e., parallax information). If a horizontal mirroring is incorrectly introduced during coordinate transformation (e.g., misusing a conventional image flipping algorithm), the positions of the left and right viewpoints in the source frame will be swapped. This reversal of the left-right order disrupts the spatial consistency of the light field data, resulting in severe depth perception errors in the 3D images reconstructed on directional optical display devices. Objects that should protrude from the screen appear sunken, or vice versa. Therefore, this embodiment further provides a processing strategy of vertical flipping only, without horizontal mirroring, to ensure the correct naked-eye 3D display effect. By strictly limiting the texture transformation matrix to act only on the Y-axis, this solution solves the coordinate system alignment problem in the fields of graphics and video, while fully preserving the horizontal parallax structure of the multi-viewpoint data, ensuring the realism of stereoscopic vision.

[0107] After completing the coordinate transformation and writing the data into the two-dimensional texture, high-precision sampling operations can be performed based on this texture. Specifically, during sampling, for each target output pixel in the interleaved frame to be constructed, the system first locates the corresponding viewpoint image region in the two-dimensional texture according to the viewpoint number determined in the aforementioned embodiment. Then, it uses the normalized coordinates of the target output pixel in the interleaved frame to be constructed. As a relative position index, sampling is performed within the viewpoint image region. Since the 2D texture has already undergone correct coordinate correction, the sampled coordinates at this point can directly correspond to the correct physical pixel positions in the source viewpoint image, eliminating the need for additional coordinate compensation calculations during the sampling stage.

[0108] The two-level processing mechanism described above (first conversion and correction, then unified sampling) not only decouples the coordinate transformation logic from the interleaving sampling logic, improving the maintainability of the shader code, but also enhances the compatibility and execution efficiency of the rendering pipeline on different hardware platforms by transforming complex external texture access into standard 2D texture sampling.

[0109] Combination Figure 3 Furthermore, a complete explanation of the offline interleaving encoding method for multi-view video is provided.

[0110] This method is divided into an offline pre-baking stage and a target terminal running stage according to the processing sequence: the offline pre-baking stage is executed on the content production side and is used to generate pre-interleaved ordinary video; the target terminal running stage is executed on the target terminal side and is used to play the pre-interleaved ordinary video. The following section combines... Figure 3 The two stages will be explained separately.

[0111] I. Offline Pre-baking Stage The input to the offline pre-baking stage consists of multi-view clips and device masks. Interleaving and encoding are completed by two levels of rendering and surface input encoding. The output is a pre-interleaved ordinary video stored in a state-based manner.

[0112] (a) Multi-viewpoint segments and equipment masks A multi-viewpoint segment refers to multiple action state segments obtained by dividing a source multi-viewpoint video. In this embodiment, the source multi-viewpoint video includes multiple action state segments of the digital human, such as standby, thinking, and answering action state segments. Each multi-viewpoint segment includes multiple source video frames, and each source video frame includes multiple viewpoint images arranged in a preset arrangement. In this embodiment, the preset arrangement is an array arrangement with a preset number of rows and columns, with five preset rows and nine preset columns. The total number of viewpoints in the multiple viewpoint images is forty-five, and the total number of viewpoints is not greater than the product of the preset number of rows and the preset number of columns.

[0113] The device mask corresponds to the target orientation optical display device and is obtained after calibration by the device. Its resolution is the same as the output resolution of the target orientation optical display device, or it can be mapped to the output resolution according to a preset sampling method. The device mask includes at least a first channel, a second channel, and a third channel. The channel value of each channel is a normalized viewpoint value, which is a value between zero and one. The channel value of each channel is used to characterize the target viewpoint corresponding to one of the multiple color components of the target pixel. In addition, a sub-pixel arrangement identifier is obtained. The sub-pixel arrangement identifier is used to characterize the correspondence between each channel of the device mask and each color component of the target pixel: when the sub-pixel arrangement identifier is in the first arrangement, the first, second, and third channels of the device mask are made to correspond to the first color component, the second color component, and the third color component, respectively; when the sub-pixel arrangement identifier is in the second arrangement, the channel values ​​read by the first and third color components are changed so that the first color component reads the channel value of the third channel, the third color component reads the channel value of the first channel, and the channel value read by the second color component remains unchanged, thereby adapting to the panel where the physical arrangement of the first and third color components is interchanged.

[0114] (II) Two-level rendering and surface input encoding The two-level rendering and surface input encoding process includes, in sequence: decoding and outputting to an external texture, the first rendering channel, the second rendering channel, surface input encoding, display timestamp pass-through, and encoding and encapsulation.

[0115] Decode and output to an external texture: Create a video decoder with an output surface, and create an external texture; The output surface of the video decoder is bound to an external texture; multi-view segments are input into the video decoder for decoding, and each source video frame is output by the video decoder. Each source video frame is written to the external texture via the output surface to obtain the source video frame located in the external texture; at the same time, when the video decoder outputs each source video frame, the display timestamp of the source video frame is read, which is in microseconds.

[0116] First rendering pass: Create a 2D texture with the same resolution as the source video frame, and use the 2D texture as the drawing target; use the source video frame located in the outer texture as the texture source and the 2D texture as the drawing target, and perform drawing, writing the pixel data of the source video frame in the outer texture into the 2D texture to obtain a 2D texture carrying the source video frame; during drawing, transform the texture coordinates of the outer texture according to the texture transformation matrix. The texture transformation matrix only flips the vertical component of the texture coordinates, while the horizontal component of the texture coordinates remains unchanged, so that the source video frame written into the 2D texture completes the vertical flip, thereby compensating for the difference in the coordinate origin between the graphics frame buffer and the input surface of the video encoder, and keeping the horizontal direction unchanged to avoid mirroring the left and right order of multiple viewpoints.

[0117] The second rendering pass: Parallax interleaving is performed using a 2D texture as the multi-viewpoint source texture and a device mask as the index texture to obtain the target interleaved frame corresponding to the source video frame. For each color component of each target pixel in the frame to be interleaved, the channel value of the channel corresponding to the sub-pixel arrangement identifier of the color component in the device mask is read. This channel value is the normalized viewpoint value. The total number of viewpoints in multiple viewpoint images is obtained. The normalized viewpoint value is multiplied by a difference, which is equal to the total number of viewpoints minus one. The integer part of the product is then taken to obtain the viewpoint index of the target viewpoint of the color component in the multiple viewpoint images. Viewpoint indexes are assigned to each viewpoint image according to the arrangement order of the multiple viewpoint images in the source video frame. This arrangement order is row-major and the viewpoint index starts from zero. The viewpoint index is divided by a divisor to obtain the quotient and remainder. The divisor is a preset column number. The integer part of the quotient is used as the row index of the viewpoint image in the source video frame, and the remainder is used as the column index of the viewpoint image in the source video frame. When the source multi-view video uses a reverse viewpoint arrangement, first, the reverse viewpoint number is obtained by subtracting the viewpoint number from a difference. This difference is equal to the total number of viewpoints minus one. Then, the reverse viewpoint number replaces the viewpoint number to perform the step of determining the row number and column number. The viewpoint image region located at the row number and column number in the source video frame is determined as the viewpoint image region corresponding to the target viewpoint. The position of the target pixel in the interleaved frame to be constructed is normalized to obtain normalized horizontal coordinates and normalized vertical coordinates. Based on this, the relative position in the viewpoint image region is obtained. The relative position is represented by the relative horizontal coordinate and the relative vertical coordinate. The relative horizontal coordinate is equal to the normalized horizontal coordinate divided by the preset number of columns, plus the value obtained by dividing the column number by the preset number of columns. The relative vertical coordinate is equal to the normalized vertical coordinate divided by the preset number of rows, plus the value obtained by dividing the row number by the preset number of rows. Based on relative horizontal and vertical coordinates, the component values ​​of the color component are sampled from the viewpoint image region in the 2D texture and used as the interleaving value of the color component of the target pixel. Each viewpoint image region is sampled at the same relative position. Each target pixel and its color component in the frame to be interleaved is traversed, and the target interleaved frame corresponding to the source video frame is constructed based on the interleaving values ​​of the color components of each target pixel. Since different color components of the same target pixel can correspond to different target viewpoints, the output pixels obtained from the interleaving form pixels that adapt to sub-pixel-level directional light emission.

[0118] Surface Input Encoding: A surface input video encoder is created, its input surface is obtained, and a graphics rendering window surface is created using this input surface. The rendering result of the second rendering channel, i.e., the target interlaced frame, is drawn onto the graphics rendering window surface, allowing it to directly enter the input surface of the video encoder. This results in the target interlaced frame being fed into the video encoder's input surface without undergoing pixel readout operations. Therefore, the decoded source video frame, the converted 2D texture, and the interlaced target frame are always located on the graphics processing unit side, avoiding the data copying overhead caused by reading the graphics processing unit's output frame by frame back to memory.

[0119] Display timestamp pass-through: After obtaining the target interleaved frame corresponding to the source video frame, the display timestamp read when the video decoder outputs the source video frame is converted from microseconds to the time unit required by the surface of the graphics rendering window, and the converted display timestamp is set on the input surface of the video encoder. Then, the graphics buffer is swapped so that the target interleaved frame uses the display timestamp of the source video frame, thereby keeping the time axis consistent before and after interleaving and avoiding deviations between the audio and video rhythm and the duration of the video segment.

[0120] Encoding and Packaging: When the multi-view segment decoding input ends, a decoding input end flag is sent to the video decoder; after the video decoder has completely output, an encoding input end signal is sent to the video encoder; after the output format of the video encoder is determined, a packaging track is created and encoded data is written until the encoding output end flag is received, resulting in pre-interleaved video.

[0121] The video encoding format is H.264, H.265, or AV1; the resolution of the target interleaved frame is consistent with the output resolution of the target directional optical display device; the video encoding bitrate is set according to the resolution.

[0122] (iii) Pre-interleaved ordinary video stored in a state-based manner During the offline phase, the aforementioned two-level rendering and surface input encoding are performed on each action state segment, such as standby, thinking, and answering, to obtain multiple pre-interlaced video segments. These segments are then categorized and saved according to their action states, forming a state-based pre-interlaced ordinary video. This pre-interlaced ordinary video is a video file within an ordinary video container, where each frame is an image with completed viewpoint interlacing, which can be directly played by the target terminal using a video hardware decoder.

[0123] II. Target Terminal Operation Phase The target terminal operation phase includes selecting pre-interlaced segments, video hardware decoding, and directional optical display.

[0124] Selecting a pre-interleaved segment: Based on the current interaction state, the target terminal selects the corresponding pre-interleaved video segment from the state-stored pre-interleaved ordinary video. During this selection process, the target terminal does not load the viewpoint arrangement (i.e., source multi-viewpoint grid) and device mask of the source video frames, nor does it perform parallax interleaving processing.

[0125] Video hardware decoding: The target terminal decodes the selected pre-interlaced video segments using a video hardware decoder to obtain the interlaced video frames.

[0126] Directional optical display: The decoded video frames are output to the display surface and displayed by the target directional optical display device.

[0127] Since the video frames themselves have already completed sub-pixel-level viewpoint interleaving according to the device mask, the target terminal does not need to perform mask sampling interleaving again when displaying.

[0128] As can be seen from the above method, the device-related viewpoint interleaving calculation is completed entirely in the offline pre-baking stage. During the target terminal's runtime, only segment selection, hardware decoding, and display are required. Therefore, the target terminal no longer performs mask index interleaving calculation frame by frame, nor does it need to maintain the graphics processing context and texture synchronization required for external texture conversion and mask sampling. This reduces the graphics processing resources and power consumption consumed by the terminal's real-time graphics processing. Simultaneously, the terminal playback chain is simplified to hardware decoding and display, reducing thread scheduling and texture synchronization steps, lowering the risk of frame drops, black frames, and frame time instability, and improving playback stability and processing efficiency. Furthermore, when the device changes its optical film, panel resolution, or subpixel arrangement, only the new device mask needs to be used to re-execute interleaving encoding and update the pre-interleaved ordinary video during the offline pre-baking stage, without changing the target terminal's basic video decoding path.

[0129] The above method can be implemented as a computer program, which can be used in, for example... Figure 4 It runs on the computer device shown.

[0130] Please see Figure 4 , Figure 4 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0131] See Figure 4 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0132] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any offline interleaving encoding method for multi-view video.

[0133] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0134] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any offline interleaving coding method for multi-view video.

[0135] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0136] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0137] The processor is used to run a computer program stored in a memory to implement any of the offline interleaving coding methods for multi-view video provided in the above embodiments.

[0138] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the offline interleaving encoding methods for multi-view video provided in the embodiments of this application.

[0139] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0140] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An offline interleaving coding method for multi-view video, characterized in that, include: Acquire source multi-view video and create a video decoder and an external texture, the video decoder having an output surface; Bind the output surface of the video decoder to the external texture; The source multi-view video is input into the video decoder for decoding. The video decoder outputs each decoded source video frame and writes each decoded source video frame into the external texture via the output surface to obtain the source video frame located in the external texture. The source video frame includes multiple viewpoint images arranged in a preset arrangement. Determine the interleaving frame to be constructed corresponding to each of the source video frames, wherein the interleaving frame to be constructed includes a plurality of target output pixels, and each of the target output pixels includes a plurality of color components; Obtain a device mask corresponding to the target directional optical display device; the device mask is used to characterize the target viewpoint corresponding to each color component of the target output pixel; For each color component of each target output pixel, the target viewpoint corresponding to each color component is determined according to the device mask, and the corresponding viewpoint image region is determined in the source video frame according to the target viewpoint and the preset arrangement. From the viewpoint image region, sample at the relative position corresponding to the position of the target output pixel in the interlaced frame to be constructed to obtain the component value of the color component, which is used as the interlacing value of the color component of the target output pixel; Based on the interleaving values ​​of the color components of each of the target output pixels, a target interleaved frame is formed, and multiple target interleaved frames are video encoded and encapsulated to generate a pre-interleaved video.

2. The offline interleaving coding method for multi-view video as described in claim 1, characterized in that, Determining the target viewpoint corresponding to each color component based on the device mask includes: Obtain the sub-pixel arrangement identifier in the device mask, the sub-pixel arrangement identifier being used to characterize the correspondence between each channel of the device mask and each color component of the target output pixel; Obtain the channel value of the channel in the device mask that corresponds to the color component arranged according to the sub-pixel identifier, wherein the channel value is a normalized viewpoint value; Obtain the total number of viewpoints in the multiple viewpoint images, and determine the viewpoint number of the target viewpoint of the color component in the multiple viewpoint images based on the channel value and the total number of viewpoints; The viewpoint image corresponding to the viewpoint number is determined as the target viewpoint corresponding to the color component.

3. The offline interleaving coding method for multi-view video as described in claim 2, characterized in that, The target output pixel includes a first color component, a second color component, and a third color component; The step of obtaining the channel value of the channel in the device mask corresponding to the color component according to the sub-pixel arrangement identifier includes: When the sub-pixel arrangement is identified as the first arrangement, the first channel, the second channel, and the third channel of the device mask are made to correspond to the first color component, the second color component, and the third color component in sequence. When the sub-pixel arrangement is identified as the second arrangement, the channel values ​​read by the first color component and the third color component are transformed so that the first color component reads the channel value of the third channel, the third color component reads the channel value of the first channel, and the channel value read by the second color component remains unchanged.

4. The offline interleaving coding method for multi-view video as described in claim 2, characterized in that, The preset arrangement is based on a preset number of rows and columns. The step of determining the corresponding viewpoint image region in the source video frame according to the target viewpoint and the preset arrangement includes: According to the arrangement order of the multiple viewpoint images in the source video frame, the viewpoint sequence number is assigned to each viewpoint image in sequence; Based on the viewpoint number and the preset number of columns, the row number and column number of the viewpoint image in the source video frame are determined; The viewpoint image region located at the row number and column number in the source video frame is determined as the viewpoint image region corresponding to the target viewpoint.

5. The offline interleaving coding method for multi-view video as described in claim 4, characterized in that, The method further includes: When the source multi-view video adopts a reverse viewpoint arrangement, the reverse viewpoint number is obtained according to the viewpoint number corresponding to the target viewpoint, and the reverse viewpoint number replaces the viewpoint number corresponding to the target viewpoint.

6. The offline interleaving coding method for multi-view video as described in claim 1, characterized in that, The step of sampling the color component values ​​from the viewpoint image region at a relative position corresponding to the position of the target output pixel in the frame to be constructed includes: In a two-dimensional texture, a corresponding viewpoint image region is determined, and the component values ​​of the color components are sampled from the viewpoint image region in the two-dimensional texture at the relative positions corresponding to the positions of the target output pixel in the interlaced frame to be constructed, and used as the interlacing values ​​of the color components of the target output pixel.

7. The offline interleaving coding method for multi-view video as described in claim 6, characterized in that, The method further includes: The two-dimensional texture is created, the resolution of which is the same as that of the source video frame, and the two-dimensional texture is used as the drawing target; Using the source video frame located in the outer texture as the texture source, the texture coordinates of the outer texture are transformed based on the texture transformation matrix so that the source video frame written in the two-dimensional texture completes the vertical flip. The pixel data of the source video frame in the external texture is written into the two-dimensional texture to obtain the two-dimensional texture carrying the source video frame.

8. A computer device, characterized in that, The computer device includes a processor, a memory, and an offline interleaving encoding program for multi-view video stored in the memory and executable by the processor, wherein when the offline interleaving encoding program for multi-view video is executed by the processor, it implements the steps of the offline interleaving encoding method for multi-view video as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an offline interleaving encoding program for multi-view video, wherein when the offline interleaving encoding program for multi-view video is executed by a processor, it implements the steps of the offline interleaving encoding method for multi-view video as described in any one of claims 1 to 7.