Multi-face display low-delay synchronization output method, device, equipment and storage medium

Abstract

The present application relates to the technical field of graphics processing, in particular to a multi-surface display low-latency synchronous output method, device, equipment and storage medium. The present application directly obtains a source image resource handle in a graphics processing unit and converts it into a shared texture resource, calls a calculation shader to perform pixel-level preprocessing in a video memory, calculates a line-of-sight direction vector based on a three-dimensional display space model and a virtual observer viewpoint through a polyhedron mapping shader and maps it to a preprocessed texture for sampling, generates video memory frame buffer data corresponding to each display plane, and finally integrates it into a unified logical display surface and configures a hardware synchronization signal to control multiple display output ports to atomically flip at a vertical synchronization moment, thereby realizing synchronous output of multi-surface display. The present application constructs a data path that resides in the video memory throughout the process, eliminates memory copying and codec transmission delay, and realizes sub-millisecond-level synchronous output of multiple display planes and geometric continuity and visual consistency in the polyhedral display space.

Description

Technical Field

[0001] This invention relates to the field of graphics processing technology, and in particular to a method, apparatus, device, and computer storage medium for low-latency synchronous output of multi-sided display based on video memory-level data paths. Background Technology

[0002] Existing multi-faceted immersive display systems typically construct a multi-channel image distribution system through the collaborative operation of a 3D rendering engine, video decoding module, and desktop capture unit. This system encompasses screen capture, network streaming media transmission, and hardware synchronizers. However, when introducing desktop content into the multi-faceted display space, existing technologies generally employ a lengthy chain of "copying from video memory to RAM, video encoding, network transmission, and then decoding," without establishing a direct connection at the video memory level. This results in a circuitous data path, wasting bandwidth resources, and introducing uncontrollable latency and jitter at the tens of millisecond level due to encoding buffering. Furthermore, traditional solutions treat the desktop as a two-dimensional plane and perform simple stretching, lacking spatial adaptation capabilities based on a three-dimensional geometric model, leading to geometric distortion and image breakage at the multi-faceted splicing points.

[0003] To address the aforementioned technical problems, the shortcomings of the prior art that this application aims to solve are: eliminating end-to-end latency and bandwidth waste caused by data copying, encoding / decoding transmission and CPU intervention between video memory and main memory; and solving the problem of visual discontinuity and geometric distortion caused by the lack of three-dimensional geometric mapping in the polyhedral display space of two-dimensional desktop content, so as to achieve sub-millisecond-level synchronous output between multiple display planes and spatiotemporal consistency in highly immersive scenes. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the end-to-end latency and bandwidth waste caused by data copying and encoding / decoding transmission between video memory and main memory in the prior art, as well as the geometric distortion and screen breakage in the multi-faceted display space caused by the lack of three-dimensional geometric mapping.

[0005] To address the aforementioned technical problems, this invention provides a method for low-latency synchronous output of multi-sided displays, comprising:

[0006] The source image resource handle that has been rendered and resides in the video memory is directly obtained within the graphics processing unit, and the source image resource handle is converted into a shared texture resource in the video memory. The compute shader is invoked to perform pixel-level preprocessing on the shared texture resource in the video memory, generating preprocessed texture data adapted to the multi-faceted display space resolution and color format; Based on the preset 3D display space model and virtual observer viewpoint, the viewing direction vector corresponding to the pixel point on each display plane is calculated by the polyhedral mapping shader, and the viewing direction vector is mapped to the preprocessed texture data for texture sampling to generate the video memory frame buffer data corresponding to each display plane. The video memory frame buffer data corresponding to each display plane is integrated into a unified logical display surface, and a hardware synchronization signal is configured to control multiple display output ports to atomically flip the logical display surface at the vertical synchronization moment, so as to realize the synchronous output of multi-faceted display.

[0007] Preferably, the step of directly acquiring the source image resource handle that has been rendered and resides in the video memory within the graphics processing unit, and converting the source image resource handle into a shared texture resource in the video memory includes: By hooking or intercepting the rendering function of the graphics application interface, after the operating system desktop compositor completes the rendering of a frame, the video memory resource handle pointing to the frame image can be directly obtained. The video memory resource handle corresponds to the texture resource stored in the video memory. The system calls internal commands from the graphics processing unit to directly convert the acquired desktop texture into a shared texture resource that can be read by subsequent shaders in the video memory, generating the converted shared texture resource. The conversion process does not involve the central processing unit.

[0008] Preferably, the step of calling the computation shader to perform pixel-level preprocessing on the shared texture resource in the video memory to generate preprocessed texture data adapted to the multi-faceted display space resolution and color format includes: The scheduling compute shader reads the original pixel data from the shared texture resource, performs a resolution adaptation operation based on the physical resolution parameters of the multi-faceted display space, and generates resolution-matched intermediate texture data. Based on a preset color space conversion matrix, the intermediate texture data matching the resolution is subjected to color space adjustment, and the adjusted data is written into a new video memory texture to obtain preprocessed texture data as the input source for space mapping.

[0009] Preferably, the step of calculating the viewing direction vector corresponding to each pixel on each display plane using a polyhedral mapping shader based on a preset three-dimensional display space model and a virtual observer's viewpoint includes: Construct a set of multiple display planes, where each display plane is defined with a center position, normal vector, up vector, and physical dimensions of width and height, and define a virtual observer viewpoint located at the geometric center of the polyhedron space; For each pixel position on the display plane, calculate the corresponding 3D point coordinates based on the right vector, width, height of the plane, and the relative position of the pixel; Based on the calculated 3D point coordinates and the virtual observer's viewpoint, the standardized 3D line-of-sight direction vector is calculated.

[0010] Preferably, the step of mapping the gaze direction vector to the preprocessed texture data for texture sampling to generate video memory frame buffer data corresponding to each display plane includes: The sampled cube face is determined based on the largest component among the absolute values ​​of each component of the standardized three-dimensional line-of-sight vector. Based on the determined cube face, the corresponding texture coordinates are calculated using the ratio between the corresponding components of the view direction vector; The preprocessed texture data is sampled using the calculated texture coordinates to obtain pixel colors, and the pixel colors are written into the frame buffer object or rendering target texture allocated to the display plane to generate the video memory frame buffer data corresponding to each display plane.

[0011] Preferably, the step of integrating the video memory frame buffer data corresponding to each display plane into a unified logical display surface, and configuring a hardware synchronization signal to control multiple display output ports to atomically flip the logical display surface at the vertical synchronization moment includes: A logical back buffer is created as a unified logical display surface, and the video memory frame buffer data corresponding to each display plane is placed in a preset rectangular area within the logical back buffer through graphics processing unit commands. For graphics processing units that support hardware synchronization, multiple display output ports are configured as a synchronization group through the driver application programming interface. One of the master ports is designated to generate a synchronization signal and control the other slave ports to follow. Frame locking is enabled before calling the rendering function, so that the graphics processing unit waits for all display controllers to be ready and then atomically flips the display surfaces of all ports when the next vertical synchronization signal arrives.

[0012] Preferably, the source image data corresponding to the source image resource handle includes one or more combinations of operating system desktop composite output, real-time output frames of 3D rendering engine, video playback frame buffer content, or remote graphics output frames. The source image data resides in the video memory of the graphics processing unit throughout the entire process of acquisition, preprocessing, spatial mapping, and synchronous output.

[0013] The present invention also provides a multi-sided display low-latency synchronous output device, comprising: The video memory-level image acquisition module is used to directly acquire the source image resource handle that has been rendered and resides in the video memory within the graphics processing unit, and convert the source image resource handle into a shared texture resource in the video memory. The in-memory preprocessing module is used to call the compute shader to perform pixel-level preprocessing on the shared texture resource in the video memory, and generate preprocessed texture data that adapts to the multi-faceted display space resolution and color format. The multi-faceted space mapping module is used to calculate the viewing direction vector corresponding to the pixel point on each display plane through the multi-faceted mapping shader based on the preset three-dimensional display space model and virtual observer viewpoint, and to map the viewing direction vector to the preprocessed texture data for texture sampling to generate the video memory frame buffer data corresponding to each display plane. The synchronous output control module is used to integrate the video memory frame buffer data corresponding to each display plane into a unified logical display surface, and configure hardware synchronization signals to control multiple display output ports to atomically flip the logical display surface at the vertical synchronization moment, so as to realize the synchronous output of multi-faceted display.

[0014] The present invention also provides a multi-sided display low-latency synchronous output device, comprising: Memory, used to store computer programs; A processor is used to implement the steps of the above-described method for low-latency synchronous output of multi-faceted display when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for low-latency synchronous output of multi-faceted displays.

[0016] The technical solution of the present invention has the following advantages compared with the prior art: The multi-faceted display low-latency synchronous output method described in this invention directly acquires and converts the source image resource handle residing in the video memory within the graphics processing unit, avoiding the redundant path of reading the source image data back from the video memory to the system memory and then re-uploading it, thus eliminating the latency caused by memory copying and encoding / decoding transmission. By calling the computation shader to perform pixel-level preprocessing of shared texture resources in the video memory, it ensures that the image data always resides in high-speed video memory during resolution adaptation and color format conversion, further reducing processing time. Based on a preset three-dimensional display space model and a virtual observer's viewpoint, it utilizes... The polyhedral mapping shader calculates the view direction vector and performs texture sampling to generate video memory frame buffer data corresponding to each display plane. This ensures the geometric continuity and visual consistency of desktop content in the polyhedral display space, effectively eliminating geometric distortion and image fragmentation at multi-faceted splicing points caused by traditional 2D stretching. By integrating the frame buffer data of each display plane into a unified logical display surface and configuring hardware synchronization signals to control the atomic flipping of multiple display output ports at vertical synchronization time, sub-millisecond refresh synchronization between multiple display planes is achieved, eliminating time deviations caused by software scheduling or transmission differences. Thus, this method constructs a data path that resides in video memory throughout, reducing end-to-end display latency to the millisecond level, significantly improving the real-time performance, spatial consistency, and synchronization accuracy of multi-faceted immersive displays. Attached Figure Description

[0017] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the implementation of a low-latency synchronous output method for multi-faceted displays provided by the present invention. Figure 2 This is a structural block diagram of a multi-sided display low-latency synchronous output device provided in an embodiment of the present invention. Detailed Implementation

[0018] The core of this invention is to provide a method, apparatus, device, and computer storage medium for low-latency synchronous output of multi-faceted displays. By resident in the video memory for data acquisition, preprocessing, spatial mapping, and hardware synchronous control, it effectively eliminates memory copying and encoding / decoding transmission delays, and ensures geometric continuity and visual consistency in the multi-faceted display space.

[0019] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please refer to Figure 1. Figure 1 The flowchart illustrates the implementation of a low-latency synchronous output method for multi-faceted displays provided by this invention; the specific operation steps are as follows: S101: Directly obtain the source image resource handle that has been rendered and resides in the video memory within the graphics processing unit, and convert the source image resource handle into a shared texture resource in the video memory; S102: Call the compute shader to perform pixel-level preprocessing on the shared texture resource in the video memory to generate preprocessed texture data that adapts to the multi-faceted display space resolution and color format; S103: Based on the preset three-dimensional display space model and virtual observer viewpoint, the viewing direction vector corresponding to the pixel point on each display plane is calculated by the polyhedral mapping shader, and the viewing direction vector is mapped to the preprocessed texture data for texture sampling to generate the video memory frame buffer data corresponding to each display plane. S104: Integrate the video memory frame buffer data corresponding to each display plane into a unified logical display surface, configure a hardware synchronization signal to control multiple display output ports to atomically flip the logical display surface at the vertical synchronization moment, and realize the synchronous output of multi-faceted display.

[0021] In some embodiments, this multi-faceted display low-latency synchronous output method operates within a single computer system equipped with a high-performance graphics processing unit. By constructing a data path that resides entirely in the video memory, it achieves synchronous distribution and low-latency display of graphics rendering output frames across multiple display surfaces. The core architecture of the system consists of a video memory-level desktop acquisition subsystem, a video memory-based image processing subsystem, a multi-faceted spatial mapping subsystem, and a synchronous output control subsystem. The subsystems interact with each other through shared texture resources in the video memory of the graphics processing unit, completely avoiding data readback to system memory or transmission over a network.

[0022] In other embodiments, the source image data resides in the video memory of the graphics processing unit throughout the entire process of acquisition, preprocessing, spatial mapping, and synchronous output, without any copying operations to system memory. By constructing a data path that resides entirely within video memory, the latency caused by system memory copying and network transmission is effectively avoided, reducing the end-to-end latency to the low millisecond level, typically 1 to 3 GPU rendering frame cycles, where video memory processing dominates. Simultaneously, the spatial mapping algorithm based on a 3D geometric model ensures that desktop content can correctly adapt to the complex polyhedral display space, maintaining visual continuity and correct perspective at planar boundaries.

[0023] In one specific embodiment, the method integrates the functions of multiple independent devices such as screen capture cards, encoders, network switches, decoders and external synchronizers required in traditional solutions into a single computer, which greatly reduces system complexity, maintenance costs and failure rates, while freeing up valuable memory bandwidth and CPU computing resources, and supports the seamless introduction of any desktop application into a multi-faceted immersive display environment.

[0024] Specifically, in step S101, resource handles are directly obtained in the video memory layer by intercepting the graphics API rendering function, and zero-copy texture format conversion is completed inside the GPU. The conversion process does not involve the central processing unit and takes microseconds. In step S102, the parallel processing capability of the computation shader is used to efficiently meet the real-time preprocessing requirements of multiple high-resolution display channels. In step S103, a mapping algorithm based on a three-dimensional geometric model is directly executed in the video memory to ensure the visual continuity and perspective correctness of the source image content at the intersection of multiple display planes, effectively eliminating the geometric distortion and screen tearing caused by traditional two-dimensional video mapping. In step S104, sub-millisecond refresh synchronization between multiple display planes is achieved through data integration at the video memory level and synchronization control at the hardware level. The atomic flip mechanism ensures strict consistency of multiple screens in the time dimension, effectively eliminating screen tearing and misalignment.

[0025] It should be noted that this method reduces end-to-end display latency to the millisecond level, achieving sub-millisecond synchronous output across multiple display planes, significantly improving the real-time performance, spatial consistency, and synchronization accuracy of multi-faceted immersive displays. The "atomic flip" refers to the simultaneous switching operation of the foreground and background display buffers of all display output ports at the instant the vertical synchronization signal arrives. This operation is completed within a single hardware clock cycle, ensuring that multiple video signals begin refreshing from the same timing point, and that the refresh time difference between any two ports is no greater than one pixel clock cycle, thereby achieving sub-millisecond synchronization accuracy.

[0026] Based on the above embodiments, in some embodiments, directly obtaining the source image resource handle that has been rendered and resides in the video memory within the graphics processing unit, and converting the source image resource handle into a shared texture resource in the video memory includes: By hooking or intercepting the rendering function of the graphics application interface, after the operating system desktop compositor completes the rendering of a frame, the video memory resource handle pointing to the frame image can be directly obtained. The video memory resource handle corresponds to the texture resource stored in the video memory. The system calls internal commands from the graphics processing unit to directly convert the acquired desktop texture into a shared texture resource that can be read by subsequent shaders in the video memory, generating the converted shared texture resource. The conversion process does not involve the central processing unit.

[0027] In some embodiments, the specific implementation of directly obtaining the source image resource handle that has been rendered and resides in video memory within the graphics processing unit, and converting the source image resource handle into a shared texture resource in the video memory, is as follows: After the operating system desktop compositor completes rendering of a frame, the video memory resource handle pointing to that frame image is directly obtained by hooking or intercepting the rendering function of the graphics application programming interface (API). Here, "hooking" refers to inserting custom code between the application and the operating system API to intercept API calls, and "intercepting" refers to capturing and processing API call requests at the driver layer. In the Windows operating system environment, the Desktop Duplication API is called to directly access the output surface of the operating system desktop compositor and obtain a video memory resource handle identified as type ID3D11Texture2D.

[0028] In other embodiments, the `Present` call of the DirectX graphics interface or the `vkQueuePresentKHR` call of the Vulkan graphics interface is intercepted at the driver layer, and a video memory resource handle containing a `VkImage` type identifier is intercepted as the source image resource handle. The `Present` call is a function in DirectX used to render a rendered frame to the screen, and the `vkQueuePresentKHR` call is a function in Vulkan used to render an image queue to the screen. This process does not require copying image data to system memory; the output is a video memory resource handle pointing to the frame image, which corresponds to the texture resource stored in video memory.

[0029] In one specific embodiment, after obtaining the handle of the source image resource, an internal command of the graphics processing unit (GPU) is invoked to directly convert the acquired desktop texture into a shared texture resource that can be read by subsequent shaders in video memory. This conversion process is executed entirely within the GPU and does not involve the central processing unit (CPU). The remapping of resource types and state switching are completed within video memory through the GPU command queue, generating the converted shared texture resource, which can be directly bound and read by subsequent compute shaders or mapping shaders.

[0030] Specifically, the entire acquisition and conversion chain is confined within the video memory, avoiding cross-bus data transmission. This makes the conversion process take only microseconds, ensuring that the source image data can enter the subsequent multi-faceted spatial mapping processing flow with extremely low latency, thus achieving seamless video memory-level connection from desktop synthesis output to shared texture resources.

[0031] It should be noted that this implementation method directly obtains resource handles at the video memory layer by intercepting graphics API rendering functions and performs zero-copy texture format conversion within the GPU. This effectively avoids the latency and bandwidth consumption caused by memory readback and CPU intervention in traditional solutions, reducing the time spent in the source data preparation stage to the microsecond level. The "hooking or interception" can be implemented at the application layer through API hooking technology or at the driver layer through filter drivers. Both methods can achieve the goal of directly obtaining resource handles at the video memory layer.

[0032] Based on the above embodiments, in some embodiments, the step of calling the computation shader to perform pixel-level preprocessing on the shared texture resource in the video memory to generate preprocessed texture data adapted to the multi-faceted display space resolution and color format includes: The scheduling compute shader reads the original pixel data from the shared texture resource, performs a resolution adaptation operation based on the physical resolution parameters of the multi-faceted display space, and generates resolution-matched intermediate texture data. Based on a preset color space conversion matrix, the intermediate texture data matching the resolution is subjected to color space adjustment, and the adjusted data is written into a new video memory texture to obtain preprocessed texture data as the input source for space mapping.

[0033] In some embodiments, the specific implementation of calling the computation shader to perform pixel-level preprocessing on the shared texture resource in video memory to generate preprocessed texture data adapted to the multi-faceted display space resolution and color format is as follows: the computation shader program is scheduled to execute through the GPU command queue, and the computation shader uses the shared texture resource in video memory generated in step S101 as the input source to directly read the original pixel data therein, which typically contains RGBA format images output by the operating system desktop compositor.

[0034] In other embodiments, resolution adaptation is performed based on the physical resolution parameters of the multi-faceted display space, namely the width and height of each display plane. Internally, the shader performs bilinear interpolation scaling or region sampling clipping on the original pixel data according to the pixel grid density of the target display plane to eliminate pixel size differences between the source image and the target display plane, generating resolution-matching intermediate texture data. This intermediate data is temporarily stored in a temporary rendering target in video memory. The bilinear interpolation refers to a scaling method that uses a weighted average of the color values ​​of the four neighboring pixels around a pixel to obtain a new pixel value.

[0035] In one specific embodiment, a color space adjustment operation is performed on the resolution-matching intermediate texture data based on a preset color space conversion matrix. Because different display devices, such as projectors or professional monitors, have different color gamut characteristics, the intermediate texture data is converted from the sRGB color space of the source image to the specific color space required by the target display device, such as DCI-P3 or Rec.2020. This conversion process is performed in parallel in each thread of the shader through matrix multiplication, with each thread independently processing one or a group of pixels, ensuring pixel-level color accuracy. The color space conversion matrix is ​​a 3x3 or 4x4 linear transformation matrix used to convert the source color vector into the target color vector.

[0036] Specifically, the data, after color space adjustment, is written into a new video memory texture. This new texture serves as the preprocessed texture data for the spatial mapping input source, and its memory handle is passed to subsequent steps for use by the polyhedral mapping shader. The entire data flow in the preprocessing process is confined within the graphics processing unit's video memory, without involving data readback to system memory, thus ensuring microsecond-level processing latency. The pixel-level preprocessing encompasses one or more combinations of resolution adaptation, geometric clipping, scaling, pixel format conversion, and color space adjustment.

[0037] It should be noted that this implementation method avoids multiple memory copies and CPU interventions caused by format mismatch in traditional solutions by continuously performing resolution adaptation and color space adjustment in video memory. This significantly reduces the end-to-end latency in the preprocessing stage and ensures the consistency of geometric resolution and color performance of the image output to the multi-faceted display space, providing a high-quality input source for subsequent high-precision spatial mapping.

[0038] Based on the above embodiments, in some embodiments, based on a preset three-dimensional display space model and a virtual observer's viewpoint, the calculation of the viewing direction vector corresponding to each pixel on each display plane using a polyhedral mapping shader includes: Construct a set of multiple display planes, where each display plane is defined with a center position, normal vector, up vector, and physical dimensions of width and height, and define a virtual observer viewpoint located at the geometric center of the polyhedron space; For each pixel position on the display plane, calculate the corresponding 3D point coordinates based on the right vector, width, height of the plane, and the relative position of the pixel; Based on the calculated 3D point coordinates and the virtual observer's viewpoint, the standardized 3D line-of-sight direction vector is calculated.

[0039] In some embodiments, the specific implementation of calculating the viewing direction vector corresponding to each pixel on each display plane based on a preset three-dimensional display space model and a virtual observer viewpoint using a polyhedral mapping shader is as follows: the system constructs a set consisting of N display planes, with the input source being pre-configured three-dimensional space parameters, including the center position, normal vector, upward vector, and the width and height of the physical dimensions of each display plane, while defining a virtual observer viewpoint located at the geometric center of the polyhedral space.

[0040] In other embodiments, the processing includes calculating the local coordinate system basis vectors for each plane, specifically by normalizing the right vector through the cross product of the normal vector and the upward vector. The cross product is the vector product of two 3D vectors, with the resulting vector perpendicular to the original two vectors; the normalization involves dividing the vector by its magnitude to make it a unit vector. The output is display plane model data containing complete geometric definitions, serving as the basis for subsequent pixel mapping.

[0041] In one specific embodiment, the polyhedral mapping shader traverses the pixel positions on each display plane in parallel, with inputs consisting of normalized two-dimensional pixel coordinates and the aforementioned generated display plane model data. The normalized two-dimensional pixel coordinates refer to mapping the pixel's horizontal and vertical coordinates to values ​​between 0 and 1, where (0,0) represents a corner point of the plane, and (1,1) represents a diagonal point. The processing involves using a spatial mapping formula to map the two-dimensional pixels to three-dimensional world coordinates.

[0042] Specifically, for each display plane pixel position First, calculate the right vector of the plane:

[0043] in, The right vector of the i-th display plane is a unit vector pointing from the center of the plane to the right. This represents the vector normalization operation, which divides a vector by its magnitude to obtain a unit vector. This represents the cross product operation of three-dimensional vectors, where the resulting vector is perpendicular to the two input vectors. This represents the upward vector of the i-th display plane, which is a unit vector pointing upwards from the center of the plane; Let represent the normal vector of the i-th display plane, which is a unit vector perpendicular to the plane and pointing towards the observer.

[0044] Then, the corresponding three-dimensional point coordinates are calculated using the formula:

[0045] Where: $ This indicates the three-dimensional coordinates of a pixel on the display plane in the world coordinate system. The three-dimensional coordinates of the center position of the i-th display plane in the world coordinate system are denoted as . ; This represents the normalized horizontal coordinates of a pixel on the display plane, with values ​​ranging from 1 to 2. ; Represents the vertical normalized coordinates of a pixel on the display plane, with values ​​ranging from 1 to 2. ; This represents the physical width of the i-th display plane; This represents the physical height of the i-th display plane; Represents the right vector of the i-th display plane; This represents the upward vector of the i-th display plane.

[0046] Subsequently, based on the calculated 3D point coordinates and the virtual observer's viewpoint, the standardized 3D line-of-sight direction vector is calculated using the following formula:

[0047] in: The standardized 3D view direction vector is a unit vector pointing from the virtual observer's viewpoint to the coordinates of a 3D point. This indicates the three-dimensional coordinates of a pixel on the display plane in the world coordinate system. The position of the virtual observer's viewpoint in the world coordinate system is denoted as . It is located at the geometric center of the polyhedral space; This represents the vector normalization operation.

[0048] It should be noted that the output of this step is the gaze direction vector corresponding to each pixel. This vector serves as a key index for texture sampling, ensuring a precise mapping from the two-dimensional pixel position to the three-dimensional spatial direction, so that each pixel on the display plane can obtain a unique and correct gaze direction. The "right vector" is a unit vector pointing to the right from the center of the display plane, which, together with the normal vector and the upward vector, constitutes a local orthogonal coordinate system of the plane.

[0049] Based on the above embodiments, in some embodiments, the step of mapping the gaze direction vector to the preprocessed texture data for texture sampling and generating video memory frame buffer data corresponding to each display plane includes: The sampled cube face is determined based on the largest component among the absolute values ​​of each component of the standardized three-dimensional line-of-sight vector. Based on the determined cube face, the corresponding texture coordinates are calculated using the ratio between the corresponding components of the view direction vector; The preprocessed texture data is sampled using the calculated texture coordinates to obtain pixel colors, and the pixel colors are written into the frame buffer object or rendering target texture allocated to the display plane to generate the video memory frame buffer data corresponding to each display plane.

[0050] In some embodiments, the specific implementation of mapping the gaze direction vector to the preprocessed texture data for texture sampling to generate the video memory frame buffer data corresponding to each display plane is as follows: the system treats the preprocessed texture data as a cube map that wraps the virtual observer's viewpoint for texture sampling. First, the sampled cube face is determined based on the largest component among the absolute values ​​of each component of the standardized three-dimensional gaze direction vector. The largest component is calculated by comparing the magnitudes of the absolute values ​​of each component, that is, the axis direction corresponding to the component with the largest absolute value is taken as the sampling face.

[0051] In other embodiments, the maximum component is determined by the following formula:

[0052] in: This represents the maximum absolute value among the components of the line-of-sight vector; Represents the x-axis component of the line-of-sight vector d; Represents the y-axis component of the line-of-sight vector d; Represents the z-axis component of the line-of-sight vector d; This indicates the operation of taking the absolute value.

[0053] When the maximum component is determined to be At that time, the corresponding texture coordinates u and v are calculated using the following formula: and

[0054] in: This represents the horizontal component of the texture coordinates, with a value range of [0,1]. Represents the vertical component of the texture coordinates, with a value range of [0,1].

[0055] When the maximum component is determined to be the negative x-axis component (i.e.) Maximum and When the value is negative, a similar calculation logic is used, adjusting the sign according to the projection direction. When the maximum component is determined to be either a y-axis component or a negative y-axis component, the texture coordinates are calculated using the ratios of the x and y components and the z component of the viewing direction vector. When the maximum component is determined to be either a z-axis component or a negative z-axis component, the texture coordinates are calculated using the ratios of the x and z components and the y and z components of the viewing direction vector. This mapping method based on the maximum component determination ensures visual continuity at the corners of the polyhedron.

[0056] In one specific embodiment, the preprocessed texture data is sampled bilinearly or trilinearly using the calculated texture coordinates to obtain the pixel color. Bilinear sampling refers to a weighted average using four texels surrounding the texture coordinates, while trilinear sampling involves linear interpolation between two mipmap levels based on bilinear sampling. The final processing step involves writing the pixel color into the framebuffer object allocated to the display plane or the rendering target texture, with the output being the video memory framebuffer data corresponding to each display plane.

[0057] Specifically, the sampled cube face is determined based on the largest absolute value of each component of the standardized 3D view direction vector; based on the determined cube face, the corresponding texture coordinates are calculated using the ratio between the corresponding components of the view direction vector, where: When the maximum component is the x-axis component (i.e.) Maximum and When it is positive, use and The ratio is used to calculate texture coordinates; When the maximum component is the negative x-axis component (i.e.) Maximum and When it is negative, use and The ratio is used to calculate texture coordinates; When the largest component is the y-axis component, use and The ratio is used to calculate texture coordinates; When the maximum component is the negative y-axis component, use The ratio is used to calculate texture coordinates; When the maximum component is the z-axis component, use The ratio is used to calculate texture coordinates; When the maximum component is the negative z-axis component, use The ratio is used to calculate texture coordinates; The calculated ratio is then subjected to a linear transformation. Mapping to the [0,1] interval yields the final texture coordinates. .

[0058] It should be noted that this implementation method directly executes a mapping algorithm based on a three-dimensional geometric model within the video memory. By utilizing precise gaze vector calculation and cube mapping mechanism, it ensures the visual continuity and perspective accuracy of desktop content at the intersection of multiple display planes, effectively eliminating the geometric distortion caused by traditional two-dimensional stretching, and significantly improving the spatial consistency and immersive experience of multi-faceted immersive displays.

[0059] Based on the above embodiments, in some embodiments, integrating the video memory frame buffer data corresponding to each display plane into a unified logical display surface, and configuring a hardware synchronization signal to control multiple display output ports to atomically flip the logical display surface at the vertical synchronization moment includes: A logical back buffer is created as a unified logical display surface, and the video memory frame buffer data corresponding to each display plane is placed in a preset rectangular area within the logical back buffer through graphics processing unit commands. For graphics processing units that support hardware synchronization, multiple display output ports are configured as a synchronization group through the driver application programming interface. One of the master ports is designated to generate a synchronization signal and control the other slave ports to follow. Frame locking is enabled before calling the rendering function, so that the graphics processing unit waits for all display controllers to be ready and then atomically flips the display surfaces of all ports when the next vertical synchronization signal arrives.

[0060] In some embodiments, the specific implementation of integrating the video memory frame buffer data corresponding to each display plane into a unified logical display surface and configuring a hardware synchronization signal to control multiple display output ports to atomically flip the logical display surface at the vertical synchronization moment is as follows: In the logical display surface creation stage, the input source is the video memory frame buffer data of each of the multiple display planes generated in claim 5, and the processing action is to allocate a continuous logical back buffer in the video memory of the graphics processing unit as a unified logical display surface.

[0061] In other embodiments, the size of the buffer is configured as a combination of a total width and a maximum height. The total width is the sum of the widths of all display planes at their output resolutions or the span calculated according to a preset layout (e.g., sequentially stitched together according to the spatial arrangement of the display planes). The maximum height is the maximum value of the heights of all display planes at their output resolutions. Subsequently, the video memory frame buffer data corresponding to each display plane is copied or mapped to a preset rectangular area within the logical back buffer via commands within the graphics processing unit. The output is a unified logical display surface texture containing all multi-faceted display content. The preset rectangular area refers to a continuous pixel block in the logical back buffer corresponding to the output of that plane, predetermined based on the relative positions of each display plane in physical space.

[0062] In one specific embodiment, during the hardware synchronization configuration phase, the input sources are the system-detected status of the graphics processing unit supporting hardware synchronization and information on multiple connected display output ports. The processing action involves calling the synchronization group configuration function through the driver application programming interface to include the display output ports connected to multiple projectors or monitors into the same synchronization group, designating one physical port as the master port to generate a synchronization signal, and controlling the remaining ports as slave ports to follow the timing of the master port. Simultaneously, a frame locking function is enabled before calling the rendering function, causing the internal logic of the graphics processing unit to enter a waiting state until all display controllers report readiness. The frame locking function refers to a hardware mechanism that forces the refresh timing of multiple display output ports to remain synchronized with the same reference clock source.

[0063] Specifically, in the atomic flip output stage, the input source is the ready synchronization group state and the unified logical display surface. The processing action is that when the next vertical synchronization signal is detected, the graphics processing unit triggers the atomic flip operation based on the frame locking mechanism, and simultaneously switches the image data corresponding to each rectangular area in the logical back buffer to the front display buffer of each display output port, eliminating the refresh time difference between multiple display planes, realizing sub-millisecond refresh synchronization between multiple display planes, and ensuring that all planes start refreshing and output native video signals under the same hardware timing.

[0064] It should be noted that this implementation method effectively avoids timing jitter caused by software synchronization by constructing a unified logical backend buffer and combining it with a hardware-level frame locking mechanism. This achieves sub-millisecond refresh synchronization between multiple display planes, ensuring a high degree of visual consistency and continuity in the multi-faceted immersive display space. At the same time, it avoids screen tearing or misalignment caused by refresh asynchrony, significantly improving the system's real-time response capability and immersive experience.

[0065] Based on the above embodiments, in some embodiments, the source image data corresponding to the source image resource handle includes one or more combinations of operating system desktop composite output, real-time output frames of 3D rendering engine, video playback frame buffer content, or remote graphics output frames. The source image data resides in the video memory of the graphics processing unit throughout the entire process of acquisition, preprocessing, spatial mapping, and synchronous output.

[0066] In some embodiments, the source image data corresponding to the source image resource handle includes one or more combinations of operating system desktop compositing output, real-time output frames from a 3D rendering engine, video playback frame buffer content, or remote graphics output frames. When the source image data is operating system desktop compositing output, the input source is the video memory surface generated by the operating system desktop window manager after completing the compositing of the current frame. The processing action is to call the desktop copy interface to directly capture the video memory resource handle of the surface, and the output result is a texture handle pointing to the desktop image data.

[0067] In other embodiments, when the source image data is a real-time output frame from a 3D rendering engine, the input source is the back buffer of the exchange chain submitted by the rendering engine after completing scene rendering. The processing action is to intercept and extract the video memory address of the buffer before the rendering function of the graphics API is called, and the output result is a texture resource containing the latest rendered frame. When the source image data is the content of a video playback frame buffer, the input source is the YUV or RGB format frame data decoded and written to video memory by the video decoder hardware unit. The processing action is to directly lock the handle of the decoded output surface, and the output result is the video frame texture to be processed. When the source image data is a remote graphics output frame, the input source is the image surface reconstructed in the local GPU video memory by the remote graphics protocol stack. The processing action is to obtain the shared handle of the reconstructed surface, and the output result is the remote desktop image texture.

[0068] In one specific embodiment, the source image data can be a combination of the above-mentioned types, such as simultaneously acquiring desktop compositing output and 3D rendering engine output, and generating a composite source image through image fusion or overlay processing in video memory. After acquiring the source image resource handle of any one or more of the above combinations, the system immediately performs a resource conversion operation within the graphics processing unit, using the GPU command queue to convert the acquired handle into a shared texture resource that can be read by subsequent shaders. This conversion process is completed entirely within the video memory controller and does not involve the central processing unit.

[0069] Specifically, the shared texture resources generated in this way serve as input for subsequent processing. The corresponding source image data strictly resides in the video memory of the graphics processing unit throughout the entire process of acquisition, preprocessing, spatial mapping, and synchronous output. No copying operation to system memory occurs, ensuring that the data path is always located at the video memory level.

[0070] It should be noted that this implementation method effectively avoids the bandwidth consumption and time delay caused by memory copying and bus transmission in traditional solutions by directly locking multiple types of video memory source image resources and avoiding data readback to system memory throughout the process. This reduces the end-to-end data processing latency to the microsecond level, while ensuring data consistency and real-time performance of multiple types of graphics sources in multi-faceted display scenarios.

[0071] Please refer to Figure 2 , Figure 2 This invention provides a structural block diagram of a multi-sided display low-latency synchronous output device; the specific device may include: The video memory-level image acquisition module 100 is used to directly acquire the source image resource handle that has been rendered and resides in the video memory within the graphics processing unit, and convert the source image resource handle into a shared texture resource in the video memory. The in-memory preprocessing module 200 is used to call the compute shader to perform pixel-level preprocessing on the shared texture resource in the video memory, and generate preprocessed texture data that adapts to the multi-faceted display space resolution and color format. The multi-faceted spatial mapping module 300 is used to calculate the viewing direction vector corresponding to the pixel point on each display plane through a multi-faceted mapping shader based on a preset three-dimensional display space model and a virtual observer viewpoint, and to map the viewing direction vector to the preprocessed texture data for texture sampling to generate the video memory frame buffer data corresponding to each display plane. The synchronous output control module 400 is used to integrate the video memory frame buffer data corresponding to each display plane into a unified logical display surface, and configure hardware synchronization signals to control multiple display output ports to atomically flip the logical display surface at the vertical synchronization moment, so as to realize the synchronous output of multi-faceted display.

[0072] The multi-sided display low-latency synchronous output device of this embodiment is used to implement the aforementioned multi-sided display low-latency synchronous output method. Therefore, the specific implementation of the multi-sided display low-latency synchronous output device can be found in the embodiment section of the multi-sided display low-latency synchronous output method above. For example, the video memory-level image acquisition module 100, the video memory preprocessing module 200, the multi-sided spatial mapping module 300, and the synchronous output control module 400 are respectively used to implement steps S101, S102, S103, and S104 in the above-mentioned multi-sided display low-latency synchronous output method. Therefore, its specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0073] In some embodiments, the multi-faceted display low-latency synchronous output device is deployed in a computer system equipped with a high-performance graphics processing unit and includes four core components: a video memory-level image acquisition module, a video memory preprocessing module, a multi-faceted spatial mapping module, and a synchronous output control module.

[0074] In other embodiments, the video memory-level image acquisition module performs step S101, which involves directly acquiring the handle of a rendered source image resource residing in video memory within the graphics processing unit, and converting the source image resource handle into a shared texture resource within the video memory. This module achieves zero-copy video memory resource acquisition by hooking or intercepting the rendering functions of the graphics application programming interface (API), without involving the central processing unit (CPU). Specifically, in a Windows operating system environment, it calls the desktop copy API to directly access the output surface of the desktop compositor, or intercepts the Present call of the DirectX graphics interface or the vkQueuePresentKHR call of the Vulkan graphics interface at the driver layer. The output of this module is a shared texture resource that can be read by subsequent shaders.

[0075] In one specific embodiment, the in-memory preprocessing module is used to execute step S102, that is, to call the computation shader to perform pixel-level preprocessing on the shared texture resource in the video memory, generating preprocessed texture data adapted to the multi-faceted display space resolution and color format. This module utilizes the GPU's parallel computing architecture to complete resolution adaptation and color space conversion operations within the video memory. The preprocessing encompasses one or more combinations of resolution adaptation, geometric clipping, scale scaling, pixel format conversion, and color space adjustment. The output of this module is the preprocessed texture data, which serves as the input source for space mapping.

[0076] Specifically, the polyhedral spatial mapping module executes step S103, which involves calculating the viewing direction vectors corresponding to pixels on each display plane using a polyhedral mapping shader based on a preset 3D display space model and a virtual observer's viewpoint. These viewing direction vectors are then mapped to the preprocessed texture data for texture sampling, generating the video memory frame buffer data corresponding to each display plane. This module constructs a set of multiple display planes, each defined with a center position, normal vector, upward vector, and physical dimensions (width and height), and defines a virtual observer's viewpoint located at the geometric center of the polyhedral space. The output of this module is the video memory frame buffer data corresponding to each display plane.

[0077] In another specific embodiment, the synchronization output control module is used to execute step S104, which integrates the video memory frame buffer data corresponding to each display plane into a unified logical display surface, configures a hardware synchronization signal to control multiple display output ports to atomically flip the logical display surface at the vertical synchronization moment, and realizes synchronous output of multi-faceted displays. For graphics processing units that support hardware synchronization, this module configures multiple display output ports into a synchronization group through the driver application programming interface, designates one master port to generate a synchronization signal and controls other slave ports to follow, and enables frame locking before calling the rendering function. The output of this module is the video signal synchronously output by all display ports.

[0078] It should be noted that this device integrates functions that would require multiple independent devices in traditional solutions into a single device. The modules interact with each other through shared texture resources in the graphics processing unit's video memory, completely avoiding data readback to system memory or transmission over a network. This significantly reduces system complexity, maintenance costs, and end-to-end display latency. The video memory-level image acquisition module, in-memory preprocessing module, multi-faceted spatial mapping module, and synchronous output control module can be software implementations of computational shaders, mapping shaders, and driver layer control logic executed on the graphics processing unit, or they can be acceleration units implemented by dedicated hardware circuitry.

[0079] A specific embodiment of the present invention also provides a multi-sided display low-latency synchronous output device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-described multi-sided display low-latency synchronous output method.

[0080] A specific embodiment of the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for low-latency synchronous output of multi-faceted displays.

[0081] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0083] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0085] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for low-latency synchronous output of multi-faceted displays, characterized in that, include: The source image resource handle that has been rendered and resides in the video memory is directly obtained within the graphics processing unit, and the source image resource handle is converted into a shared texture resource in the video memory. The compute shader is invoked to perform pixel-level preprocessing on the shared texture resource in the video memory, generating preprocessed texture data adapted to the multi-faceted display space resolution and color format; Based on the preset 3D display space model and virtual observer viewpoint, the viewing direction vector corresponding to the pixel point on each display plane is calculated by the polyhedral mapping shader, and the viewing direction vector is mapped to the preprocessed texture data for texture sampling to generate the video memory frame buffer data corresponding to each display plane. The video memory frame buffer data corresponding to each display plane is integrated into a unified logical display surface, and a hardware synchronization signal is configured to control multiple display output ports to atomically flip the logical display surface at the vertical synchronization moment, so as to realize the synchronous output of multi-faceted display.

2. The method according to claim 1, characterized in that, The step of directly acquiring the source image resource handle that has been rendered and resides in the video memory within the graphics processing unit, and converting the source image resource handle into a shared texture resource in the video memory includes: By hooking or intercepting the rendering function of the graphics application interface, after the operating system desktop compositor completes the rendering of a frame, the video memory resource handle pointing to the frame image can be directly obtained. The video memory resource handle corresponds to the texture resource stored in the video memory. The system calls internal commands from the graphics processing unit to directly convert the acquired desktop texture into a shared texture resource that can be read by subsequent shaders in video memory, generating the converted shared texture resource. The conversion process does not involve the central processing unit.

3. The method according to claim 1, characterized in that, The step of calling the compute shader to perform pixel-level preprocessing on the shared texture resource in the video memory to generate preprocessed texture data adapted to the multi-faceted display space resolution and color format includes: The scheduling compute shader reads the original pixel data from the shared texture resource, performs a resolution adaptation operation based on the physical resolution parameters of the multi-faceted display space, and generates resolution-matched intermediate texture data. Based on a preset color space conversion matrix, the intermediate texture data matching the resolution is subjected to color space adjustment, and the adjusted data is written into a new video memory texture to obtain preprocessed texture data as the input source for space mapping.

4. The method according to claim 1, characterized in that, The calculation of the viewing direction vector corresponding to each pixel on each display plane using a polyhedral mapping shader, based on a preset three-dimensional display space model and a virtual observer's viewpoint, includes: Construct a set of multiple display planes, where each display plane is defined with a center position, normal vector, up vector, and physical dimensions of width and height, and define a virtual observer viewpoint located at the geometric center of the polyhedron space; For each pixel position on the display plane, calculate the corresponding 3D point coordinates based on the right vector, width, height of the plane, and the relative position of the pixel; Based on the calculated 3D point coordinates and the virtual observer's viewpoint, the standardized 3D line-of-sight direction vector is calculated.

5. The method according to claim 4, characterized in that, The step of mapping the gaze direction vector to the preprocessed texture data for texture sampling, and generating video memory frame buffer data corresponding to each display plane includes: The sampled cube face is determined based on the largest component among the absolute values ​​of each component of the standardized three-dimensional line-of-sight vector. Based on the determined cube face, the corresponding texture coordinates are calculated using the ratio between the corresponding components of the view direction vector; The preprocessed texture data is sampled using the calculated texture coordinates to obtain pixel colors, and the pixel colors are written into the frame buffer object or rendering target texture allocated to the display plane to generate the video memory frame buffer data corresponding to each display plane.

6. The method according to claim 1, characterized in that, The step of integrating the video memory frame buffer data corresponding to each display plane into a unified logical display surface, and configuring a hardware synchronization signal to control multiple display output ports to atomically flip the logical display surface at the vertical synchronization moment includes: A logical back buffer is created as a unified logical display surface, and the video memory frame buffer data corresponding to each display plane is placed in a preset rectangular area within the logical back buffer through graphics processing unit commands. For graphics processing units that support hardware synchronization, multiple display output ports are configured as a synchronization group through the driver application programming interface. One of the master ports is designated to generate a synchronization signal and control the other slave ports to follow. Frame locking is enabled before calling the rendering function, so that the graphics processing unit waits for all display controllers to be ready and then atomically flips the display surfaces of all ports when the next vertical synchronization signal arrives.

7. The method according to claim 1, characterized in that, The source image data corresponding to the source image resource handle includes one or more combinations of operating system desktop composite output, real-time output frames of 3D rendering engine, video playback frame buffer content, or remote graphics output frames. The source image data resides in the video memory of the graphics processing unit throughout the entire process of acquisition, preprocessing, spatial mapping, and synchronous output.

8. A multi-sided display low-latency synchronous output device, characterized in that, include: The video memory-level image acquisition module is used to directly acquire the source image resource handle that has been rendered and resides in the video memory within the graphics processing unit, and convert the source image resource handle into a shared texture resource in the video memory. The in-memory preprocessing module is used to call the compute shader to perform pixel-level preprocessing on the shared texture resource in the video memory, and generate preprocessed texture data that adapts to the multi-faceted display space resolution and color format. The multi-faceted space mapping module is used to calculate the viewing direction vector corresponding to the pixel point on each display plane through the multi-faceted mapping shader based on the preset three-dimensional display space model and virtual observer viewpoint, and to map the viewing direction vector to the preprocessed texture data for texture sampling to generate the video memory frame buffer data corresponding to each display plane. The synchronous output control module is used to integrate the video memory frame buffer data corresponding to each display plane into a unified logical display surface, and configure hardware synchronization signals to control multiple display output ports to atomically flip the logical display surface at the vertical synchronization moment, so as to realize the synchronous output of multi-faceted display.

9. A multi-sided display low-latency synchronous output device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of a low-latency synchronous output method for multi-faceted displays as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the low-latency synchronous output method for multi-faceted displays as described in any one of claims 1 to 7.

Images