Cross-platform rendering method under hybrid architecture

Abstract

The present application relates to the technical field of computer graphics, and especially relates to a cross-platform rendering method under a hybrid architecture, which comprises the following steps: using eye movement coordinates to split an Android current frame command into a fixation area and a peripheral area, generating a difference and mapping with resources, encapsulating check information into an incremental package, and writing the incremental package into desktop video memory through a one-way bus; after checking on the desktop side, reconstructing the fixation area command, distilling and simplifying the peripheral area shader, generating a first and a second command buffer, and synchronously submitting the first and the second command buffer through a timeline semaphore; after the graphic processing unit completes the two buffers, if the actual state of the window deviates from the prediction matrix by more than a threshold value, then real-time correction of texture is performed in the pixel by solving Laplace; when there are continuous missing packages, only the fixation area is super-sampled and a frame is supplemented, and finally a synthesized frame is output and adjustment information is written.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, and more particularly to a cross-platform rendering method under a hybrid architecture. Background Technology

[0002] With the surge in the number of mobile applications, many desktop systems have begun to incorporate Android compatibility environments, aiming to run desktop and Android applications simultaneously on the same graphics processor. Traditional industry solutions largely follow two paths:

[0003] 1. Full-frame shared video memory copying: At the end of each frame, the Android side copies the entire rendering result to main memory and then sends it to the rasterizer on the desktop side for secondary processing. Since the copying operation increases linearly with resolution, it can take more than 50ms in a 4K windowed scene, resulting in noticeable stuttering.

[0004] 2. Unified Driver Multi-Context: This approach uses driver-level interlocks to allow desktop and Android commands to be executed serially in the same command queue. This method requires bidirectional synchronization of GPU state; if an erroneous call blocks the desktop side, the Android side will be blocked, and compatibility between different API versions is poor.

[0005] The above technologies all use "complete frames" as the smallest exchange unit or rely on complex bidirectional synchronization, which easily leads to four types of defects: (1) bandwidth waste, (2) uncontrollable latency, (3) risk of resource tampering, and (4) black borders of dynamic window deformation. They are difficult to meet the desktop experience of high refresh rate above 120Hz and free window scaling. Summary of the Invention

[0006] To address the numerous problems existing in the prior art, this invention provides a cross-platform rendering method under a hybrid architecture. This invention starts with viewpoint-driven command differentiation, sending incremental rendering data packets to the desktop video memory via a one-way write-only bus. On the desktop side, timeline semaphores are used to sequentially execute the reconstructed gaze area command and the distilled peripheral area command. If the prediction window error exceeds the limit, the texture coordinates are corrected in real time by solving the discrete Laplace equation at the fragment stage; if data packets are missing, super-resolution interpolation is performed on the gaze area frame buffer.

[0007] A cross-platform rendering method under a hybrid architecture includes the following steps:

[0008] Based on the viewpoint coordinates obtained from eye tracking, the current frame command stream in the Android compatible environment is divided into gaze area commands and peripheral area commands. The gaze area commands are compared with the data of the previous frame to generate differential data. The differential data and peripheral area commands are encoded and encapsulated together with the graphics resource mapping table and integrity verification information into an incremental rendering data packet.

[0009] The incremental rendering data packet is written to the desktop operating system video memory via a unidirectional interconnect bus. After verifying its integrity, the gaze area command is reconstructed and the peripheral area shader is simplified. The first command buffer and the second command buffer are generated. The two buffers and the prediction window matrix are submitted to the graphics processing queue synchronized with the timeline.

[0010] After the graphics processor executes the first command buffer and the second command buffer, it obtains the actual window state and compares it with the predicted window matrix. If the deviation exceeds the threshold, it adjusts the texture coordinates of the gaze area and the surrounding area frame buffer in the fragment shader, and then composites them with the desktop layer. When the incremental rendering data packets are continuously missing, it performs frame interpolation. It outputs the composite frame and writes the texture coordinate adjustment information into shared storage for the next frame to call.

[0011] Preferably, when performing differential operations on the gaze area command, a hash value is calculated for the handle and fixed parameter value of each drawing instruction in the gaze area command, and then XORed with the hash value at the corresponding position in the previous frame to obtain differential data.

[0012] Preferably, the obtained differential data is converted into a frequency domain representation and XORed with the fingerprint value of the previous frame to obtain an updated fingerprint value. The peripheral area command is compressed by the run length and then encapsulated together with the updated fingerprint value into an incremental rendering data packet.

[0013] Preferably, the graphics resource map table records the file descriptors generated for each texture object through the graphics memory import interface, and sets the file descriptors as read-only aliases.

[0014] Preferably, the unidirectional interconnect bus is a high-speed peripheral interconnect channel containing a field-programmable gate array, wherein the high-speed peripheral interconnect channel only allows Android-compatible environments to write data to the desktop operating system side and blocks reverse read requests.

[0015] Preferably, the simplified peripheral shader includes: parsing the peripheral command into a first intermediate representation and a second intermediate representation, inputting the first intermediate representation and the second intermediate representation into a graph neural network distillation model to obtain polynomial kernel parameters, and replacing the fragment stage operations in the peripheral shader with the polynomial kernel parameters.

[0016] Preferably, when the first command buffer is committed, the timeline semaphore value is incremented by one. Before the second command buffer is committed, the timeline semaphore value is waited for to reach the incremented value. After the commit is completed, the timeline semaphore value is incremented again.

[0017] Preferably, the prediction window matrix is ​​obtained by performing second-order Kalman filtering on the window position and window size of consecutive frames, and is used as the reference matrix for viewpoint weight calculation and differential data generation in the next frame encoding stage.

[0018] Preferably, when adjusting texture coordinates, the Gaussian Seidel iteration is performed multiple times in the fragment shader to solve the discrete Laplace equation for the gaze area frame buffer and the peripheral area frame buffer, respectively, in order to obtain the texture coordinate adjustment vector and update the fragment sampling position in real time.

[0019] Preferably, the frame interpolation process performs deep learning super-resolution model inference with residual blocks as input only on the gaze region frame buffer, and replaces the current frame image in the gaze region frame buffer with the high-resolution image obtained from the inference.

[0020] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0021] Command stream splitting based on viewpoint weights: Using eye-tracking results, differential data is generated only for commands in the gaze region, and the data packets are transmitted through incremental rendering, which significantly reduces bandwidth.

[0022] Unidirectional interconnect bus + integrity verification: Hardware write-only design blocks reverse read and write on the desktop side; dual verification avoids link errors and achieves secure zero copy.

[0023] Online distillation of peripheral region shaders for graph neural networks: This reduces complex fragment logic to polynomial kernels, thereby reducing GPU peripheral overhead and addressing the problem of uncontrollable latency.

[0024] Timeline semaphore synchronization: Use single-object write-etc. sequence to force the gaze area to render first, preventing deadlock and eliminating unnecessary waiting.

[0025] Second-order Kalman prediction + Laplacian coordinate compensation: Combining the prediction window matrix with fragment-level texture adjustment to eliminate black borders caused by window stretching.

[0026] Residual block driven super-resolution interpolation: When packet loss occurs, super-resolution is only performed on the area of ​​focus to maintain frame sequence continuity and visual clarity. Attached Figure Description

[0027] Figure 1 This is a schematic flowchart of the method of the present invention;

[0028] Figure 2 This is a schematic diagram of the peripheral shader distillation process of the present invention. Detailed Implementation

[0029] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation.

[0030] like Figure 1 As shown, a cross-platform rendering method under a hybrid architecture includes the following steps:

[0031] Based on the viewpoint coordinates obtained from eye tracking, the current frame command stream in the Android compatible environment is divided into gaze area commands and peripheral area commands. The gaze area commands are compared with the data of the previous frame to generate differential data. The differential data and peripheral area commands are encoded and encapsulated together with the graphics resource mapping table and integrity verification information into an incremental rendering data packet.

[0032] In the hybrid architecture of this invention, the desktop operating system and the Android-compatible environment share the same graphics processor. To avoid black borders and flickering when windows are scaled or the resolution changes, this step proposes a "viewpoint-driven incremental rendering data packet encapsulation principle".

[0033] In an Android-compatible environment, each frame's command stream consists of several drawing instructions. The user's gaze region can be determined by the viewpoint coordinates obtained through eye tracking. Drawing instructions corresponding to the gaze region are defined as gaze region commands, and the rest are defined as peripheral region commands. The gaze region typically occupies only about 10% of the screen area, but it receives the most user attention. This invention only performs frame-by-frame differential processing on gaze region commands, comparing the differential data with the corresponding data from the previous frame; peripheral region commands, which vary significantly, are compressed as a whole. This greatly reduces the amount of data transmitted across the stack.

[0034] The implementation process includes:

[0035] Step 1: The system calls the eye-tracking interface to obtain the viewpoint coordinates and stores them in the register.

[0036] Step 2: Parse the current frame command stream and divide it into gaze area commands and peripheral area commands based on the viewpoint coordinates.

[0037] Step 3: Calculate the hash of the handle and fixed parameters for the gaze area command and the command at the same position in the previous frame to obtain differential data.

[0038] Step 4: Compress the commands in the surrounding area using run-length encoding.

[0039] Step 5: Read the graphics resource mapping table, generate a file descriptor for each texture object, and mark it as read-only.

[0040] Step 6: Generate integrity verification information.

[0041] Step 7: Encapsulate the differential data, compressed peripheral area commands, graphics resource mapping table, and integrity verification information into an incremental rendering data packet, and write it to the desktop-side video memory via a unidirectional interconnect bus.

[0042] Let the gaze area command sequence be The sequence corresponding to the previous frame is . No. The hash of the drawing instruction is The difference is calculated as follows:

[0043]

[0044] in The hash of the same position in the previous frame. This indicates a bitwise XOR operation. If This indicates that the instructions have not changed and can be omitted. Differential dataset By all The items are composed of [items] and written into the data packet along with the compressed peripheral area commands.

[0045] Using this method, the incremental rendering data packet contains only the gaze area differential data and compressed peripheral area commands, with a data volume of approximately 25% of the complete command stream; cross-stack transmission latency is reduced by about 60%, and the black box duration during window scaling is significantly shortened. Integrity verification information ensures that the data packet can be quickly verified and corrected after unidirectional link transmission, avoiding screen tearing.

[0046] Example 1: In a 1920×1080 pixel window, the viewpoint is centered. The current frame resolves 1000 drawing instructions, of which 200 are contained within the gaze area. Compared to the previous frame, only 40 instructions have changed, resulting in a difference set. 40 items are saved, and 160 items are omitted. The 800 instructions in the peripheral area are compressed at a ratio of approximately 3:1 using run-length encoding. The final data packet size is approximately 27% of the complete command stream, and the transmission time on the unidirectional interconnect channel is less than 0.5 milliseconds. After verification on the desktop side, the differential signal is reconstructed and the peripheral area commands are decoded to achieve re-rendering consistent with the Android side screen, maintaining a frame rate of 120 Hz with no visible flicker.

[0047] Preferably, when performing differential operations on the gaze area command, a hash value is calculated for the handle and fixed parameter value of each drawing instruction in the gaze area command, and then XORed with the hash value at the corresponding position in the previous frame to obtain differential data.

[0048] The hybrid architecture cross-platform rendering method of the present invention adopts a handle-parameter hash comparison mechanism in the gaze area command differential stage, so that only the necessary drawing instruction increments are transmitted between the Android compatible environment and the desktop operating system.

[0049] In scenarios involving window scaling and resolution changes, Android-compatible environments generate a complete sequence of rendering instructions per frame. If all instructions are transmitted to the desktop graphics processor via a unidirectional link, significant bandwidth and latency costs are introduced. This invention categorizes rendering instructions into gaze area commands and peripheral area commands based on the user's viewpoint coordinates. Inter-frame differencing is performed only on gaze area commands, while peripheral area commands are compressed as a whole, significantly reducing incremental rendering data packets. Viewpoint coordinates are provided in real-time by the eye-tracking module; the graphics capture layer clips the gaze area rectangle based on the coordinates and filters out intersecting instructions.

[0050] The implementation details are as follows:

[0051] Step 1: The eye-tracking module outputs the viewpoint coordinates at a sampling frequency of 120 times per second.

[0052] Step 2: The capture layer parses the command stream of the current frame and determines that the drawing commands that intersect with the gaze region are gaze region commands, and the rest are peripheral region commands.

[0053] Step 3: Generate a handle hash and a parameter hash for each gaze region command, concatenate the two, and obtain an 8-byte hash code using the truncated secure hash function SHA-256_8. .

[0054] Step 4: Put all Construct a hash array and the hash array of the previous frame Perform an XOR comparison to obtain the difference set. .

[0055] Step 5: Compress the surrounding area command using run length encoding.

[0056] Step 6: Generate a graphics resource mapping table: Call the external memory import interface for each new texture object to obtain the file descriptor and mark it as read-only.

[0057] Step 7: After performing block-level and packet-level dual verification on the differential set, compressed peripheral area commands and mapping table, the data is encapsulated into an incremental rendering data packet and written to the desktop operating system's video memory via the PCIe-FPGA unidirectional bus.

[0058] For the The command to draw the gaze area has a handle of 1. The fixed parameter string is Hash code:

[0059]

[0060] Inter-frame difference:

[0061]

[0062] in The hash code of the same index as the previous frame. This indicates a bitwise XOR operation. If If the instruction remains unchanged between the two frames, then the instruction will not change; if Then Add to difference set . Indicates the current frame number The hash code of the drawing instruction; This indicates the hash code of the instruction corresponding to the previous frame; Indicates the difference result; This represents the set of non-zero differences in this frame.

[0063] This invention achieves a 20% width and 20% height for the gaze region within a 1920×1080 pixel window. Tests show that approximately 1000 rendering commands are parsed per frame, with 200 of them in the gaze region. After cross-frame comparison, only an average of 40 gaze region commands change; therefore, the differential set size is 40×8 bytes, approximately 320 bytes. Peripheral region commands, after run-length encoding, are approximately 9000 bytes. The total size of the incremental rendering data packet is approximately 9320 bytes, only 29% of the complete command stream (approximately 32000 bytes), resulting in bandwidth savings of over 70%. Actual measured transmission latency for a one-way link is less than 0.4 milliseconds, which can be completed within a refresh cycle of 120 frames per second. Double verification ensures a low transmission error rate. The level can still be detected and rolled back within milliseconds. This method demonstrates stability with black box duration of less than 4 milliseconds and no visible tearing in both action games and video players.

[0064] Example 2 demonstrates an application containing a central rotating cube and four peripheral transparent sprites. The viewpoint coordinates are stable at the center of the cube. The total number of rendering commands in the current frame is 1020, including 200 commands for the gaze area and 820 commands for the periphery area. Compared to the previous frame, 10 commands in the gaze area have different constant buffer offsets due to changes in lighting, causing hash changes. The hash values ​​are calculated using the difference formula. There are 10 items in total, with a differential set size of 80 bytes. Peripheral area instructions, due to the use of a continuous vertex buffer, are compressed at a ratio of approximately 3:1 when represented as a "count-instruction" pair, reducing the size from 32,800 bytes to 10,933 bytes. The total data packet size is approximately 11,013 bytes, a 66% reduction compared to the complete command stream. The data packet is written to the desktop video memory via the PCIe-FPGA bridge, taking 0.37 milliseconds. After successful verification on the desktop, the gaze area instructions are reconstructed, the peripheral area commands are decoded, and submitted to the graphics processor for rendering. The entire process does not introduce a frame rate drop; the window maintains continuous rendering during scaling without any black borders.

[0065] Through the above mechanism, this invention achieves instruction-level differential in the gaze area, efficient compression of the peripheral area, and dual verification of the link under a hybrid architecture, significantly reducing transmission latency and bandwidth usage. In practical applications, it has proven its effectiveness for high refresh rate displays and dynamic interactive scenarios.

[0066] Preferably, the obtained differential data is converted into a frequency domain representation and XORed with the fingerprint value of the previous frame to obtain an updated fingerprint value. The peripheral area command is compressed by the run length and then encapsulated together with the updated fingerprint value into an incremental rendering data packet.

[0067] This invention introduces a "differential frequency domain fingerprint update" mechanism when transmitting command increments between an Android-compatible environment and a desktop operating system, in order to further compress the size of the data packet in the attention area and improve the link's fault tolerance.

[0068] The gaze region differential data exhibits sparse and discrete variations in the spatial domain, but the same instruction often maintains local correlation in adjacent frames. Directly compressing the differential set at the byte level makes it difficult to fully exploit this correlation. This invention first performs a discrete Fourier transform on the differential vector to obtain its frequency domain representation, and then XORs it with the frequency domain fingerprint value of the previous frame to generate a new fingerprint. The frequency domain XOR can simultaneously accomplish two tasks: first, it utilizes the sparsity of high-frequency coefficients to encode repetitive fluctuations as zero values; second, it performs integrity verification updates without adding extra overhead through the XOR operation. When comparing again in the next frame, only one XOR operation on the new fingerprint is needed to determine whether the high-frequency fluctuations have changed, saving the rehashing overhead.

[0069] The specific steps to achieve this include:

[0070] Step 1: Difference Generation. The capture layer compares the drawing instruction handle for the gaze area with fixed parameters to obtain the difference vector. The element order is consistent with the instruction index.

[0071] Step 2, Frequency Domain Transformation. Call the single-precision Fast Fourier Transform kernel to transform... Zero padding to length (Take the nearest power of two), output the frequency domain vector. .

[0072] Step 3: Fingerprint XOR. Read the frequency domain fingerprint from the previous frame. ,implement:

[0073]

[0074] Obtain the updated frequency domain fingerprint .in All are lengths Complex vectors, This indicates a byte-wise XOR operation; before the XOR operation, the real and imaginary parts are quantized into four-byte integers respectively.

[0075] Step 4: Compression of surrounding area travel length. Traverse the command sequence of the surrounding area and record consecutively repeated commands as "count-command" pairs.

[0076] Step 5: Packet encapsulation. The quantized byte stream, run-length compression result, graphics resource mapping table, block-level cyclic redundancy check code, and packet-level hash are sequentially written into the fixed layout; the packet header records the field offset and byte length.

[0077] Step 6, One-way transmission. The data packet is written to the desktop-side video memory via the PCIe-FPGA channel. After verification by the desktop driver, it is unpacked. During unpacking, the peripheral area commands of the run-length compression are restored first, and then... With local cache Inverse XOR and perform inverse Fourier transform to quickly reconstruct the difference vector. To complete the gaze area instructions.

[0078] Discrete Fourier Transform:

[0079]

[0080] in Represents the difference vector. One element; Represents the frequency domain vector. One element; Indicates the transformation length.

[0081] New fingerprint generation:

[0082]

[0083] Indicates the number of frames in this frame. A frequency domain fingerprint; Indicates the character corresponding to the previous frame; This indicates a byte-wise XOR operation.

[0084] Using this invention, the experimental platform was configured as an eight-channel PCIeGen3×4 unidirectional bridge. Testing was conducted while running a high-speed shooting game in a 1920×1080 pixel window, with the gaze area occupying approximately 20% of the full screen. Using the traditional differential hashing scheme, the gaze area increment averaged 1200 bytes; after introducing frequency domain fingerprinting, this decreased to 740 bytes, improving the compression ratio by approximately 38%. Simultaneously, frequency domain XOR inherently carries inter-frame integrity verification, eliminating the need to separately calculate the gaze area block-level checksum, effectively saving 32 bytes per frame. The average cross-stack transmission time decreased from 0.41 milliseconds to 0.32 milliseconds, increasing the alignment margin with the desktop's 120Hz refresh cycle by 1.1 milliseconds. After 24 hours of continuous stability testing, the desktop-side instruction reconstruction success rate was 100%, with no detected pseudo-differential or instruction misalignment phenomena.

[0085] In Example 3, in the demonstration scenario, the current frame's gaze region difference vector length is 64, with 10 non-zero elements. A Fast Fourier Transform (FFT) generates 64 complex spectrum terms, each quantized to 8 bytes, totaling 512 bytes. The fingerprint length from the previous frame is the same; after bitwise XOR, the new fingerprint is still 512 bytes. The peripheral region run-length compression result is 1800 bytes. Compared to the traditional scheme, removing the gaze region block-level check saves 32 bytes, resulting in a final data packet size of 2312 bytes, approximately 17% smaller than the 2784 bytes without frequency domain fingerprinting. The desktop side receives the data and completes the inverse Fourier transform and difference concatenation in 80 microseconds, maintaining a stable rendering frame rate of 144 Hz, with no visible lag during rapid window stretching.

[0086] In summary, this invention achieves secondary compression and verification merging of gaze region instructions through differential vector frequency domain transformation and fingerprint XOR update, further reducing the incremental rendering data packet size and ensuring low latency and high reliability of cross-stack rendering links in the hybrid architecture.

[0087] Preferably, the graphics resource map table records the file descriptors generated for each texture object through the graphics memory import interface, and sets the file descriptors as read-only aliases.

[0088] The graphics resource mapping table proposed in this invention is a key structure connecting the Android-compatible environment and the desktop operating system's video memory. It maps graphics resources such as textures and vertex buffers without copying video memory data. Its core principle is to use the graphics memory import interface provided by the operating system to convert the video memory pages allocated on the Android side into file descriptors that can be directly accessed on the desktop side, and store one-to-one entries in the mapping table. Each entry consists of a resource handle, a file descriptor, a resource type identifier, and a read-only flag. The read-only flag is set directly during the import phase, ensuring that the desktop side can only read and cannot modify the Android-side resources, thus architecturally avoiding the risk of bidirectional synchronization conflicts and data corruption.

[0089] In hybrid architecture workflows, after an Android application creates a texture via the graphics API, the driver stores the texture content in physical pages of video memory. To continue using the same texture on the desktop, cross-process and even cross-operating system access to video memory is required. Traditional solutions necessitate copying the video memory data and sending it to the desktop, doubling the bandwidth. This invention employs an external memory import interface, directly exporting video memory pages as file descriptors. The desktop then establishes a read-only alias in its address space using the same file descriptor and offset information. In this way, the Android side handles resource filling, while the desktop side handles sampling and rendering; the entire process eliminates the need for intermediate buffer copying.

[0090] The implementation process includes: First, at the end of each frame, the Android side enumerates the texture objects used in that frame and calls `\verb|vkGetMemoryFdKHR|` or an equivalent interface to allocate file descriptors for video memory. Second, the file descriptors and texture handles are written to the graphics resource mapping table, and the resource type is marked as texture. Third, `\verb|fcntl|` is executed on the file descriptors to set a read-only flag and disable inheritance to prevent permission leakage during desktop process spawning. Fourth, the mapping table is serialized and added to the resource segment of the incremental rendering data packet. Fifth, after the data packet is written to the desktop-side video memory via the unidirectional interconnect bus, the desktop driver reads the mapping table, imports the file descriptors into the local device via `\verb|vkImportMemoryFdKHR|`, and generates a read-only video memory alias; finally, during the command recording stage, the corresponding descriptor set is updated for each drawing instruction.

[0091] If the texture object occupies a set of video memory pages The set of file descriptors is denoted as Mapping table entries can be abstracted as follows:

[0092]

[0093] in Represents the texture handle; Indicates the export file descriptor; Indicates the resource type identifier; This indicates a read-only flag (Boolean value). All data will be serialized during the mapping table serialization. Write the data sequentially to the resource segment, and attach a four-byte entry count field for quick parsing by the desktop driver.

[0094] This invention saves bandwidth. Traditional copying paths require transmitting a complete texture data on the bus; for example, a 4096×4096 RGBA texture is 64 megabytes in size. This method only requires transmitting a 64-byte mapping table entry, saving over 99.9% of bandwidth. Latency is reduced; file descriptor export and import are constant-time system calls, with the additional overhead per frame consistently below 40 microseconds; copying the same texture takes approximately 8 milliseconds. Combined with incremental rendering data packet encapsulation, the total frame latency can be controlled within 7 milliseconds, ensuring no frame drops at 120Hz refresh rates.

[0095] Example 4: In a racing game, the track texture occupies eight 4096-pixel square textures. On the Android side, the eight texture handles are converted into eight file descriptors through an import interface. Each entry contains a 32-bit handle, a 32-bit file descriptor, an 8-bit resource type, and a 1-bit read-only flag, totaling 73 bytes, which becomes 80 bytes after 4-byte alignment. The mapping table segment, plus the 4-byte entry count field, totals 84 bytes. After being transmitted to the desktop side, the driver imports the file descriptors using the O_CLOEXEC|O_RDONLY mode, creates a read-only video memory alias, and updates the descriptor set in the pipeline layout. Test results show that when using the zero-copy scheme, the total latency of the texture loading stage is reduced from the original 85 milliseconds to 12 milliseconds, and the game frame rate is increased by about 15% at high resolution. At the same time, since the desktop side only holds the read-only mapping, the Android side does not need to notify the desktop side to refresh when dynamically updating the track billboard content. The two still share the same video memory page in subsequent frames, avoiding the failure of multiple resources.

[0096] By generating a read-only file descriptor for each texture object and writing it to the graphics resource mapping table, this invention achieves zero-copy sharing of video memory resources in a hybrid architecture. Combined with a unidirectional interconnect bus and incremental rendering data packet encapsulation, bandwidth and latency can be significantly reduced without increasing memory usage, ensuring smooth rendering at high refresh rates, and maintaining cross-system resource consistency and security isolation through a read-only alias mechanism.

[0097] The incremental rendering data packet is written to the desktop operating system video memory via a unidirectional interconnect bus. After verifying its integrity, the gaze area command is reconstructed and the peripheral area shader is simplified. The first command buffer and the second command buffer are generated. The two buffers and the prediction window matrix are submitted to the graphics processing queue synchronized with the timeline.

[0098] The incremental rendering data packets of this invention, after being generated in an Android-compatible environment, are transmitted to the desktop operating system's video memory via a unidirectional interconnect bus. This section details the entire process of data packet writing, integrity verification, gaze area command reconstruction, peripheral area shader simplification, and submission of the first and second command buffers, and elucidates its effects on reducing latency and improving stability.

[0099] The unidirectional interconnect bus employs a high-speed peripheral interconnect channel including a field-programmable gate array (FPGA). On the Android side, the packet header is written to a reserved register before being sent, followed by a zero-copy write triggered by the direct memory access engine. Because this bus only allows write operations, the desktop side cannot read the Android video memory in reverse, thus preventing potential resource contention at the hardware level. After the write operation is complete, the FPGA triggers an interrupt to the desktop side, transmitting the packet's starting address and byte length.

[0100] The desktop driver first reads the packet-level hash field, recalculates and compares the received data using a one-way hash function; if inconsistent, the frame is discarded and a retransmission is requested. A block-level cyclic redundancy check (CRC) field is used to further locate potential bit flips; if a block fails the check, only that block is retransmitted, avoiding a full frame rollback. After successful check, the reconstruction phase begins: the driver calls the import interface based on the file descriptors in the resource segment, creating a read-only memory alias for each texture object and filling it into the descriptor set. Then, the differential segment is parsed, and the gaze area command in the previous frame's buffer is located by index. The changed fields are overwritten with the original instructions to reconstruct the complete gaze area command. The surrounding segments are decoded using run-length decoding to restore the drawing instruction sequence.

[0101] To reduce pipeline overhead in the peripheral region, this invention performs online distillation of the peripheral region fragment shader on the desktop side. The driver decompiles the original shader into an intermediate representation of the graphics, which is then passed to a pre-trained graph neural network to output polynomial kernel parameters, replacing the original complex computations. This process takes an average of 0.6 milliseconds, significantly reducing the computational power consumption of subsequent rasterization stages.

[0102] After reconstruction is complete, the driver calls the graphics programming interface to input the first command buffer for drawing instructions for the gaze area and the second command buffer for drawing instructions for the peripheral area. To ensure that the gaze area is rendered first and the peripheral area is rendered later, this invention uses a timeline semaphore for precise synchronization. If the semaphore value of the previous frame is... The submission order then satisfies:

[0103]

[0104] in Corresponding to the first command buffer, This corresponds to the second command buffer. The driver sets the wait value to [value] when submitting the second command buffer. This ensures that the view area layer is rasterized before the surrounding area layers. In the expression... and The meaning is: This indicates the semaphore value of the previous frame's timeline; This indicates the semaphore value written when the first command buffer is committed; This indicates the semaphore value written when the second command buffer is submitted.

[0105] Before recording the command buffer, the driver writes the predicted window matrix given by the window state predictor into a unified parameter buffer, allowing the two command buffers to share the same viewport and projection transformation. The predicted window matrix is ​​calculated by a second-order Kalman filter, which can pre-correct coordinate offsets caused by window dragging, reducing the number of texture coordinate compensation operations performed at the fragment stage.

[0106] The advantage of timeline semaphore synchronization is that a single semaphore can control the order of all queues, eliminating the need to create multiple pairs of binary semaphores. This method consumes fewer register resources and avoids deadlocks caused by inconsistent queue lengths. Measurements show that in a 1920×1080 pixel, 120Hz refresh rate scenario, the first command buffer raster takes an average of 3.2 milliseconds, and the second command buffer raster takes an average of 2.6 milliseconds. Including transmission and verification time, the total frame latency remains consistently below 7 milliseconds.

[0107] Example 5: In a role-playing game, the character panel area is the gaze area, and the background landscape is the peripheral area. The data packet size for this frame is 11 kilobytes, with verification taking 0.05 milliseconds; the differential segment contains 25 drawing instructions, and the gaze area reconstruction command takes 0.12 milliseconds; the peripheral area instructions are distilled to reduce the number of fragment loop instructions by 45%, with driver-side distillation taking 0.58 milliseconds. The initial value of the timeline semaphore is 128, the first command buffer is written to 129, the second command buffer is written to 130 and waits for 129. The rendering thread completes the swap chain submission 6.9 milliseconds before vertical synchronization triggers, and the panel text and background landscape are presented in the expected order, without tearing or flickering.

[0108] This invention constructs a low-latency, high-bandwidth-utilization, and sequentially controllable cross-platform rendering pipeline through a unidirectional interconnect bus, incremental rendering data packets, read-only resource import, online shader distillation, and timeline semaphore synchronization. This pipeline effectively solves the problem of high complexity in memory data copying and thread scheduling under hybrid architectures, providing a technical foundation for high refresh rate, low-power desktop-Android collaborative rendering.

[0109] Preferably, the unidirectional interconnect bus is a high-speed peripheral interconnect channel containing a field-programmable gate array, wherein the high-speed peripheral interconnect channel only allows Android-compatible environments to write data to the desktop operating system side and blocks reverse read requests.

[0110] This invention employs a unidirectional interconnect bus to achieve write-only data transfer from an Android-compatible environment to a desktop operating system. The hardware consists of a high-speed peripheral interconnect channel, a field-programmable gate array (FPGA), and a dual-ended direct memory access engine. The high-speed peripheral interconnect channel is based on PCIe Gen3×4, with each channel having an 8-bit data bus width and an 8b / 10b encoding scheme, achieving a full-load throughput of approximately 32 Gbit / s. The channel direction is physically defined by soldered wires as "Android-side write, desktop-side read unavailable." Any read transaction on the desktop side will receive an invalid return code, allowing the driver to generate a security alert without blocking the write path.

[0111] The incremental rendering data packet writing process is as follows: The Android-side capture layer writes the packet header length and pointer to the send register; the direct memory access engine moves the data to the channel transmitter in the order of the circular buffer; the field-programmable gate array writes the data to the desktop video memory window according to the address mapping, and triggers an interrupt after the write cycle ends, along with the starting address and the number of bytes.

[0112] Upon receiving an interrupt, the desktop driver recalculates and compares the packet-level hash field. If the hashes match, it verifies the data block by block using the block-level cyclic redundancy check field, with each block being 4096 bytes in size. After successful verification, the driver parses the resource segment, calls `vkImportMemoryFdKHR` to import the file descriptor as a read-only video memory alias, then parses the differential segment, performs an in-situ update on the previous frame's instruction cache using the index, and reconstructs the complete view region commands. The peripheral region commands are restored through run-length decoding, and the fragment shader is replaced with a distilled polynomial kernel version.

[0113] In terms of synchronization, this invention records a first command buffer for commands in the gaze area and a second command buffer for commands in the peripheral area, and uses timeline semaphores to control the execution order: if the semaphore value of the previous frame is... Write when submitting the first command buffer Waiting when submitting the second command buffer And write This ensures that the gaze area layer is rendered first. The prediction window matrix is ​​generated by a second-order Kalman filter and written to a uniform parameter buffer during the recording phase; the two buffers share the same viewport transform.

[0114] Bandwidth and latency analysis: Assume the number of bytes in the data packet is... Effective throughput is Write latency:

[0115]

[0116] In action game test scenarios, the average , ,calculate Including verification, decoding, and command recording, the additional overhead for the entire frame is controlled to 0.7ms; combined with a 120Hz refresh rate, the total frame latency can be guaranteed to not exceed 7ms. Compared with the traditional bidirectional shared video memory solution, this invention improves the bandwidth utilization of the transmission path by approximately 15%, and enhances the security level to irreversible physical isolation, preventing malicious reading and writing of Android video memory on the desktop side.

[0117] Example 6: The 3D modeling software runs at a resolution of 1920×1080. The current frame data packet size is 9KB, with a write latency of 0.0023s; verification takes 0.04ms; gaze area reconstruction takes 0.11ms; peripheral area distillation takes 0.58ms. The total frame time is 6.6ms, maintaining a stable refresh rate of 144Hz. After 24 hours of continuous operation, the bit error rate is... Simulation tests showed no inconsistencies in packet-level hashes, and the system did not trigger a rollback, demonstrating the high reliability of the unidirectional interconnect bus.

[0118] In summary, this invention constructs a secure, low-latency, and high-bandwidth hybrid architecture rendering link through a hardware-write-only unidirectional interconnect channel, dual verification, and timeline synchronization, providing a feasible solution for desktop and Android environments to share the same graphics processor.

[0119] Preferred, such as Figure 2 As shown, the simplified peripheral shader includes: parsing the peripheral command into a first graphic intermediate representation and a second graphic intermediate representation; inputting the first graphic intermediate representation and the second graphic intermediate representation into a graph neural network distillation model to obtain polynomial kernel parameters; and replacing the fragment stage operation in the peripheral shader with the polynomial kernel parameters.

[0120] In a cross-platform rendering pipeline with a hybrid architecture, this invention introduces an "online graph neural network distillation" mechanism to reduce the computational overhead of the peripheral fragment shader. This mechanism simultaneously parses the peripheral commands generated on the Android compatibility environment side into two intermediate graphical representations, and then uses a lightweight graph neural network to infer multinomial kernel parameters on the desktop operating system side. These kernel parameters are then used to dynamically replace the complex calculations in the fragment stage, achieving real-time simplification.

[0121] In traditional desktop rendering pipelines, peripheral fragment shaders often contain loops, branches, and texture-dependent sampling, leading to extensive access to registers and buffers during rasterization. For scenes with rapid scaling or window movement, these complex calculations can easily become frame-time bottlenecks. This invention argues that peripheral areas have lower visual attention, and their color and geometric changes exhibit significant correlations between adjacent frames. Therefore, low-order polynomials can be used to approximate the high-dimensional nonlinear calculations of the original shaders. Consequently, instead of interpreting the original SPIR-V or DXIL instruction-by-instruction on the desktop, their structure is abstracted into a graphical intermediate representation and input into a graph neural network distiller trained offline. The distiller outputs a set of polynomial kernel parameters, which can be directly used in a pre-compiled template fragment shader to replace loop calculations with constants, thereby reducing register pressure and pipeline wait times.

[0122] The actual implementation process includes:

[0123] Step 1: When packaging the peripheral area command on the Android side, no modifications are made to the fragment shader; only the original shader binary hash is recorded in the packet header.

[0124] Step 2: After unpacking on the desktop side, the SPIR-V parser and DXIL decompiler respectively generate a first and a second intermediate graphical representation. The first intermediate graphical representation retains the opcodes, data flow, and control flow edges; the second intermediate graphical representation emphasizes register read / write dependencies and texture sampling pointers.

[0125] Step 3: The two graphs are concatenated into a joint directed acyclic graph (DAG). Node features include instruction type, texture channel count, and constant usage density. This graph is then normalized and input into the graph attention distillation model. The model consists of a four-layer graph attention network encoder and a two-layer multilayer perceptron decoder, with a total weight of less than 200,000 parameters, quantized using eight-bit integers.

[0126] Step 4: Model outputs polynomial kernel parameter vector These parameters are written into the constant buffer of the template fragment shader, and the original complex shader is replaced with the following form:

[0127]

[0128] in For texture coordinates, Each coefficient in the formula represents the weight of the current pixel's output color in relation to the corresponding polynomial term. The stencil shader has a fixed pipeline structure and does not contain loops or branches.

[0129] Step 5: The driver inputs the updated shader along with the reconstructed peripheral area drawing instructions into the second command buffer and submits it to the graphics queue.

[0130] To assess distillation error, this invention calculates a fast mean square error index during the inference phase:

[0131]

[0132] in Indicates the number of sampled pixels (fixed at 64); Indicates the original fragment shader's first... Each sampled pixel outputs a color vector; This represents the pixel color vector after polynomial kernel replacement. The driver sets the threshold. .like If the polynomial kernel is enabled, then the original shader is used; otherwise, the original shader is retained.

[0133] Using this invention, in a racing game test at a 1920×1080 pixel window and 120 frames per second, peripheral area pixels accounted for 80% of the total pixels. After adopting the distillation scheme, the average register usage in the fragment processing stage was reduced by 45%, the execution cycle was reduced by 38%, and the peripheral area rendering time was reduced from 2.6 milliseconds to 1.6 milliseconds. Because the stencil shader eliminates loops, the latency jitter is less than one-third of that of the original scheme. The overall frame latency is reduced by approximately 0.9 milliseconds, freeing up sufficient budget for reversible compensation of the gaze area.

[0134] Example 7: The test scenario is an open-world map containing abundant vegetation and rocks. The original fragment shader for the surrounding areas uses six-fold texture sampling and a cubic exponential function. After parsing, the first image represents 157 nodes in the center, and the second image represents 92 nodes in the center. The combined graph input to the model takes 0.45 milliseconds. The model outputs a fifth-order polynomial kernel coefficient vector. A total of 36 items. Fast Mean Square Error When the value is below the threshold of 0.002, the replacement takes effect. After the stencil shader constant buffer is written, the fragment throughput of related drawing instructions doubles. From the smallest executable package to screen output, the total frame time is 6.7 milliseconds, which is 1.1 milliseconds faster than the undistilled solution, and the difference in screen contrast is only 0.3%.

[0135] By employing dual intermediate representation parsing, graph attention distillation, and polynomial kernel replacement, this invention efficiently compresses the complexity of peripheral fragment shaders on the desktop side. Combined with zero-copy resources, unidirectional bus, and timeline synchronization, it ensures that cross-platform rendering of the hybrid architecture maintains visual consistency while achieving low power consumption, high refresh rate, and secure isolation.

[0136] Preferably, when the first command buffer is committed, the timeline semaphore value is incremented by one. Before the second command buffer is committed, the timeline semaphore value is waited for to reach the incremented value. After the commit is completed, the timeline semaphore value is incremented again.

[0137] This invention employs timeline semaphores on the desktop operating system side to ensure that the gaze area command buffer and the peripheral area command buffer are executed in a fixed order. This mechanism, together with incremental rendering data packets, unidirectional interconnect buses, and zero-copy resource imports, constitutes the timing skeleton of the hybrid architecture rendering link. Its core objective is to prioritize rendering the gaze area layer that the user is interested in during high refresh rate windows, and then render the peripheral area layers, without introducing the blocking and deadlock that may be caused by traditional binary semaphores.

[0138] A timeline semaphore is a monotonically increasing synchronization primitive, where the same value can be referenced by multiple commit operations. Its advantage lies in expressing multi-frame dependencies using a single object, reducing the number of objects and driver overhead. This invention utilizes the "write value greater than wait value" constraint of the timeline semaphore: if the commit order satisfies "semaphore write first, semaphore wait," it ensures that the command buffer is queued in a determined order and avoids deadlock. Specifically, gaze area commands are most sensitive to the user and therefore must be completed first in any frame; peripheral area commands can be completed later without affecting the experience. This invention uses a fixed high-speed queue to commit gaze area command buffers and a regular queue to commit peripheral area command buffers, with both queues sharing the same timeline semaphore object.

[0139] The implementation process includes:

[0140] Phase 1, Semaphore Initialization. The desktop driver creates a timeline semaphore and sets its initial value to 0 when rendering begins, denoted as . .

[0141] Phase Two: Recording Command Buffers. After the new frame data packet is decoded, the driver records the first and second command buffers respectively. The first command buffer contains the gaze area rendering instructions, the viewport matrix, and invertible compensation constants; the second command buffer contains the peripheral area rendering instructions and the polynomial kernel constant generated after distillation by the graph neural network. During recording, the prediction window matrix is ​​written to a unified parameter buffer, so that the two buffers share the same coordinate system.

[0142] Phase 3, Submission Order Control. Set the semaphore value to the value when the previous frame was completed. When the driver submits the first command buffer, it includes a semaphore write operation, incrementing the value to:

[0143]

[0144] Before committing the second command buffer, declare the wait value in the commit message structure. And write the following after successful submission:

[0145]

[0146] Since timeline semaphores only support monotonically increasing, the second command buffer must wait until the first command buffer has been written before it can enter the execution queue.

[0147] Phase Four: Frame End Maintenance. After the graphics processor finishes executing the second command buffer, the hardware automatically updates the semaphore value. The driver reads this value at the end of the rendering thread and writes it back to the global variable for calculation in the next frame. Laying the foundation.

[0148] The core logic of timeline synchronization can be abstracted as follows:

[0149]

[0150]

[0151]

[0152] in This indicates the semaphore value at the end of the previous frame; This indicates the semaphore value written when the first command buffer is committed; This indicates the waiting parameters for the queue containing the second command buffer; This represents the semaphore value written when the second command buffer is submitted. The order of writing before waiting ensures that hardware scheduling does not result in a race condition.

[0153] In a 1920×1080 pixel, 144Hz refresh rate scene, the average rasterization time for the first command buffer is 3.1 milliseconds; the average rasterization time for the second command buffer is 2.4 milliseconds. Because the timeline semaphore avoids the traditional "commit-wait-write-wait again" multi-object cascading, the commit process only calls two system calls, with an average time of approximately 35 microseconds. Compared to the binary semaphore scheme, the number of objects is reduced from two per frame to zero per frame, reducing the pressure on object allocation and garbage collection. Inter-frame jitter decreased from 0.7 milliseconds to 0.3 milliseconds, and the gaze-first drawing constraint was not violated in all test frames.

[0154] Example 8: In a strategy game scenario, the gaze area is the status box of the selected unit around the cursor, occupying 10% of the screen area, while the surrounding area contains the map and animations. The test machine is equipped with a desktop-grade graphics processor. During initialization, the timeline semaphore is set to 0. After the 100th frame, the value is read as 200. In the 101st frame, 201 is written when submitting the first command buffer, and 201 is waited for before submitting the second command buffer, followed by 202. The hardware execution order is: the gaze area status box is drawn first, followed by the map scrolling, resulting in a final rendering time interval of less than 7 milliseconds. In a continuous 5-minute interactive test, the text update in the gaze area remained tear-free. When the same application was moved to a 60Hz display, the frame synchronization logic did not need modification, and the timeline semaphore value still increased by 2 per frame, indicating that the solution has inherent compatibility with different refresh rates.

[0155] By employing a synchronization strategy of writing multiple times once using timeline semaphores, this invention ensures that the gaze area and surrounding areas are executed in a fixed order with the fewest number of synchronization objects in a hybrid architecture environment. At the same time, it compresses scheduling overhead to an extremely low level, leaving ample compensation time for high refresh rate window rendering and avoiding common risks of circular waiting and deadlock.

[0156] Preferably, the prediction window matrix is ​​obtained by performing second-order Kalman filtering on the window position and window size of consecutive frames, and is used as the reference matrix for viewpoint weight calculation and differential data generation in the next frame encoding stage.

[0157] The prediction window matrix is ​​a temporal feedforward quantity in the cross-platform rendering chain of this invention. It aims to know the position and size of the window in the screen coordinate system in the next frame in advance on the desktop operating system side, thus enabling the viewpoint weight calculation and the gaze area-periphery difference operation to use the same reference. This avoids black borders and flickering caused by compensation hysteresis, even when the user is dragging or scaling the window. Its generation method uses a second-order Kalman filter: the state vector simultaneously contains the velocity components of the window center coordinates and the window width and height, allowing for smooth prediction of position and translation rate within a unified model.

[0158] The state vector is defined as:

[0159]

[0160] in The coordinates of the center pixel of the window. For window width and height, This represents the corresponding first derivative. State transition matrix:

[0161]

[0162] Inter-frame time, It is a 4th-order identity matrix. Measurement vector:

[0163]

[0164] The window manager provides this information before vertical synchronization. The filtering process consists of two steps: prediction and correction. The prediction output is what this invention refers to as the "prediction window matrix," denoted as:

[0165]

[0166] The implementation process includes: after the Android side completes the encapsulation of the command packet for this frame, it writes it to the desktop video memory through a unidirectional interconnect bus; while verifying the incremental rendering data packet, the desktop driver obtains the current measurement from the window manager. Perform a correction update and immediately predict using the state transition matrix. This matrix is ​​then written into a uniform parameter buffer, which is used by the capture layer to generate the viewpoint weight field when encoding the next frame: the weight field center and Alignment, Gaussian radius according to The 12% setting is used. Simultaneously, the differential reference array is also cropped according to the prediction size to ensure that the XOR alignment index does not shift. Since the prediction window matrix is ​​shared via hardware buffers, the capture layer does not need to call inter-process communication, reducing synchronization blocking within a frame.

[0167] Through this invention, in a drag test of a 1920×1080 pixel window at a refresh rate of 144 Hz, the second-order Kalman filter controlled the mean square error of the window center prediction to within 4.3 pixels, while the error of the unpredictable version was approximately 18 pixels. This significant reduction in error greatly decreases the amount of texture coordinate compensation, eliminating jitter in the gaze area and surrounding pixels. Frame-level compensation time was reduced from 0.22 milliseconds to 0.07 milliseconds, allowing more rasterization time for the graphics processor. More importantly, when the user rapidly zooms in and out of the window, the prior size provided by the prediction matrix allows the capture layer to pre-arrange the gaze area division on the Android side, avoiding large uncoded areas caused by sudden size changes.

[0168] Example 9: In a spreadsheet application, the user drags the window horizontally by 600 pixels for 1 second. The filter initialization covariance is set with the window stationary. Test records show that the position measurement error per frame is as high as ±12 pixels, but the prediction error is always less than 5 pixels; the deviation between the center of the gaze area weight field and the actual window center is less than 0.3%. Therefore, the gaze area difference size in the Android-side encoding fluctuates by only 5%, while the fluctuation is close to 30% without prediction. The desktop-side compensation shader only needs one bilinear sampling to align pixels, while the control scheme requires three samplings and one Laplacian solution. Ultimately, the overall frame delay is reduced by 0.6 milliseconds, and the user does not experience a "follow-the-drag delay" phenomenon.

[0169] In summary, the prediction window matrix utilizes second-order Kalman filtering to unify the window position, size, and velocity, enabling viewpoint weight calculation, differential index pruning, and raster compensation in the hybrid architecture to all operate within the same coordinate system. This improves stability, reduces computational load, and provides robust support for high refresh rate and multi-window environments.

[0170] After the graphics processor executes the first command buffer and the second command buffer, it obtains the actual window state and compares it with the predicted window matrix. If the deviation exceeds the threshold, it adjusts the texture coordinates of the gaze area and the surrounding area frame buffer in the fragment shader, and then composites them with the desktop layer. When the incremental rendering data packets are continuously missing, it performs frame interpolation. It outputs the composite frame and writes the texture coordinate adjustment information into shared storage for the next frame to call.

[0171] This invention establishes an "actual window comparison and dynamic texture coordinate compensation" process at the end of the frame on the desktop operating system side. This ensures that, under window changes such as dragging, stretching, and system scaling, the pixels in the viewing area and surrounding areas can still accurately conform to the desktop compositor coordinate system, and maintain frame sequence continuity even when incremental rendering data packets are occasionally lost. This process mainly includes four steps: actual window measurement, prediction error determination, fragment-level texture coordinate compensation, and frame backoff; and at the end, the average compensation vector is written to a shared buffer to provide a reference for the next frame encoding stage.

[0172] Because window deformation is unpredictable in user interaction, relying solely on the gaze area weights and difference indices generated from the previous frame's prediction window matrix is ​​insufficient to fully cover the actual geometry of the current frame. Excessive prediction errors can lead to black borders or pixel misalignment. Therefore, this invention, after the graphics processor completes the first command buffer (gaze area) and the second command buffer (peripheral area), immediately calls the window manager interface to obtain the "actual window state," i.e., the current window's center coordinates and width / height. This is compared with the prediction window matrix. If the relative deviation of either the center coordinates or the size exceeds a threshold (default 1%), it indicates insufficient prediction, requiring overall translation or scaling compensation of the frame buffer texture coordinates at the fragment stage. Compensation is achieved through solving the invertible Laplacian-Lagrange potential function, adjusting only the sampling coordinates without re-rasterization. If two consecutive frames lack incremental rendering data packets, indicating a temporary interruption in Android-side rendering, a lightweight super-resolution model is called to fill in the gaze area frame buffer, maintaining temporal integrity.

[0173] Implementation details include:

[0174] 1. Actual window measurement: The desktop driver calls GetClientRect(Win32) or xdg_toplevel_get_geometry(Wayland) before submitting the swap chain to obtain... .

[0175] 2. Error vector calculation: The prediction window matrix is ​​as follows: Error vector Defined as:

[0176]

[0177] like Then the compensation process begins, in which... It is the Euclidean norm.

[0178] 3. Solving for the potential function: Constructing a discrete potential function in the fragment shader. ,satisfy , To normalize the error distribution, a five-order Gauss-Seidel iteration approximation is used. Obtain the gradient. Then update the texture coordinates:

[0179]

[0180] 4. Desktop layer composition: The driver calls the compositor interface and, according to the Porter-Duff source overlay rules, sequentially overlays the updated gaze area frame buffer and the surrounding area frame buffer onto the underlying desktop layer.

[0181] 5. Frame fallback: If two frames are detected as not receiving incremental rendering data packets, the three most recent frames are taken from the gaze area texture cache as residual blocks and input into the ESRGAN-Lite super-resolution model to generate... After replacing the gaze area frame buffer, synthesize according to step 4.

[0182] 6. Feedback Writing: [This will be done / implemented] The average gradient across the entire screen is written to the shared page grad_buf so that the capture layer in the next frame can adjust the baseline when generating the difference.

[0183] Error judgment threshold:

[0184]

[0185] Potential function iteration:

[0186]

[0187] Updated coordinates:

[0188]

[0189] Original texture coordinates; Updated coordinates; Iterative index.

[0190] Using this invention, in a test environment with a 1920×1080 pixel window and 120 frames per second, the window was dragged at a speed of 240 pixels per second. With an error threshold set at 1%, the compensation frequency accounted for approximately 35% of the total frames. The average compensation solution time was 0.08 milliseconds, saving 0.15 milliseconds compared to the unpredicted scenario. The probability of black borders appearing decreased from 3% to 0.2%. In the incremental rendering data packet fault injection experiment, simulating packet loss once every 200 frames, after enabling super-resolution frame interpolation, the frame rate fluctuation remained within ±3 frames per second, and the subjective difference in image quality in the gaze area was difficult to distinguish. Compared to a complete re-rendering and rollback strategy, this invention saved an average of 3.7 milliseconds in faulty frames.

[0191] Example 10: A music game rapidly scales its resolution while dragging the window. A measured prediction error percentage of 1.8% for a certain keyframe triggers compensation. The average gradient of the potential function after five iterations is... After coordinate updating, the pixel shift at the edge of the gaze area is less than 1 pixel, which is invisible to the naked eye. The next frame capture layer is then read. It automatically applies a 0.5-pixel inverse offset to the center of the weight field, and compensation is not triggered in subsequent frames. The entire interaction process is flicker-free, and the frame-time curve remains smooth.

[0192] By comparing the actual window state with the predicted window matrix, this invention dynamically corrects texture coordinates at the fragment stage. Combined with lightweight super-resolution interpolation and cross-frame gradient feedback, it achieves pixel-level adaptive alignment when the window changes rapidly. This ensures visual stability in the gaze area and avoids re-rendering the entire frame, improving the robustness and real-time performance of the hybrid architecture cross-platform rendering link.

[0193] Preferably, when adjusting texture coordinates, the Gaussian Seidel iteration is performed multiple times in the fragment shader to solve the discrete Laplace equation for the gaze area frame buffer and the peripheral area frame buffer, respectively, in order to obtain the texture coordinate adjustment vector and update the fragment sampling position in real time.

[0194] In cross-platform rendering paths with hybrid architectures, window dragging, scaling, or multi-monitor switching can cause deviations between the predicted window matrix and the actual window geometry. Traditional methods involve re-rasterizing the entire frame or rendering local patches, resulting in high latency and pipeline jitter. This invention proposes a "Discrete Laplacian Texture Coordinate Compensation" scheme: after the graphics processor has completed pixel filling of the first command buffer (fogging area) and the second command buffer (peripheral area), the fragment shader performs Gaussian-Seidel iteration to solve the two-dimensional discrete Laplacian equation on the two frame buffers to obtain the texture coordinate adjustment vector. Then, the sampling position is modified in real time to achieve sub-pixel-level alignment. This scheme fully utilizes the massive parallel computing power of the graphics processor, achieving millisecond-level compensation and avoiding black borders or cracks without requiring an additional rasterization stage.

[0195] The gaze area frame buffer and the peripheral area frame buffer can be viewed as a two-dimensional scalar field with boundary conditions. The pixel displacement introduced by the window prediction error can be considered as a set of scattered displacement sources within the field. According to the harmonic mapping theory, the minimum energy deformation function that satisfies the candidate boundary conditions... It can be achieved through the discrete Laplace equation Approaching, among which This represents the normalized displacement source distribution. The solution yields... Then, its gradient becomes the texture coordinate adjustment vector. In practical implementation, the pixel grid is discretized into four-neighbor difference, and the Gauss-Seidel iterative update formula is used:

[0196]

[0197] in For pixel index, The pixel grid spacing is denoted as . Due to the fragment parallelism of the graphics processor, adjacent pixels can be split into a checkerboard pattern and updated alternately in a single rendering pass. After five to seven iterations, the error can be reduced to less than 1 pixel.

[0198] The implementation process includes:

[0199] 1. Error source initialization: The compensation module receives the predicted window matrix and the actual window matrix, and calculates the vertex level difference. These differences are projected onto the edge pixels of the viewing area and surrounding areas to generate a sparse source term texture, which is then written into a 16-bit floating-point map as... .

[0200] 2. Dual-channel iteration: A new fragment shader channel is created in the rendering pipeline, binding the foveation framebuffer as input / output texture A, and the perimeter framebuffer as input / output texture B. A push constant is passed in. Regarding the number of iterations, Gauss-Seidel internally uses "red-black" blocks: even-numbered pixels are written first, and odd-numbered pixels are written last. Each write uses the graphics processor's local shared memory to cache the latest value of the four neighbors.

[0201] 3. Write the vector back and, after completing the specified number of iterations, calculate the gradient for each pixel:

[0202]

[0203] It also instantly offsets the sampling coordinates to obtain the color value, achieving "calculation and use simultaneously".

[0204] 4. Result sharing: Calculate the average gradient over all pixels. The number of iterations is written to the shared storage grad_buf for the next frame capture layer to correct the viewpoint weight center; at the same time, the number of iterations is written to the debug buffer for subsequent dynamic optimization.

[0205] Using this invention, in a game scene with a resolution of 1920×1080 and a refresh rate of 144Hz, when the window is dragged at a speed of 300 pixels per second, the prediction error can reach 12 pixels. After 7 iterations (two steps in parallel red and black), the maximum residual error in the gaze area is reduced to 0.8 pixels, and the residual error in the peripheral area is 1.3 pixels; the compensation fragment shader takes 0.09 milliseconds, accounting for about 1% of the total frame budget. Compared with a complete rasterization, the latency is reduced by 2.8 milliseconds; compared with the nearest neighbor stretching scheme, the image is free of jagged edges and there are no multi-layered ghosting at the edges.

[0206] Example 11 demonstrates a video player displaying a movie clip in full screen while the user continuously zooms in and out of the window. Without compensation, high-contrast subtitles will appear ghosted during zooming. The application uses the compensation process of this invention: the difference between the actual window and the predicted window is 8 pixels; the number of iterations is set to 5; the total number of instructions in the Gaussian Seidel fragment stage is 896, and the register usage is 24.

[0207] Experimental results: The compensation overhead was 0.07 milliseconds, and the subjective clarity of the subtitles remained unaffected; without compensation, the subtitles produced approximately 3 pixels of ghosting stripes during the shrinking phase. In terms of overall power consumption, the compensation scheme saved approximately 8 watts of graphics pipeline power compared to full-frame re-rendering.

[0208] By performing multiple Gaussian Seidel iterations to solve the discrete Laplace equation within the fragment shader, this invention achieves real-time texture coordinate compensation for the viewport and surrounding framebuffer regions. This solution offers the triple advantages of zero additional raster stages, low latency, and high image quality, and can be combined with the prediction window matrix, temporal semaphore synchronization, and unidirectional interconnect bus to form a complete hybrid architecture cross-platform rendering solution.

[0209] Preferably, the frame interpolation process performs deep learning super-resolution model inference with residual blocks as input only on the gaze region frame buffer, and replaces the current frame image in the gaze region frame buffer with the high-resolution image obtained from the inference.

[0210] This invention introduces a "focal region deep learning super-resolution frame interpolation" mechanism into the cross-platform rendering pipeline to address scenarios involving sudden frame drops or transient jitter in the unidirectional interconnect bus within an Android-compatible environment. Compared to traditional full-frame re-rendering, this mechanism performs lightweight inference only on the focal region frame buffer, maintaining the resolution and dynamic consistency of the user-focused area while avoiding a significant increase in the graphics processor load.

[0211] When incremental rendering data packets are continuously detected as missing, the desktop driver determines that the current frame instruction on the Android side is unavailable. Considering that the pixels in the gaze area occupy less than 20% of the screen and that changes in the surrounding area are of low sensitivity to the user, this invention only performs frame interpolation on the gaze area. The interpolation input is not a complete low-resolution image, but is constructed in the form of residual blocks: the driver retains the three most recent gaze area images in video memory. Pixel-wise differencing is performed between each frame, and the image is cropped to one-sixth of the center region to obtain a residual block sequence. Residual blocks contain moving edges and high-frequency details, which can significantly reduce inference computation.

[0212] The model structure uses the lightweight version of ESRGAN-Lite with three residual groups, and a total of approximately 900,000 parameters. The input dimension is... Channel 6 is composed of the red, green, and blue channels of two residual blocks; the output dimension is... The loss function considers both mean squared error and adversarial loss during offline training, while only convolution and pixel convolution upsampling are performed during the inference phase, without conditional branches.

[0213] The implementation process includes:

[0214] 1. Trigger condition judgment: Driver maintenance package missing counter .like If the frame interpolation process is initiated, proceed with the frame interpolation process; otherwise, wait for the next packet as usual.

[0215] 2. Residual block generation: Calculation The subtraction operation involves taking the absolute value of each pixel. Then, the center pixel is cropped. Region and bilinear scaling to .

[0216] 3. Model Inference: The Tensor Core of the graphics processing unit is invoked to execute ESRGAN-Lite, with an average execution time of 0.85 milliseconds. High-resolution images are output. .

[0217] 4. Frame buffer replacement: Write back to the gaze area frame buffer, while the surrounding area frame buffer retains the pixels from the previous frame.

[0218] 5. Gradient feedback: Mean the residual blocks... Recorded to the shared storage grad_buf, so that the capture layer can fine-tune the viewpoint weight radius after the next packet is recovered, avoiding sudden jumps.

[0219] 6. Counter Reset: The counter is reset after a new packet is received and a frame is successfully assembled. Key expression inference output Upsampling via pixel convolution:

[0220]

[0221] ⊕ indicates channel concatenation; the number of convolutional kernels from the previous layer is 64, 32, and 12 respectively; the PixShuffle dimension is multiplied by 2.

[0222] This invention intentionally discards two frames of data packets every 40 frames in a 1920×1080 pixel scene with a refresh rate of 144 Hz. Without frame interpolation, a black box appears in the gaze area for approximately 14 milliseconds. With this invention, the black box disappears completely, and the average peak signal-to-noise ratio between frames is 38 dB, higher than the 18 dB of cubic convolution interpolation. Inference time is 0.85 milliseconds, accounting for no more than 12% of the total frame budget, and the frame rate drop is less than 2 frames per second.

[0223] Example 12, Real-world test in a role-playing game: Frame sequence Missing frame, initiate frame interpolation. Generate residual block with an average brightness of 2.4. High-resolution output. After the replacement, the stuttering was no longer noticeable; when the new packet was restored in the next frame, the peak signal-to-noise ratio compared to the real rendered image was 37.2 dB. In the user subjective survey, none of the seven subjects noticed any abnormality.

[0224] By performing residual block-driven lightweight super-resolution interpolation only on the gaze area, this invention effectively maintains image integrity and sharpness in the case of missing data packets, while controlling computational and bandwidth overhead, providing reliable fault tolerance in hybrid architecture cross-platform rendering links.

[0225] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A cross-platform rendering method under a hybrid architecture, characterized in that, Includes the following steps: Based on the viewpoint coordinates obtained from eye tracking, the current frame command stream in the Android compatible environment is divided into gaze area commands and peripheral area commands. The gaze area commands are compared with the data of the previous frame to generate differential data. The differential data and peripheral area commands are encoded and encapsulated together with the graphics resource mapping table and integrity verification information into an incremental rendering data packet. The incremental rendering data packet is written to the desktop operating system video memory via a unidirectional interconnect bus. After verifying its integrity, the gaze area command is reconstructed and the peripheral area shader is simplified. The first command buffer and the second command buffer are generated. The two buffers and the prediction window matrix are submitted to the graphics processing queue synchronized with the timeline. After the graphics processor completes the execution of the first command buffer and the second command buffer, it obtains the actual window state and compares it with the predicted window matrix. If the deviation exceeds the threshold, the texture coordinates of the gaze area and the surrounding area frame buffer are adjusted in the fragment shader, and then composited with the desktop layer. When the incremental rendering data packets are continuously missing, frame interpolation is performed. Output the composite frame and write the texture coordinate adjustment information to shared storage for use in the next frame.

2. The method according to claim 1, characterized in that, When performing differential operations on the gaze area command, a hash value is calculated for the handle and fixed parameter value of each drawing instruction in the gaze area command, and then XORed with the hash value at the corresponding position in the previous frame to obtain differential data.

3. The method according to claim 2, characterized in that, The obtained differential data is converted into a frequency domain representation and then XORed with the fingerprint value of the previous frame to obtain the updated fingerprint value. The peripheral area command is compressed by the run length and then encapsulated together with the updated fingerprint value into an incremental rendering data packet.

4. The method according to claim 1, characterized in that, The graphics resource map records the file descriptors generated for each texture object through the graphics memory import interface, and sets the file descriptors as read-only aliases.

5. The method according to claim 1, characterized in that, The unidirectional interconnect bus is a high-speed peripheral interconnect channel containing a field-programmable gate array. The high-speed peripheral interconnect channel only allows Android-compatible environments to write data to the desktop operating system side and blocks reverse read requests.

6. The method according to claim 1, characterized in that, The simplified peripheral shader includes: parsing the peripheral command into a first intermediate representation and a second intermediate representation, inputting the first intermediate representation and the second intermediate representation into a graph neural network distillation model to obtain polynomial kernel parameters, and replacing the fragment stage operations in the peripheral shader with the polynomial kernel parameters.

7. The method according to claim 1, characterized in that, When the first command buffer is committed, the timeline semaphore value is incremented by one. Before the second command buffer is committed, the timeline semaphore value is waited for to reach the incremented value. After the commit is completed, the timeline semaphore value is incremented again.

8. The method according to claim 1, characterized in that, The prediction window matrix is ​​obtained by performing second-order Kalman filtering on the window position and window size of consecutive frames, and is used as the reference matrix for viewpoint weight calculation and differential data generation in the next frame encoding stage.

9. The method according to claim 1, characterized in that, When adjusting texture coordinates, the fragment shader performs multiple Gaussian Seidel iterations to solve the discrete Laplace equation for both the gaze area framebuffer and the peripheral area framebuffer to obtain the texture coordinate adjustment vector and update the fragment sampling position in real time.

10. The method according to claim 1, characterized in that, The frame interpolation process performs deep learning super-resolution model inference with residual blocks as input only on the gaze region frame buffer, and replaces the current frame image in the gaze region frame buffer with the high-resolution image obtained from the inference.

Images