GPU encoding method based on rendering synchronization, electronic device, chip, storage medium

Abstract

This application provides a GPU encoding method, electronic device, chip, and storage medium based on rendering synchronization. The method includes: acquiring encoding-related information of multiple virtual machines bound to the same GPU encoding core; determining a frame dropping scheme for each virtual machine when the current encoding demand exceeds the processing capacity of the GPU encoding core based on the encoding-related information; obtaining the target rendering timing for each virtual machine that minimizes the average encoding latency through a collaborative adjustment algorithm based on the encoding-related information and the frame dropping scheme; sending a synchronization instruction containing the target rendering timing and the frame dropping scheme to the virtual machine, triggering the virtual machine to determine whether the current frame is a frame to be dropped after capturing a screen image according to the frame dropping scheme in the synchronization instruction; if so, triggering the virtual machine to adjust the screen image rendering timing of the local machine by modifying the vertical synchronization timing of the virtual display according to the target rendering timing in the synchronization instruction, so that the encoding request timing matches the target rendering timing.

Description

Technical Field

[0001] This application relates to the field of quantum computing technology, and in particular to a graphics processing unit (GPU) encoding method, electronic device, chip, and storage medium based on rendering synchronization. Background Technology

[0002] With the rapid development of cloud desktop technology, CPU virtualization's display drivers and virtual monitors are gradually failing to meet users' demands for smooth cloud desktop performance. Meanwhile, the large amount of video encoding required by cloud desktop transmission protocols further increases the operational burden on cloud desktops. To improve the smoothness of cloud desktop usage and offload CPU computing power, using GPUs for adaptation encoding and rendering has become a key development direction for cloud desktop technology.

[0003] GPU hardware encoding offers significant performance advantages over software encoding. Software encoding is limited by CPU computing power and can typically only meet the desktop video encoding requirements of 1080p 60fps, while GPU hardware encoding is not limited by this. A physical GPU usually contains multiple encoding cores independent of the rendering core. Through virtualization technology, the capabilities of a single encoding core can be distributed to multiple vGPUs on cloud desktops, achieving a reasonable allocation of encoding capabilities.

[0004] In the process of allocating encoding capabilities, encoding synchronization issues are inevitably involved. Currently, the scheduling schemes for GPU encoding cores are mainly divided into time-slice partitioning and preemptive scheduling. Based on the real-time requirements of cloud desktops, preemptive scheduling is usually used to provide encoding capabilities to cloud desktops. However, in existing technologies, most related patents only involve the implementation of GPU virtualization and do not consider the encoding synchronization problem. This leads to encoding conflicts and high encoding latency when multiple cloud desktops share the same GPU encoding core.

[0005] For example, a single chip supports 1080p. 60fps If five encoding cores are allocated to ten 1080p 30fps cloud desktops, and all cloud desktops send encoding requests simultaneously, the encoding cores must encode sequentially. This results in an encoding latency of 0-33ms for the ten cloud desktops, with an average latency of 16ms. Since cloud desktops are used in real-time video scenarios, they cannot pre-buffer encoded frames, and this latency directly impacts the user experience, leading to a decline in performance. Summary of the Invention

[0006] This application provides a GPU encoding method, electronic device, chip, storage medium, and program product based on rendering synchronization. By coordinating the rendering timing of cloud desktop virtual displays and based on an active frame dropping mechanism, it solves the encoding conflict problem when multiple cloud desktops share a GPU encoding core, significantly reduces the average encoding latency, and improves the real-time performance and smoothness of cloud desktops.

[0007] The technical solution of this application embodiment is implemented as follows: This application provides a GPU encoding method based on rendering synchronization, including: Obtain encoding-related information from multiple virtual machines bound to the same GPU encoding core; Based on the encoding-related information, when it is determined that the current encoding requirements exceed the processing capacity of the GPU encoding core, a frame dropping scheme is determined for each virtual machine. Based on the encoding-related information and the frame dropping scheme, the target rendering timing for each virtual machine that minimizes the average encoding latency is obtained through a collaborative adjustment algorithm. The synchronization command containing the target rendering timing and the frame dropping scheme is sent to the virtual machine. The virtual machine is triggered to determine whether the current frame is a frame to be dropped after capturing the screen image according to the frame dropping scheme in the synchronization command. If so, the virtual machine is triggered to adjust the screen image rendering timing of the local machine by modifying the vertical synchronization timing of the virtual display according to the target rendering timing in the synchronization command, so that the actual encoding request timing matches the target rendering timing.

[0008] As one implementation method, adjusting the screen image rendering timing of the local machine by modifying the vertical synchronization timing of the virtual display includes: The virtual synchronization VSYNC signal of the virtual display is intercepted by the filter driver; Based on the target rendering timing, a software timer is dynamically set to control the triggering sequence of VSYNC, so as to delay or advance the time for obtaining rendering data from the display cache, thereby adjusting the timing of screen capture and encoding.

[0009] As one implementation method, dynamically setting a software timer to control the triggering sequence of VSYNC based on the target rendering timing includes: Create a filter driver that dynamically controls the VSYNC timing. The filter driver hooks the VSYNC signal of the virtual display, controls the image synthesis frequency in the desktop, and dynamically adjusts the VSYNC timing. The DpiInitialize entry function, which interacts with the Windows display driver model and the graphics kernel subsystem in the HOOK operating system, is replaced with a timer function that is dynamically controlled by a software timer to trigger VSYNC. Furthermore, the filter driver provides a timing control interface for the cloud desktop to actively call in order to adjust the VSYNC timing, and determines the VSYNC triggering time as the rendering time of the cloud desktop.

[0010] As one implementation, the method further includes: Adjust the virtual display driver and add an information query interface and a control interface. The information query interface is called by the host process to obtain information such as the number of virtual screens, virtual screen resolution, virtual screen refresh rate, and rendering timing. The control interface is based on the VSYNC timing of the filter driver. By delaying the time when the virtual display obtains rendering data from the display cache, the rendering timing of the virtual display is adjusted, thereby indirectly adjusting the screen capture timing of the cloud desktop. Obtain the virtual display resolution, virtual display refresh rate, protocol screen capture frame rate, binding relationship with the GPU encoding core, and the upper limit of the GPU encoding core capabilities for each cloud desktop. Based on the timing of the cloud desktop calling the encoding application programming interface (API), obtain the encoding timing and encoding interval for each cloud desktop.

[0011] As one implementation method, determining the frame dropping scheme for each virtual machine includes: Input N information for all cloud desktops bound to the same GPU encoding core i , i∈{1,2,...,n}; Filter out T i F i The cloud desktop with the largest value i, where T i F is the encoding time for a single frame of this cloud desktop. i D is the encoding frame rate of this cloud desktop. i The timing for encoding cloud desktop i to be first acquired within the sampling period of the encoding collaboration process; Discard one frame of encoded data for cloud desktop i in sequence, and update the corresponding N for that cloud desktop. i China F i D i E i Three parameters, and calculate the updated E i The variance; where E i This is an array of encoding intervals; Repeat the above process until the value of E after discarding each frame of the cloud desktop is calculated. i The variance is calculated, and the frame with the smallest variance is selected for discarding.

[0012] As one implementation, the method further includes: Input all cloud desktop information updated after frame loss N i ; Create a set S to include all cloud desktops, and simultaneously create initial information N0=(0,0,0,{0}), where the parameters of N0 are defined the same as those of N... i Consistent; Select the encoding frame rate F from set S. i The smallest cloud desktop i is removed from set S.

[0013] As one implementation, the method further includes: Calculate the value of t that minimizes the average coding delay F(t), where F(t) = F(T0, D0, E0, T). i D i +t,E i F(·) is the average coding delay calculation function, T0, D0, and E0 are the single-frame coding time, first coding timing, and coding interval arrays in N0, respectively, and t takes the value t=E i (k) E0(j)+D i D0+T i T0, where k∈{1,2,...,sizeof(E i )}, j∈{1,2,...,sizeof(E0)}, sizeof(E i ), sizeof(E0) and sizeof(E0) are the encoding interval arrays E i The number of elements in E0; The first coding timing of updating cloud desktop i D i =D i +t, connect N0 and N i Merge to obtain the new N0 = f(N0, N i ), where f(·) is the encoding information merging function, used to integrate the encoding parameters of the two cloud desktops; The host will update the N calculated for each cloud desktop. i The information is sent to the corresponding cloud desktop, which is based on N. i D in i The rendering timing is updated through the aforementioned timing control interface; based on N i E in i Active frame dropping is performed. When capturing the screen, it is determined whether the current frame is a dropped frame. If it is a dropped frame, no encoding operation is performed.

[0014] The electronic device provided in this application includes a processor and a memory. The memory is used to store computer programs, and the processor is used to call and run the computer programs stored in the memory to execute the above-described GPU encoding method based on rendering synchronization.

[0015] The chip provided in this application includes a processor for calling and running a computer program from a memory, causing a device equipped with the chip to execute the above-described GPU encoding method based on rendering synchronization.

[0016] The computer-readable storage medium provided in this application embodiment is used to store a computer program that causes a computer to execute the above-described GPU encoding method based on rendering synchronization.

[0017] The computer program product provided in this application includes computer program instructions that cause a computer to execute the above-described GPU encoding method based on rendering synchronization.

[0018] The embodiments of this application have the following beneficial effects: By controlling the VSYNC timing and adjusting the rendering timing through a filter driver, combined with an active frame dropping mechanism and a coordination algorithm, the encoding requests from multiple cloud desktops are staggered to reach the GPU encoding core, completely solving the encoding conflict problem in the preemptive scheduling mode and avoiding stuttering caused by passive frame dropping. The active frame dropping mechanism ensures controllable load on the encoding core, and the coordination algorithm optimizes rendering and encoding timing, enabling synchronous encoding of multiple cloud desktops. The average encoding latency can be significantly reduced from the existing 16ms, improving the real-time operating experience of the cloud desktop. The filter driver is applicable to most Windows systems, and the modified virtual display driver solution is compatible with existing GPU virtualization technology without requiring GPU hardware modifications, reducing implementation costs. Simultaneously, the active frame dropping mechanism and coordination algorithm can be dynamically adjusted according to the encoding core capacity and the number of cloud desktops, adapting to GPU cloud desktop clusters of different sizes. The active frame dropping mechanism selects the optimal frame dropping position by calculating the encoding interval variance, ensuring the uniformity of the encoding interval after frame dropping; the coordination algorithm ensures the stability of the overall encoding process by synchronizing the encoding timing of each cloud desktop, improving the smoothness of cloud desktop usage. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the GPU encoding method based on rendering synchronization provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the implementation process of the GPU encoding method based on rendering synchronization provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the adjustment of the desktop screen capture timing according to an embodiment of this application; Figure 4These are schematic diagrams of multiple cloud desktops according to embodiments of this application; Figure 5 This is a schematic diagram of screen capture when supporting two-way screen capture in an embodiment of this application; Figure 6 This application embodiment is based on cloud computer information. Schematic diagram of the obtained encoding timing image; Figure 7 This is a schematic diagram illustrating the adjustment of the rendering timing for cloud desktops 1 and 2 according to an embodiment of this application; Figure 8 This is a schematic diagram of the composition structure of a GPU encoding device based on rendering synchronization according to an embodiment of this application; Figure 9 This is a schematic structural diagram of an electronic device provided in an embodiment of this application; Figure 10 This is a schematic structural diagram of the chip according to an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0021] Figure 1 This is a flowchart illustrating the GPU encoding method based on rendering synchronization provided in an embodiment of this application, as shown below. Figure 1 As shown, the GPU encoding method based on rendering synchronization in this application embodiment specifically includes the following processing steps: Step 101: Obtain encoding-related information from multiple virtual machines bound to the same GPU encoding core.

[0022] In this embodiment, the virtual display driver is adjusted, and an information query interface and a control interface are added. The information query interface is called by the host process to obtain information such as the number of virtual screens, virtual screen resolution, virtual screen refresh rate, and rendering timing. The control interface, based on the VSYNC timing of the filter driver, adjusts the rendering timing of the virtual display by delaying the time when the virtual display obtains rendering data from the display cache, thereby indirectly adjusting the screen capture timing of the cloud desktop. The virtual display resolution, virtual display refresh rate, protocol screen capture frame rate, binding relationship with the GPU encoding core, and upper limit of the GPU encoding core capability of each cloud desktop are obtained. Based on the timing of the cloud desktop calling the encoding application programming interface (API), the encoding timing and encoding interval corresponding to each cloud desktop are obtained.

[0023] Specifically, on the physical host, a separate encoding collaboration process is launched for each GPU encoding core, or all encoding cores are managed within a single process. This process obtains configuration information for all running cloud desktops (virtual machines) through the virtualization platform's management interface (such as libvirt, XenAPI, etc.). For example, the following information is collected for each cloud desktop: virtual display resolution, virtual display refresh rate, protocol capture frame rate, binding relationship with the encoding core, and the upper limit of the encoding core's capabilities. Simultaneously, by monitoring when each cloud desktop calls the encoding application programming interface (API), the initial encoding timing and encoding interval array for each cloud desktop are obtained. The number of pixels is calculated based on the resolution, and the single-frame encoding time is calculated in conjunction with the encoding core's capabilities. Finally, the protocol encoding frame rate, single-frame encoding time, initial encoding timing, and encoding interval array are used as the encoding information for that cloud desktop.

[0024] Step 102: Based on encoding-related information, determine a frame dropping scheme for each virtual machine when it is determined that the current encoding requirements exceed the processing capacity of the GPU encoding core.

[0025] In this embodiment, the input is N, which contains information about all cloud desktops bound to the same GPU encoding core. i , i∈{1,2,...,n}; Filter out T i F i The cloud desktop with the largest value i, where T i F is the encoding time for a single frame of this cloud desktop. i D is the encoding frame rate of this cloud desktop. i The timing for encoding cloud desktop i to be first acquired within the sampling period of the encoding collaboration process; Discard one frame of encoded data for cloud desktop i in sequence, and update the corresponding N for that cloud desktop. i China F i D i E i Three parameters, and calculate the updated E i The variance; where E i The encoding interval array is used; the above process is repeated until the value of E after discarding each frame of the cloud desktop is calculated. i The variance is calculated, and the frame with the smallest variance is selected for discarding.

[0026] Step 103: Based on encoding-related information and frame dropping scheme, obtain the target rendering timing for each virtual machine that minimizes the average encoding latency through a collaborative adjustment algorithm.

[0027] In this embodiment, the core function of the filter driver is to hook the virtual synchronization signal (VSYNC) of the virtual display. By actively controlling the image compositing frequency within the desktop and dynamically adjusting the VSYNC timing, the rendering timing of the cloud desktop is controlled. The VSYNC signal is the key signal for triggering image compositing by the virtual display. Each time the VSYNC signal is triggered, it drives the graphics card to compose the desktop image required by the cloud desktop. Therefore, controlling the VSYNC timing can indirectly control the rendering timing of the cloud desktop.

[0028] The system intercepts the virtual synchronization signal VSYNC from the virtual display using a filter driver. Based on the target rendering timing, a software timer is dynamically set to control the VSYNC trigger sequence, adjusting the timing of screen capture and encoding by delaying or advancing the retrieval of rendering data from the display cache. Specifically, a filter driver is created to dynamically control the VSYNC timing. This filter driver hooks the VSYNC signal of the virtual display, controls the image compositing frequency within the desktop, and dynamically adjusts the VSYNC timing. It also hooks the DpiInitialize entry function, which interacts with the Windows display driver model and the graphics kernel subsystem in the operating system, replacing it with a timer function dynamically controlled by the software timer to trigger VSYNC. Furthermore, the filter driver provides a timing control interface for the cloud desktop to actively call to adjust the VSYNC timing, and determines the VSYNC trigger timing as the cloud desktop's rendering timing.

[0029] The virtual display driver in the cloud desktop was modified to add a VSYNC timing modification function. This function mainly includes two core interfaces: an information query interface and a control interface. These two interfaces work together to achieve a closed loop of querying and adjustment. The information query interface is called by the coding collaboration process running on the host side. When the coding collaboration process sends a query request, the virtual display driver extracts and returns core information from its stored configuration information, such as the number of virtual screens, virtual screen resolution, virtual screen refresh rate, and rendering timing (i.e., the VSYNC trigger timing determined in step 1), providing data support for subsequent coding collaboration calculations. The control interface, based on the VSYNC timing of the filter driver, adjusts the rendering timing. By intentionally delaying the time it takes for the virtual display to retrieve rendering data from the display cache by one step, the rendering timing of the virtual display is shifted by a preset amount. Since the cloud desktop screen capture operation must wait for the screen rendering to complete, the shift in rendering timing will indirectly lead to the shift in screen capture timing, ultimately achieving a precise correspondence between the screen capture timing and the adjusted rendering timing, laying the foundation for avoiding coding conflicts.

[0030] The rendering frame interval of a virtual monitor is determined by its fixed frame rate. For example, a monitor with a 1fps refresh rate renders at fixed times such as 1s, 2s, and 3s. By delaying the time it takes for the virtual monitor to acquire rendering data (e.g., delaying by 0.5s), the original rendering times of 3s and 4s can be adjusted to 3.5s and 4.5s, thereby causing the screen capture timing to shift synchronously and indirectly adjusting the encoding timing.

[0031] Taking Windows systems, such as Windows 10, as an example, during the filter driver initialization phase, the DpiInitialize entry function, which interacts with WDDM (Windows Display Driver Model) and DXGKRNL (Display Kernel), is hooked programmatically. This native entry function is then replaced with a timing function that dynamically controls VSYNC using a software timer. This breaks the fixed VSYNC trigger cycle of the virtual display, enabling dynamic adjustment of the VSYNC timing.

[0032] Step 104: Send the synchronization instruction containing the target rendering timing and frame dropping scheme to the virtual machine. Trigger the virtual machine to determine whether the current frame is a frame to be dropped after capturing the screen image according to the frame dropping scheme in the synchronization instruction. If so, trigger the virtual machine to adjust the screen image rendering timing of the local machine by modifying the vertical synchronization timing of the virtual display according to the target rendering timing in the synchronization instruction, so that the actual encoding request timing matches the target rendering timing.

[0033] In this embodiment, the input is N, which contains information about all cloud desktops bound to the same GPU encoding core. i , i∈{1,2,...,n}; select T i F i The cloud desktop with the largest value i, where T i F is the encoding time for a single frame of this cloud desktop. i D is the encoding frame rate of this cloud desktop. i The timing for encoding cloud desktop i to be first acquired and encoded within the sampling period of the encoding collaboration process; discarding one frame of encoded data for cloud desktop i in sequence, and updating the N corresponding to cloud desktop. i China F i D i E i Three parameters, and calculate the updated E i The variance; where E i The encoding interval array is used; the above process is repeated until the value of E after discarding each frame of the cloud desktop is calculated. i The variance is calculated, and the frame with the smallest variance is selected for discarding.

[0034] Input all cloud desktop information updated after frame loss N iCreate a set S, and include all cloud desktops in set S. Simultaneously, create initial information N0 = (0,0,0,{0}), where the parameter definition of N0 is the same as that of N... i Consistent; Select the encoding frame rate F from set S. i The smallest cloud desktop i is removed from set S.

[0035] Calculate the value of t that minimizes the average coding delay F(t), where F(t) = F(T0, D0, E0, T). i D i +t,E i F(·) is the average coding delay calculation function, T0, D0, and E0 are the single-frame coding time, first coding timing, and coding interval arrays in N0, respectively, and t takes the value t=E i (k) E0(j)+D i D0+T i T0, where k∈{1,2,...,sizeof(E i )}, j∈{1,2,...,sizeof(E0)}, sizeof(E i ), sizeof(E0) and sizeof(E0) are the encoding interval arrays E i The number of elements in E0; The first coding timing of updating cloud desktop i D i =D i +t, connect N0 and N i Merge to obtain the new N0 = f(N0, N i ), where f(·) is the encoding information merging function, used to integrate the encoding parameters of the two cloud desktops; The host will update the N calculated for each cloud desktop. i The information is sent to the corresponding cloud desktop, which is based on N. i D in i The rendering timing is updated through the timing control interface; based on N i E in i Active frame dropping is performed. When capturing the screen, it is determined whether the current frame is a dropped frame. If it is a dropped frame, no encoding operation is performed.

[0036] Update all cloud desktop information N i (i∈{1,2,...,n}) are used as input to ensure that the calculation is based on the adjusted computing load state.

[0037] Create a set S to include all cloud desktops bound to the current encoding core for subsequent path-by-path processing. Simultaneously, create initial information N0 = (0,0,0,{0}), which serves as a temporary storage unit to store the synchronized encoding information; initially, it contains no actual encoding data. Select the encoding frame rate F from set S. i Remove the smallest cloud desktop i from set S—selecting the cloud desktop with the lowest encoding frame rate as the priority coordination object can reduce the complexity of synchronization calculation and ensure the stability of the synchronization scheme (the lower the frame rate, the larger the encoding interval, and the more adjustment space).

[0038] The core processing method for calculating the rendering timing adjustment t that minimizes average encoding latency is as follows: Through the function F(T0,D0,E0,T) i D i +t,E i The function calculates the average encoding latency. Its core logic is based on the merged encoding information N0 (parameters T0, D0, E0) and the adjusted encoding information of the current cloud desktop i (parameter T). i D i +t、E i The encoding timing of both components is simulated, and the average encoding delay when they work together is calculated. The value of t must satisfy t=E. i (k) E0(j)+D i D0+T i T0, where k∈{1,2,...,sizeof(E i )}, j∈{1,2,...,sizeof(E0)}, sizeof(E i ), sizeof(E0) and sizeof(E0) are the encoding interval arrays E i The number of elements in E0; the purpose of setting this range is to ensure that the encoding interval of the cloud desktop i after adjustment is as close as possible to the encoding interval of the merged information N0, so as to avoid new encoding conflicts caused by excessive adjustment.

[0039] Iterate through all possible values ​​of t, calculate the average encoding latency for each t value, and select the t value with the smallest average encoding latency as the final rendering timing adjustment. Based on the calculated adjustment t, update the D of cloud desktop i. i (i.e. D) i =D i +t), which makes the rendering timing of cloud desktop i (with D) i (Binding) Synchronous adjustment; then through the function f(N0,N iThe function merges the encoding information of cloud desktop i with N0 and updates N0 to the merged encoding information. The core logic of this function is to integrate the encoding timing, encoding interval, and single-frame encoding time parameters of the two encoding information streams to form a unified encoding timing model for subsequent synchronization calculations of other cloud desktops. It checks if set S is empty. If not empty, it means that some cloud desktops have not yet completed synchronization coordination, and returns to step 5.3 to continue processing the next cloud desktop; if empty, it means that all cloud desktops have completed synchronization coordination, and outputs the final cloud desktop synchronization scheme. This scheme includes the adjusted rendering timing of each cloud desktop (updated D). i ) and active frame dropping parameters (updated F) i E i ).

[0040] The host-side coding collaboration process will ultimately synchronize the solution (including the updated N for each cloud desktop i). i The information is sent to the corresponding cloud desktop i, ensuring that each cloud desktop obtains its own rendering timing adjustment parameters and active frame dropping parameters. After receiving the synchronization scheme, cloud desktop i, based on the adjusted D... i The parameters, through the timing control interface provided by the filter driver in step 1, send VSYNC timing adjustment commands to adjust their own VSYNC trigger timing, thereby achieving synchronous updates of the rendering timing—the update of the rendering timing depends entirely on the timing control interface in step 1, ensuring adjustment accuracy and avoiding timing deviations. Cloud Desktop i is based on the adjusted E... i Parameters are used for frame determination during the screen capture phase: when the current frame is captured, it is compared with E. i The parameters determine whether the frame is a preset discard frame; if it is a discard frame, no encoding operation is performed and the frame is skipped directly; if it is not a discard frame, the frame is sent to the GPU encoding core for encoding. Active frame dropping is only performed during the screen capture stage and does not affect the normal rendering process of the cloud desktop. It only filters out the encoded frames that need to be discarded to ensure the smoothness of the encoding process.

[0041] Assume there are three cloud desktops, originally N1=(5,0.1,0,{0.2,0.2,0.2,0.2,0.2}), N2=(5,0.1,0.1,{0.2,...}), and N3=(2,0.1,0,{0.4,0.6}). The total requirement is 5×0.1+5×0.1+2×0.1=1.2>H (assuming H=1.0). Using the above algorithm, it's possible to discard one frame each from N1 and N2, resulting in N1'=(4,0.1,0,{0.2,0.2,0.2,0.4}), N2'=(4,0.1,0.1,{0.2,0.2,0.2,0.4}), with N3 remaining unchanged. The total requirement then becomes 0.4+0.4+0.2=1.0, which satisfies the requirement.

[0042] The following specific examples further illustrate the essence of the technical solutions in the embodiments of this application.

[0043] Figure 2 This is a schematic diagram illustrating the implementation flow of the GPU encoding method based on rendering synchronization provided in the embodiments of this application, as shown below. Figure 2 As shown in the embodiments of this application, the GPU encoding method based on rendering synchronization mainly includes the following steps: creating a filter driver that can hook the VSYNC signal of the virtual display, actively control the image synthesis frequency within the desktop, and dynamically control the VSYNC timing; adding a VSYNC timing modification function, which can reset the VSYNC timing of the virtual display and provide other additional information query interfaces; implementing an encoding collaboration process running on the host, which can obtain the virtual display resolution, virtual display refresh rate, cloud desktop protocol settings, GPU encoding core binding relationship, and GPU encoding core capabilities; setting an active frame dropping mechanism to ensure that the encoding core does not exceed the expected encoding capabilities; calculating the optimal cloud desktop encoding coordination scheme based on the encoding requirements, GPU binding relationship, and GPU capabilities of each virtual machine, and synchronizing the rendering timing of each cloud desktop accordingly; and updating the encoding process of each cloud desktop based on the calculated frame dropping and rendering timing points.

[0044] Taking Windows systems as an example, each time VSYNC is triggered, it drives the graphics card to synthesize an image, which is the desktop image that the cloud desktop needs to obtain.

[0045] Within a Windows 10 virtualized desktop, active control of VSYNC can be achieved by writing a filter driver that actively hijacks the interaction between WDDM and DXGKRNL. Specifically, during the driver initialization phase, the `HOOKDpiInitialize` entry function is replaced with a timer function that dynamically controls VSYNC using a software timer.

[0046] In addition, a timing control interface needs to be added to the filter driver to allow the cloud desktop to actively adjust the VSYNC timing. Within the cloud desktop, the VSYNC trigger timing can be easily determined as the rendering timing.

[0047] Modify the virtual display driver in the cloud desktop to provide information query and control interfaces. The information query interface can be called by the host process to query information including the number of virtual screens, virtual screen resolution, virtual screen refresh rate, rendering timing (VSYNC trigger timing), etc.

[0048] The control interface adjusts the rendering timing based on the filter-driven VSYNC timing. Its core function is to adjust the rendering timing of the virtual display and the screen capture timing of the cloud desktop based on adjusting the rendering timing of the virtual display.

[0049] Monitors are limited by a fixed frame rate, and the rendering frame interval is also fixed. Assuming a monitor with a 1fps refresh rate completes its first frame in 1 second, its rendering time will remain at 2 seconds, 3 seconds, etc. Since cloud desktop screen capture needs to wait for the image to finish rendering, the timing of cloud desktop screen capture perfectly matches this rendering time.

[0050] The core of the rendering timing adjustment function is, based on the virtual display driver, to delay the time it takes for the virtual monitor to retrieve rendering data from the display cache, thereby altering the rendering timing to a certain extent. For example... Figure 1 In the process, the virtual display refresh time for the third frame was adjusted by 0.5 seconds, causing the rendering times for frames 3 and 4 to change from 3 seconds and 4 seconds to 3.5 seconds and 4.5 seconds, respectively. This indirectly adjusts the timing of screen capture and encoding. Figure 3 As shown.

[0051] A typical GPU has multiple encoding cores, and the capabilities of each core are distributed to multiple cloud desktops. Therefore, each solution calculation is performed using a single encoding core and a bound cloud desktop as the unit of solution calculation.

[0052] Develop a coding collaboration process running on the host to obtain necessary information, including the virtual display resolution, virtual display refresh rate, protocol screen capture frame rate, binding relationship with the coding core, and the upper limit of the coding core's capabilities for each cloud desktop. Furthermore, this process can determine the coding timing and interval for each cloud desktop i based on the timing of its coding API calls.

[0053] First, let's look at the n-way cloud desktop. Enter the definition, where It is based on the resolution and the number of pixels calculated. It is the encoding frame rate of the cloud desktop protocol; H is the single-frame encoding time, which can be calculated using formula (2); H is the encoding capability corresponding to the encoding core, and the unit is (pixels). frame); It is the moment when the encoding cloud desktop i is first obtained for encoding within the sampling period of the encoding collaboration process (usually one second, because all screen refresh rates are based on seconds); It is the encoding interval array.

[0054] (1) (2) like Figure 4 As shown, cloud desktops 1, 2, and 3 correspond to... , , ; The vGPU in a GPU cloud PC is derived from hardware GPU virtualization, and the encoding capabilities used by the transmission protocol are also provided based on GPU virtualization. The GPU's encoding core, similar to a CPU, can allocate encoding capabilities to the vGPU based on preemptive and time-slice scheduling modes. Clearly, when a cloud desktop calls upon the vGPU for encoding, cloud desktops on the same GPU encoding core may experience encoding conflicts. For preemptive encoding scheduling, there is a significant conflict regarding the existence of preemptible encoding threads, while for time-slice scheduling, due to the rigid time scheduling mechanism, encoding latency is consistently present.

[0055] like Figure 5 As shown, assuming a single encoding core can only handle two screen captures, and the time for capturing and processing is zero during program execution (meaning the captured image can be encoded immediately after rendering), if cloud desktop A and cloud desktop B render at the same time, cloud desktop B will obviously experience an additional 0.5s encoding delay. If the rendering time of cloud desktop B is delayed by 0.5s, then cloud desktop A and cloud desktop B become completely synchronously encoded, with both having zero encoding delay. In real-world cloud desktop scenarios, a single core is bound to more than two machines; therefore, when encoding conflicts occur, the overall encoding delay is often high, impacting the user experience.

[0056] In real-world cloud desktop scenarios, the following factors also influence coding collaboration: actual coding time, primarily affected by the resolution of the cloud desktop image, but also including factors such as the type of encoded frames and scene complexity. During algorithm execution, this computation is mainly based on the desktop image resolution; The actual encoding interval is related to the cloud desktop's screen capture frame rate and the virtual desktop's refresh rate. In other words, the encoding interval is calculated by sampling the virtual desktop's refresh rate based on the cloud desktop's frame rate. For example, if the virtual desktop is at 60fps and the cloud desktop is set to 30fps, the actual encoding interval is sampling every two frames, which is 33ms. However, if the cloud desktop is set to 24fps, the actual encoding interval is a sampling cycle of two or three frames, resulting in a cycle of 33ms and 50ms. During actual algorithm execution, the encoding interval can be directly obtained by the encoding core, as all encoding requests are sent directly to the encoding core.

[0057] From image rendering to encoding, there is a screen capture and processing interval, which may vary depending on the cloud desktop. Therefore, the algorithm calculates based on encoding time rather than rendering time, thus eliminating errors in this part of the time.

[0058] If the encoding requirements of multiple cloud desktops exceed the upper limit of the core encoding capacity, a certain frame dropping mechanism needs to be implemented.

[0059] Figure 6 Based on cloud computer information The obtained encoded time sequence image, such as Figure 6 As shown in the diagram, the total encoding capacity required by the three cloud desktops in this encoding timing diagram exceeds the core encoding capacity. This will lead to severe stuttering and passive frame dropping in the cloud desktops, resulting in significant latency. Passive frame dropping refers to the situation where, when the cloud computer's encoding thread inputs the current frame for encoding, the previous frame is discarded before encoding begins. For example, in the passive frame dropping mechanism for cloud computers 1 and 2, frames 14 and 23 will be discarded due to conflicts with frames 15 and 24. The average encoding latency throughout the entire encoding cycle is 0.6s / 10.

[0060] To address this, the frame dropping mechanism in this application ensures that the frame intervals are appropriately balanced during frame dropping. The process is as follows: Input the core of all cloud desktop information. ; choose The cloud desktop with the highest value; Discard one frame in sequence and update. middle Three parameters, calculated The variance of each frame is calculated; this process is repeated until the variance of each discarded frame is obtained. Variance, and select the frame with the smallest variance to discard; Calculate and judge If true, continue selecting. Select the cloud desktop with the largest value (i), otherwise exit. Based on the above process, it is possible to... Figure 5 The information has been updated as follows:

[0061]

[0062] Compared to passive frame dropping, the active frame dropping in this embodiment can maintain the reasonableness of the encoding interval after frame dropping as much as possible. Furthermore, the encoded information after frame dropping can be used as input for calculating the synchronization scheme.

[0063] Clearly, simply dropping frames can only guarantee the capacity requirements of the encoding core, and may not necessarily reduce the average encoding latency.

[0064] However, by adjusting the rendering timing, the overall encoding latency can be reduced. As shown in the image below, based on the active frame dropping algorithm, frames 14 and 23 are still selected for dropping, but the rendering timing for cloud desktops 1 and 2 is adjusted. The resulting average encoding latency is 0.1s / 10. Figure 7 As shown.

[0065] Therefore, the active frame dropping combined with a certain rendering delay method in the embodiments of this application can effectively reduce the average encoding delay.

[0066] Based on the above theory, the rendering timing adjustment mechanism for reducing encoding latency in this application embodiment is as follows: Input the core of all cloud desktop information. ; Perform proactive frame dropping and update cloud desktop information. ; Create a collection S and put it into all cloud desktops; create ; Select the cloud desktop i with the smallest encoded frame rate value from set S, and remove i from set S; calculate The obtained value of t when the average coding delay is minimized, where t takes the value of ;

[0067] The range of values ​​for t is designed to ensure that the encoding intervals of the two pieces of information are as close as possible.

[0068] renew and merge Determine if set S is empty; if it is empty, return to process 4.

[0069] in, It is to calculate the average coding delay, This involves merging the encoded information from both inputs.

[0070] After calculating the encoding scheme corresponding to the minimum average encoding delay, the host will send the calculated encoding information. Send it to the corresponding cloud desktop i, and the cloud desktop will be based on Update rendering timing; based on Implement an active frame dropping mechanism.

[0071] The timing of the rendering update is based on the interface provided in step one; the active frame dropping mechanism only needs to determine whether the current frame is a dropped frame when the screen is captured, and if it is a dropped frame, no encoding is performed.

[0072] The embodiments of this application can be applied to various GPU virtualization cloud desktop products, such as public cloud desktop services, private cloud virtual desktop infrastructure (VDI), and cloud gaming in edge computing scenarios. They can effectively improve encoding performance in high-concurrency scenarios, enhance user experience, and have high commercial value.

[0073] Figure 8 This is a schematic diagram of the composition structure of the GPU encoding device based on rendering synchronization provided in the embodiments of this application, as shown below. Figure 8 As shown, the GPU encoding device based on rendering synchronization in this application embodiment includes: The acquisition unit 80 is used to acquire encoding-related information of multiple virtual machines bound to the same graphics processor GPU encoding core; The determining unit 81 is used to determine a frame dropping scheme for each virtual machine when it is determined that the current encoding requirement exceeds the processing capacity of the GPU encoding core based on the encoding-related information. The delay unit 82 is used to obtain the target rendering timing of each virtual machine that minimizes the average encoding delay by using a collaborative adjustment algorithm based on the encoding-related information and the frame dropping scheme. Triggering unit 83 is used to send a synchronization instruction containing the target rendering timing and the frame dropping scheme to the virtual machine, triggering the virtual machine to determine whether the current frame is a frame to be dropped after capturing the screen image according to the frame dropping scheme in the synchronization instruction. If so, the virtual machine is triggered to adjust the screen image rendering timing of the local machine by modifying the vertical synchronization timing of the virtual display according to the target rendering timing in the synchronization instruction, so that the actual encoding request timing matches the target rendering timing.

[0074] One implementation involves modifying the vertical synchronization timing of the virtual display to adjust the local screen image rendering timing. Specifically, this involves: intercepting the virtual synchronization VSYNC signal of the virtual display through a filter driver; dynamically setting a software timer to control the VSYNC trigger timing based on the target rendering timing, thereby delaying or advancing the time for retrieving rendering data from the display cache and adjusting the timing of screen capture and encoding. The dynamic setting of the software timer to control the VSYNC trigger timing based on the target rendering timing includes: creating a filter driver that dynamically controls the VSYNC timing; hooking the VSYNC signal of the virtual display to control the image compositing frequency within the desktop and dynamically adjusting the VSYNC timing; hooking the DpiInitialize entry function, which interacts with the Windows display driver model and the graphics kernel subsystem in the operating system, and replacing it with a timer function dynamically controlled by the software timer to trigger VSYNC; and the filter driver providing a timing control interface for the cloud desktop to actively call to adjust the VSYNC timing, and determining the VSYNC trigger timing as the cloud desktop's rendering timing.

[0075] The GPU encoding device based on rendering synchronization provided in this application embodiment also includes: Add unit ( Figure 8(Not shown in the image) is used to adjust the virtual display driver, add an information query interface and a control interface; the information query interface is called by the host process to obtain information on the number of virtual screens, virtual screen resolution, virtual screen refresh rate and rendering timing; the control interface, based on the VSYNC timing of the filter driver, adjusts the rendering timing of the virtual display by delaying the time when the virtual display obtains rendering data from the display cache, thereby indirectly adjusting the screen capture timing of the cloud desktop; it obtains the virtual display resolution, virtual display refresh rate, protocol screen capture frame rate, binding relationship with the GPU encoding core and the upper limit of the GPU encoding core capability of each cloud desktop, and obtains the encoding timing and encoding interval corresponding to each cloud desktop based on the timing of the cloud desktop calling the encoding application programming interface API.

[0076] As one implementation method, a frame dropping scheme is determined for each virtual machine, specifically including: inputting N information of all cloud desktops bound to the same GPU encoding core. i , i∈{1,2,...,n}; select T i F i The cloud desktop with the largest value i, where T i F is the encoding time for a single frame of this cloud desktop. i D is the encoding frame rate of this cloud desktop. i The timing for encoding cloud desktop i to be first acquired and encoded within the sampling period of the encoding collaboration process; discarding one frame of encoded data for cloud desktop i in sequence, and updating the N corresponding to cloud desktop. i China F i D i E i Three parameters, and calculate the updated E i The variance; where E i The encoding interval array is used; the above process is repeated until the value of E after discarding each frame of the cloud desktop is calculated. i Given the variance of the data, select the frame with the smallest variance to discard. Input N, the updated cloud desktop information after frame dropping. i Create a set S, and include all cloud desktops in set S. Simultaneously, create initial information N0 = (0,0,0,{0}), where the parameter definition of N0 is the same as that of N... i Consistent; Select the encoding frame rate F from set S. i The smallest cloud desktop i is removed from set S.

[0077] The GPU encoding device based on rendering synchronization provided in this application embodiment also includes: Update unit ( Figure 8 (not shown in the diagram) is used to calculate the value of t that minimizes the average coding delay F(t), where F(t) = F(T0, D0, E0, T). i Di +t,E i F(·) is the average coding delay calculation function, T0, D0, and E0 are the single-frame coding time, first coding timing, and coding interval arrays in N0, respectively, and t takes the value t=E i (k) E0(j)+D i D0+T i T0, where k∈{1,2,...,sizeof(E i )}, j∈{1,2,...,sizeof(E0)}, sizeof(E i ), sizeof(E0) and sizeof(E0) are the encoding interval arrays E i The number of elements in E0; the timing of the first encoding of cloud desktop i (D). i =D i +t, connect N0 and N i Merge to obtain the new N0 = f(N0, N i ), where f(·) is the encoding information merging function, used to integrate the encoding parameters of the two cloud desktops; the host end will calculate the updated N for each cloud desktop. i The information is sent to the corresponding cloud desktop, which is based on N. i D in i The rendering timing is updated through the aforementioned timing control interface; based on N i E in i Active frame dropping is performed. When capturing the screen, it is determined whether the current frame is a dropped frame. If it is a dropped frame, no encoding operation is performed.

[0078] In an exemplary embodiment, each processing unit and related module in the GPU encoding device based on rendering synchronization of this application embodiment can be implemented by one or more central processing units (CPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components.

[0079] Regarding the apparatus in the above embodiments, the specific manner in which each module and unit performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0080] Figure 9 This is a schematic structural diagram of an electronic device 900 provided in an embodiment of this application. The electronic device may be a personal computer or a server. Figure 9 The illustrated electronic device 900 includes a processor 910, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0081] Optionally, such as Figure 9 As shown, the electronic device 900 may further include a memory 920. The processor 910 can retrieve and run computer programs from the memory 920 to implement the methods described in the embodiments of this application.

[0082] The memory 920 can be a separate device independent of the processor 910, or it can be integrated into the processor 910.

[0083] Optionally, such as Figure 9 As shown, the electronic device 900 may also include a transceiver 930, which the processor 910 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0084] The transceiver 930 may include a transmitter and a receiver. The transceiver 930 may further include antennas, and the number of antennas may be one or more.

[0085] The electronic device 900 can implement the corresponding processes of the various methods implemented in the embodiments of this application, which will not be described in detail here for the sake of brevity. That is to say, the electronic device in the embodiments of this application can be implemented as a server or as a client. For the sake of simplicity, they will not be presented separately. The main difference lies in the different methods corresponding to the programs they run, while their hardware structures are exactly the same.

[0086] Figure 10 This is a schematic structural diagram of the chip according to an embodiment of this application. Figure 10 The chip 1000 shown includes a processor 1010, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0087] Optionally, such as Figure 10As shown, chip 1000 may further include memory 1020. Processor 1010 can retrieve and run computer programs from memory 1020 to implement the methods described in this embodiment.

[0088] The memory 1020 can be a separate device independent of the processor 1010, or it can be integrated into the processor 1010.

[0089] Optionally, the chip 1000 may also include an input interface 1030. The processor 1010 can control the input interface 1030 to communicate with other devices or chips, specifically, to acquire information or data sent by other devices or chips.

[0090] Optionally, the chip 1000 may also include an output interface 1040. The processor 1010 can control the output interface 1040 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.

[0091] The chip can implement the corresponding processes of the various methods in the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0092] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0093] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. The functionality of a chip can be implemented using different processing methods depending on the program running in its memory.

[0094] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0095] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0096] This application also provides a computer-readable storage medium for storing a computer program. This computer program causes a computer to execute the corresponding processes implemented by the various methods of this application embodiment; for brevity, these will not be elaborated upon here.

[0097] This application also provides a computer program product, including computer program instructions. These computer program instructions cause a computer to execute the corresponding processes implemented by the various methods of this application embodiment; for brevity, they will not be described in detail here.

[0098] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0103] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A GPU encoding method based on rendering synchronization, characterized in that, The method includes: Obtain encoding-related information from multiple virtual machines bound to the same GPU encoding core; Based on the encoding-related information, when it is determined that the current encoding requirements exceed the processing capacity of the GPU encoding core, a frame dropping scheme is determined for each virtual machine. Based on the encoding-related information and the frame dropping scheme, the target rendering timing for each virtual machine that minimizes the average encoding latency is obtained through a collaborative adjustment algorithm. The synchronization command containing the target rendering timing and the frame dropping scheme is sent to the virtual machine. The virtual machine is triggered to determine whether the current frame is a frame to be dropped after capturing the screen image according to the frame dropping scheme in the synchronization command. If so, the virtual machine is triggered to adjust the screen image rendering timing of the local machine by modifying the vertical synchronization timing of the virtual display according to the target rendering timing in the synchronization command, so that the actual encoding request timing matches the target rendering timing.

2. The method according to claim 1, characterized in that, The method of adjusting the screen image rendering timing of the local machine by modifying the vertical synchronization timing of the virtual display includes: The virtual synchronization signal VSYNC of the virtual display is intercepted by the filter driver; Based on the target rendering timing, a software timer is dynamically set to control the triggering sequence of VSYNC, so as to delay or advance the time for obtaining rendering data from the display cache, thereby adjusting the timing of screen capture and encoding.

3. The method according to claim 2, characterized in that, The step of dynamically setting a software timer to control the triggering sequence of VSYNC based on the target rendering timing includes: Create a filter driver that dynamically controls the VSYNC timing. The filter driver hooks the VSYNC signal of the virtual display, controls the image synthesis frequency in the desktop, and dynamically adjusts the VSYNC timing. The DpiInitialize entry function, which interacts with the Windows display driver model and the graphics kernel subsystem in the HOOK operating system, is replaced with a timer function that is dynamically controlled by a software timer to trigger VSYNC. Furthermore, the filter driver provides a timing control interface for the cloud desktop to actively call in order to adjust the VSYNC timing, and determines the VSYNC triggering time as the rendering time of the cloud desktop.

4. The method according to claim 1, characterized in that, The method further includes: Adjust the virtual display driver and add an information query interface and a control interface. The information query interface is called by the host process to obtain information such as the number of virtual screens, virtual screen resolution, virtual screen refresh rate, and rendering timing. The control interface is based on the VSYNC timing of the filter driver. By delaying the time when the virtual display obtains rendering data from the display cache, the rendering timing of the virtual display is adjusted, thereby indirectly adjusting the screen capture timing of the cloud desktop. Obtain the virtual display resolution, virtual display refresh rate, protocol screen capture frame rate, binding relationship with the GPU encoding core, and the upper limit of the GPU encoding core capabilities for each cloud desktop. Based on the timing of the cloud desktop calling the encoding application programming interface (API), obtain the encoding timing and encoding interval for each cloud desktop.

5. The method according to claim 1, characterized in that, The process of determining a frame dropping scheme for each virtual machine includes: Input N information for all cloud desktops bound to the same GPU encoding core i , i∈{1,2,...,n}; Filter out T i F i The cloud desktop with the largest value i, where T i F is the encoding time for a single frame of this cloud desktop. i D is the encoding frame rate of this cloud desktop. i The timing for encoding cloud desktop i to be first acquired within the sampling period of the encoding collaboration process; Discard one frame of encoded data for cloud desktop i in sequence, and update the corresponding N for that cloud desktop. i China F i D i E i Three parameters, and calculate the updated E i The variance; where E i This is an array of encoding intervals; Repeat the above process until the value of E after discarding each frame of the cloud desktop is calculated. i The variance is calculated, and the frame with the smallest variance is selected for discarding.

6. The method according to claim 5, characterized in that, The method further includes: Input all cloud desktop information updated after frame loss N i ; Create a set S to include all cloud desktops, and simultaneously create initial information N0=(0,0,0,{0}), where the parameters of N0 are defined the same as those of N... i Consistent; Select the encoding frame rate F from set S. i The smallest cloud desktop i is removed from set S.

7. The method according to claim 5, characterized in that, The method further includes: Calculate the value of t that minimizes the average coding delay F(t), where F(t) = F(T0, D0, E0, T). i D i +t,E i F(·) is the average coding delay calculation function, T0, D0, and E0 are the single-frame coding time, first coding timing, and coding interval arrays in N0, respectively, and t takes the value t=E i (k) E0(j)+D i D0+T i T0, where k∈{1,2,...,sizeof(E i )}, j∈{1,2,...,sizeof(E0)}, sizeof(E i ), sizeof(E0) and sizeof(E0) are the encoding interval arrays E i The number of elements in E0; The first coding timing of updating cloud desktop i D i =D i +t, connect N0 and N i Merge to obtain the new N0 = f(N0, N i ), where f(·) is the encoding information merging function, used to integrate the encoding parameters of the two cloud desktops; The host will update the N calculated for each cloud desktop. i The information is sent to the corresponding cloud desktop, which is based on N. i D in i The rendering timing is updated through the aforementioned timing control interface; based on N i E in i Active frame dropping is performed. When capturing the screen, it is determined whether the current frame is a dropped frame. If it is a dropped frame, no encoding operation is performed.

8. An electronic device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the GPU encoding method based on rendering synchronization as described in any one of claims 1 to 7.

9. A chip, comprising: A processor for retrieving and running a computer program from memory, causing a device equipped with the chip to perform a GPU encoding method based on rendering synchronization as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs that cause a computer to perform a GPU encoding method based on rendering synchronization as described in any one of claims 1 to 7.

Images