Remote desktop resolution optimization method and system in wayland environment

Abstract

The invention provides a remote desktop resolution optimization method and system in a wayland environment, and the method comprises the steps: S1, obtaining a wayland protocol of screen display data, and adding a mode event in the wayland protocol, so as to achieve the communication between a remote desktop client and a local synthesizer; s2, the synthesizer calculates to obtain the optimal resolution, and sends the optimal resolution to a server side of the remote desktop through a mode event; and S3, after receiving the optimal resolution, the server issues the optimal resolution to the graphic processing unit and updates the modified display effect in the next frame of picture, and the step S2 is executed repeatedly until a stable state is reached. The method has the advantages of high flexibility, high efficiency and the like, and can allow the user to adjust the resolution of the remote desktop in real time under the condition of keeping the remote session uninterrupted.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a method and system for optimizing the remote desktop resolution in a Wayland environment. Background Art

[0002] With the development of computer graphics technology and network communication, remote desktop technology has become an indispensable part of the modern computing environment. Especially for users who need to share resources or collaborate between different locations, remote desktops provide great convenience. However, traditional remote desktop solutions have limitations in supporting high-resolution and high-refresh-rate displays, as well as dynamically adjusting display settings. For a long time, the X Window System (X11) has been the main display server protocol in Unix-like systems. It supports network-transparent operations, enabling graphical interfaces to be displayed locally or remotely over the network. However, some design flaws in X11, such as performance issues, security, and complexity, have led to the birth of a new generation of display server protocol - Wayland. Wayland, with its simplified design, higher security, and better hardware acceleration support, has gradually become the new favorite in the open-source community and has been adopted as the default graphical display protocol by major Linux distributions.

[0003] The advantages of Wayland over X11 are particularly evident in its support for high-resolution and high-refresh-rate displays. Due to its more efficient architecture, Wayland can provide a smoother user experience, especially when dealing with graphics-intensive applications. In addition, Wayland has also optimized the input event processing flow, reducing latency and improving response speed. Although Wayland has brought many improvements, it does not natively support the remote display function, which is in sharp contrast to X11. To implement the remote desktop function, other remote desktop protocols (such as VNC and RDP) must be used. This approach introduces a new challenge, namely how to achieve the ability to dynamically adjust the resolution without interrupting the remote session.

[0004] In existing remote desktop technologies, whether it is the Windows operating system or the Ubuntu system, modifying the resolution usually requires disconnecting the remote desktop connection, adjusting the settings, and then reconnecting. This process not only increases the operational complexity but also brings unnecessary interruptions and delays, affecting the user experience. For example, under the Windows operating system, modifying the resolution requires exiting the remote desktop software and reconnecting; when using the VNC service in Linux systems such as Ubuntu, the service also needs to be restarted to change the resolution settings. It should be noted that although Chinese Patent Application 202210604669.6 solves the problem of automatic reconnection after the remote desktop connection is interrupted, it does not address the key pain point of online resolution adjustment. Similarly, Chinese Patent Application 202310823579.0 focuses on solving the problem of obtaining the hardware cursor in the remote desktop under the Wayland environment and does not involve the functional optimization of resolution adjustment. Summary of the Invention

[0005] The technical problem to be solved by the present invention: Aiming at the above problems of the prior art, a method and system for optimizing the remote desktop resolution in the Wayland environment with high flexibility and efficiency are provided, which can allow users to adjust the resolution of the remote desktop in real time without disconnecting the remote session.

[0006] To solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0007] A method for optimizing the remote desktop resolution in the Wayland environment includes the following steps:

[0008] Step S1: Obtain the Wayland protocol of the screen display data, add a mode event to the Wayland protocol to implement communication between the remote desktop client and the local compositor;

[0009] Step S2: The Wayland compositor of the remote desktop application end calculates the optimal resolution data based on the local display attributes and sends the optimal resolution data to the remote desktop server through the mode event;

[0010] Step S3: After receiving the optimal resolution, the server distributes the optimal resolution to the graphics processing unit and updates the modified display effect in the next frame of the picture. Repeat Step S2 until a stable state is reached.

[0011] As a further improvement of the method of the present invention: In Step S1, the method for implementing communication between the remote desktop client and the local compositor further includes adding a dbus interface to the local compositor application program. The dbus interface is used to query the currently available resolution configurations and transmit the new configuration information to the local compositor.

[0012] As a further improvement of the method of the present invention: in the step S2, the method for the synthesizer to obtain the optimal resolution includes:

[0013] Step S201: Access the graphics card hardware through the / dev / dri / cardX device node;

[0014] Step S202: Use drmModeGetResources to obtain the Connector information of the current graphics card;

[0015] Step S203: Traverse all supported resolutions from the modes linked list of the Connector, and filter out the optimal mode by whether the type attribute supports DRM_MODE_TYPE_PREFERRED.

[0016] As a further improvement of the method of the present invention: in the step S3, in the step S301 of sending the optimal resolution to the graphics processing unit: the server parses the received optimal resolution and scaling attribute, synchronizes with the display backend through the sync request of the wl_display protocol, and compares the horizontal resolution, vertical resolution, and refresh rate attributes of the new configuration and the current configuration; if it is detected that the resolution or scaling ratio is inconsistent, the done event of wl_output is triggered to notify the client to update;

[0017] Step S302: When the server updates the resolution / scaling, ensure the security of the configuration switch with the help of the atomic submission and dynamic rollback mechanisms; use the atomic submission interface provided by the graphics processor driver to send the display configuration, so that the resolution / scaling modification is an atomic operation. If the new configuration causes display anomalies, it will automatically roll back to the nearest valid configuration;

[0018] Step S303: For X applications running on the XWayland compatibility layer, the server will synchronize the scaling ratio through the Xft.dpi attribute and call the xrandr interface to adjust the virtual resolution of the X11 client to prevent UI element misalignment and avoid display anomalies.

[0019] As a further improvement of the method of the present invention: in the step S3, the graphics processing unit includes a display backend, and the display backend uses the atomic submission interface provided by the graphics processor driver to update the display parameters of the CRTC and Connector modules. When the resolution changes, the frame buffer is reallocated, and a large-sized buffer is created to match the physical display requirements; a double-buffering mechanism is established. While the current buffer is being displayed, the second buffer is being rendered, so that the new resolution or scaling parameter takes effect only in the next vertical synchronization period, avoiding screen tearing.

[0020] As a further improvement of the method of the present invention: The graphics processing unit further includes a rendering backend, which applies a scaling factor when sampling textures from the client buffer through the GL renderer, and triggers the start of rendering of the next frame when detecting a frame event of the wlr_output protocol.

[0021] As a further improvement of the method of the present invention: The graphics processing unit further includes a Wayland client application, which receives scaling and resolution update events in real time through the wl_output protocol, triggers redrawing or buffer size adjustment according to the received update events, responds to the DPI value to adjust the font and control sizes, and synchronously updates the projection matrix to make the window geometry calculation consistent with the scaling parameters.

[0022] The present invention also provides a remote desktop resolution optimization system in a Wayland environment, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the remote desktop resolution optimization method in the Wayland environment.

[0023] The present invention also provides a computer-readable storage medium, in which a computer program / instructions are stored, and the computer program / instructions are programmed or configured to execute the remote desktop resolution optimization method in the Wayland environment through a processor.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] When the present invention modifies the resolution through a remote desktop client, mode and scale events are added to the obtained Wayland protocol to realize the communication between the remote desktop client and the local compositor, and the optimal resolution and scaling data are obtained through the compositor. The optimal resolution and scaling data are sent to the server of the remote desktop through the mode and scale events, and the modified display effect is obtained after processing. This enables the remote client to communicate with the local compositor by sending events, dynamically request to modify the resolution and scaling ratio without restarting or disconnecting, and can effectively solve the problem that the remote desktop needs to interrupt the connection when modifying the resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the desktop graphics output of the Wayland compositor remote desktop in the embodiment of the present invention.

[0027] Figure 2 It is a flowchart of the remote desktop resolution optimization method in the Wayland environment in the embodiment of the present invention.

[0028] Figure 3This is the flowchart for modifying the resolution or scaling of the Wayland compositor mirror remote desktop in the embodiments of the present invention. Detailed implementation manners

[0029] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0030] The abbreviations and key terms involved in the present invention are as follows:

[0031] Wayland: It is a communication protocol library for building display servers and client applications. It aims to replace the X Window System (X11) as the graphics display system on Linux and other operating systems.

[0032] Wayland Compositor: It is the Wayland server side that communicates with Wayland clients and also has window composition and management functions.

[0033] As Figure 1 shown, the remote desktop system in this embodiment is mainly divided into two parts, namely local physical output and remote desktop output. The user sets configuration parameters such as resolution and scaling parameters on the local client, and the configuration parameters are passed to the local physical output through data copying. After receiving the configuration parameters, the local physical output module passes them to the rendering and composition unit. At the same time, the configuration parameters are transmitted to the remote desktop output module through the network and processed by the compositor at the remote end. The rendered and composed images are respectively submitted to the local display device and the remote display device, and the new display effect is applied when the next frame is updated, realizing seamless display effect update. The whole process ensures that the display settings can be adjusted in real time and efficiently both locally and remotely. Its detailed technical solution is as follows:

[0034] As Figure 2 shown, the method for optimizing the remote desktop resolution in the Wayland environment of this embodiment includes the following steps:

[0035] Step S1: Extend the Wayland protocol to support resolution adjustment communication;

[0036] For the Wayland protocol for obtaining screen display data, add a mode event to the Wayland protocol to realize the communication between the remote desktop client and the local compositor.

[0037] This embodiment extends on the standard Wayland protocol by adding new event types, enabling richer interaction between the remote desktop client and the compositor.

[0038] Step S2: The compositor calculates and sends the optimal resolution settings;

[0039] The compositor calculates the optimal resolution and sends the optimal resolution to the server of the remote desktop via the mode event.

[0040] In this embodiment, the method for the compositor to obtain the optimal resolution includes:

[0041] Step S201: Access the graphics card hardware through the / dev / dri / cardX device node;

[0042] Step S202: Use drmModeGetResources to obtain the Connector information of the current graphics card;

[0043] Step S203: Traverse all supported resolutions from the modes linked list of the Connector, and filter out the optimal mode by whether the type attribute supports DRM_MODE_TYPE_PREFERRED;

[0044] In this embodiment, it also includes adding a scale event to the Wayland protocol and calculating the optimal scaling data by the compositor.

[0045] Step S3: The server applies and dynamically updates the optimal resolution;

[0046] After receiving the optimal resolution, the server distributes the optimal resolution to the graphics processing unit and updates the modified display effect in the next frame. Step S2 is repeatedly executed until a stable state is reached.

[0047] In this embodiment, the process of distributing the optimal resolution to the graphics processing unit includes:

[0048] Step S301: The server parses the received optimal resolution and scaling attributes, synchronizes with the display backend through the sync request of the wl_display protocol, and compares the horizontal resolution, vertical resolution, and refresh rate attributes of the new configuration with the current configuration; if it detects that the resolution or scaling ratio is inconsistent, it triggers the done event of wl_output to notify the client to update;

[0049] Step S302: When the server's resolution / scale is updated, the atomic commit and dynamic rollback mechanisms are used to ensure the security of configuration switching. The display configuration is sent through the atomic commit interface provided by the graphics processor driver to ensure that the resolution / scale modification is an atomic operation. If the new configuration causes display anomalies (such as exceeding the EDID support range), it will automatically roll back to the nearest valid configuration;

[0050] Step S303: For X applications running on top of the XWayland compatibility layer, the server will synchronize the scaling ratio through the Xft.dpi property and call the xrandr interface to adjust the virtual resolution of the X11 client to prevent UI element misalignment and avoid display anomalies.

[0051] In this embodiment, the display backend updates the specific display parameters of the CRTC and Connector modules through the atomic commit interface provided by the graphics processor driver to ensure the atomicity of the hardware mode setting; the resolution change may trigger the reallocation of the frame buffer, creating a larger buffer size to match the physical display requirements; a double-buffering mechanism is established, and the second buffer is rendered simultaneously during display to ensure that the new resolution / scale parameters take effect only in the next vertical synchronization period, avoiding tearing.

[0052] The rendering backend applies the scaling factor when sampling the texture of the client buffer through the GL renderer; the next frame start rendering is triggered by the frame event of the wlr_output protocol, notifying the client in advance to prepare new data to avoid the screen tearing caused by partial updates;

[0053] The wayland client application receives the scaling and resolution update events through the wl_output protocol, triggering redrawing or buffer size adjustment; responding to the DPI value and adjusting the font and control sizes; synchronously updating the projection matrix to ensure that the window geometry calculation is consistent with the scaling parameters;

[0054] In summary, in this embodiment, the control center in the remote desktop modifies the resolution and scale through the kde protocol; sends a resolution and scale property modification request to the server through the wayland kde protocol. After receiving the resolution or scale modification request, the server notifies the display backend, the rendering backend, and other wayland client applications, and then updates the modified display effect in the next frame update.

[0055] The method of this embodiment allows for dynamically adjusting the resolution and scaling settings of a remote desktop without interrupting the remote connection, thus providing a smoother and more user-friendly experience and effectively avoiding the problem in the prior art where it is usually necessary to disconnect and re-establish the connection to apply new resolution or scaling settings. At the same time, in this embodiment, by directly adding mode and scale event types to the Wayland protocol, changes in resolution and scaling can take effect immediately when the next frame of the screen is updated, reducing the user's waiting time and improving work efficiency.

[0056] In this embodiment, since the Wayland display server protocol itself is extended or improved to support new functional features, it is ensured that the optimal resolution and scaling data can be efficiently transmitted from the local compositor to the remote desktop server, reducing unnecessary data transmission and processing latency.

[0057] Embodiment Two

[0058] In this embodiment, dbus messages are sent in the remote application window to modify the resolution or scaling; dbus, as a lightweight IPC (Inter-Process Communication), is used for inter-process communication or communication between a process and the kernel. After the compositor application software receives the dbus message, it sends a resolution and scaling attribute modification request through the kde (K Desktop Environment) protocol, notifies the display backend, rendering backend, and other Wayland client applications when processing the request, and updates the display effect after the modification in the next frame of the screen.

[0059] In this embodiment, the method for implementing communication between the remote desktop client and the local compositor further includes adding a dbus interface in the compositor application program. The dbus interface is used to query the currently available resolution configurations and transmit the new configuration information to the compositor.

[0060] In this embodiment, the method for sending the optimal resolution to the remote desktop server further includes sending the optimal resolution to the remote desktop server through the dbus interface.

[0061] Specifically, the user selects and issues the configuration through the packaged UI interface or shortcut in the remote desktop. The underlying implementation is to transmit the configuration information across processes through the dbus interface to issue and make the resolution or scaling attributes take effect. Of course, the Wayland protocol can also be used here. This embodiment applies the existing inter-process communication method to a new scenario. The main advantage compared to the prior art is that there is no need to close the remote connection and no need to re-remote; the configuration can be directly modified and take effect immediately.

[0062] In a specific application embodiment, the method for modifying the resolution or scaling of the Wayland compositor remote desktop is as Figure 3 shown and includes:

[0063] Step 1: The system queries the currently available resolution and scaling options for the user to make a selection.

[0064] Step 2: The user sets the required resolution and scaling parameters through the remote desktop client interface.

[0065] Step 3: For the wayland protocol that obtains the screen display data, add a dbus interface to the compositor application to query the optional resolution configurations and send configuration information;

[0066] Step 4: The wayland compositor obtains the optimal resolution and scaling data and notifies the remote compositor application through the dbus interface;

[0067] Step 5: Based on the wayland events or the mode and scale information received through dbus communication, the remote desktop server obtains the resolution and scaling data for the display, issues the configuration, and updates the screen;

[0068] Step 6: When the user modifies the resolution again through the remote desktop client window, repeat Steps 3 and 4 to update the remote desktop display until a stable display state is reached or the specific requirements of the user are met.

[0069] Specifically, the wayland compositor on the remote desktop application side calculates the optimal resolution and scaling configuration based on the local display attributes (such as screen resolution, refresh rate, scaling ratio, etc.) to ensure the best display effect of the remote desktop on the local device. The obtained optimal resolution and scaling configuration can be automatically applied to the remote desktop connection by the system without user intervention, or the user can manually adjust it according to the requirements and issue it to the remote desktop server. After receiving the above configuration, the remote desktop server will re-render the content of the remote desktop according to the new resolution and scaling settings. After re-rendering, the remote desktop server sends the updated display content back to the remote desktop application side to ensure that the remote desktop content seen by the user on the local device is generated according to the optimal configuration, with a clear, accurate display effect and no distortion or unnecessary cropping.

[0070] This embodiment also provides a remote desktop resolution optimization system in a wayland environment, including a microprocessor and a memory connected to each other. The microprocessor is programmed or configured to execute the remote desktop resolution optimization method in a wayland environment.

[0071] This embodiment also provides a computer-readable storage medium, in which a computer program / instructions are stored. The computer program / instructions are programmed or configured to execute the remote desktop resolution optimization method in a wayland environment through a processor.

[0072] This embodiment also provides a computer program product, including a computer program / instructions, which are programmed or configured to execute the remote desktop resolution optimization method in the wayland environment through a processor.

[0073] Those skilled in the art should understand that the above embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks. The above are only the preferred embodiments of the present invention and do not impose any form of limitation on the present invention. Although the present invention has been disclosed above with the preferred embodiments, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change, and modification made to the above embodiments according to the technical essence of the present invention without departing from the technical solution of the present invention should fall within the scope of the technical solution of the present invention.

[0074] The above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the concept of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should also be regarded as within the protection scope of the present invention.

Claims

1. A remote desktop resolution optimization method under a Wayland environment, characterized in that: The following steps are involved: Step S1: obtaining the wayland protocol of screen display data, adding a mode event to the wayland protocol, and realizing the communication between the remote desktop client and the local synthesizer; Step S2: the wayland synthesizer of the remote desktop application side calculates the optimal resolution data according to the local display properties, and sends the optimal resolution data to the remote desktop server side through the mode event; Step S3: After receiving the optimal resolution data, the server sends the optimal resolution data to the graphics processing unit, updates the modified display effect in the next frame, and repeats step S2 until a stable state is reached.

2. The remote desktop resolution optimization method under the wayland environment according to claim 1, characterized in that: In the step S1, the implementation of communication between the remote desktop client and the local synthesizer also includes adding a dbus interface in the local synthesizer application, wherein the dbus interface is used to query the current optional resolution configuration and transmit the new configuration information to the local synthesizer.

3. The remote desktop resolution optimization method under the wayland environment according to claim 1 is characterized in that: In step S2, the method for the synthesizer to obtain the optimal resolution includes: Step S201: Access the graphics card hardware through the / dev / dri / cardX device node; Step S202: Use drmModeGetResources to obtain the Connector information of the current graphics card; Step S203: traverse all supported resolutions from the modes list of Connector, and select the best mode based on whether the type attribute supports DRM_MODE_TYPE_PREFERRED.

4. The remote desktop resolution optimization method under the wayland environment according to claim 1, characterized in that: In the step S2, the method of sending the best resolution to the server of the remote desktop also includes sending the best resolution to the server of the remote desktop through the dbus interface.

5. The remote desktop resolution optimization method under the wayland environment according to claim 1, characterized in that: In step S3, the method of sending the optimal resolution to the graphics processing unit includes: Step S301: The server parses the received optimal resolution and scaling attributes, synchronizes with the display backend through the sync request of the wl_display protocol, and compares the horizontal resolution, vertical resolution, and refresh rate attributes of the new configuration with the current configuration; if it is detected that the resolution or scaling ratio is inconsistent, the done event of wl_output is triggered to notify the client to update; Step S302: When the server-side resolution / zoom is updated, the security of configuration switching is achieved through atomic submission and dynamic rollback mechanisms; the display configuration is issued through the atomic submission interface provided by the graphics processor driver, so that the resolution / zoom modification is an atomic operation. If the new configuration causes display abnormality, it will automatically roll back to the most recent valid configuration; Step S303: For X applications running on the XWayland compatibility layer, the server synchronizes the scaling ratio through the Xft.dpi attribute and calls the xrandr interface to adjust the virtual resolution of the X11 client to prevent UI element misalignment and display abnormalities.

6. The remote desktop resolution optimization method under the wayland environment according to claim 1, characterized in that: In step S3, the graphics processing unit includes a display backend, which uses the atomic submission interface provided by the graphics processor driver to update the display parameters of the CRTC and Connector modules, reallocates the frame buffer when the resolution changes, and creates a large-size buffer to match the physical display requirements; establishes a double buffer mechanism, and renders the second buffer while displaying the current buffer, so that the new resolution or scaling parameters take effect only in the next vertical synchronization cycle to avoid screen tearing.

7. The remote desktop resolution optimization method under the wayland environment according to claim 6 is characterized in that: The graphics processing unit also includes a rendering backend that applies a scaling factor when texture sampling is performed on a client buffer by a GL renderer and triggers the start of rendering of a next frame when a frame event of the wlr_output protocol is detected.

8. The remote desktop resolution optimization method under the wayland environment according to claim 6 is characterized in that: The graphics processing unit also includes a wayland client application, which receives scaling and resolution update events in real time through the wl_output protocol, triggers redrawing or buffer size adjustment according to the received update events, responds to the DPI value to adjust the font and control size, and synchronously updates the projection matrix so that the window geometry calculation is consistent with the scaling parameters.

9. A remote desktop resolution optimization system under a Wayland environment, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the remote desktop resolution optimization method under the wayland environment as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instruction stored therein, characterized in that: The computer program / instruction is programmed or configured to execute the remote desktop resolution optimization method under the wayland environment described in any one of claims 1 to 8 through a processor.

Images