Remote display control method and apparatus based on uefi firmware

Abstract

The application relates to a UEFI firmware-based remote display control method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a protocol instance configured in a computer equipment through a standard interface of UEFI firmware in the computer equipment; in the case that the protocol instance comprises a graphic output protocol instance and the frame buffer address of the graphic output protocol instance is a valid address, and the computer equipment is connected with a display equipment, establishing an authentication process of the computer equipment and a remote client based on a connection request sent by the remote client; in the case that the authentication is passed, obtaining interface configuration content of the UEFI firmware matched with an input event based on a graphic output protocol instance corresponding to the display equipment in response to the input event triggered by the remote client; and displaying the interface configuration content through the remote client according to display attribute information corresponding to the display equipment. The method can reduce the remote control cost.

Description

Technical Field

[0001] This application relates to the field of computer firmware and remote management technology, and in particular to a remote display control method, device, computer equipment, computer-readable storage medium and computer program product based on UEFI firmware. Background Technology

[0002] With the expansion of data centers and the surge in edge computing device deployments, the need for remote configuration, diagnostics, and maintenance of computing devices at the firmware level (before the operating system boots) is becoming increasingly urgent. Traditional operations and maintenance rely on local keyboards, monitors, mice, or dedicated Baseboard Management Controller (BMC) hardware. The former cannot meet the remote management needs of large-scale clusters, while the latter is difficult to popularize in consumer-grade and lightweight edge devices due to its high cost and large motherboard footprint.

[0003] In traditional technologies, out-of-band remote control can be achieved by integrating dedicated hardware into the motherboard. For example, dedicated hardware can be hardware that supports Active Management Technology (AMT) or hardware that supports Desktop and Mobile Architecture for System Hardware (DASH).

[0004] However, relying on dedicated hardware increases hardware costs, which in turn increases the control costs of remote control. Summary of the Invention

[0005] Therefore, it is necessary to provide a remote display control method, device, computer equipment, computer-readable storage medium, and computer program product based on UEFI firmware that can reduce the cost of remote control, addressing the aforementioned technical problems.

[0006] In a first aspect, this application provides a remote display control method based on UEFI firmware, including:

[0007] Obtain the protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device;

[0008] If it is determined that the computing device is connected to the display device based on the fact that the protocol instance includes a graphics output protocol instance and the frame buffer address of the graphics output protocol instance is a valid address, an authentication process between the computing device and the remote client is established based on the connection request sent by the remote client.

[0009] If authentication is successful, in response to an input event triggered by the remote client, the interface configuration content of the UEFI firmware that matches the input event is obtained based on the graphics output protocol instance corresponding to the display device;

[0010] The interface configuration content is displayed through the remote client according to the display attribute information corresponding to the display device.

[0011] In one embodiment, the method further includes:

[0012] If the protocol instance does not include a graphics output protocol instance, a request is sent to the memory allocation interface of the UEFI boot service; the request is used to request physical memory.

[0013] The frame buffer address of the physical memory is filled into the structure allocated in the UEFI memory pool to obtain a virtual graphics output protocol instance;

[0014] By installing the virtual graphics output protocol instance onto the UEFI handle of the computing device, it is determined that the computing device is connected to the display device.

[0015] In one embodiment, the computing device communicates with the remote client via a physical network interface card (NIC), the NIC including a transmit buffer and a receive buffer;

[0016] In response to an input event triggered by the remote client, the interface configuration content of the UEFI firmware matching the input event is obtained based on the graphics output protocol instance corresponding to the display device, including:

[0017] In response to an input event triggered by the remote client, the input message corresponding to the input event is read from the receive buffer;

[0018] Based on the input message, determine the event instance corresponding to the input event;

[0019] Based on the event instance and the graphics output protocol instance corresponding to the display device, the interface configuration content of the UEFI firmware matching the input event is obtained; the interface configuration content is cached in the transmission buffer, and the remote client reads the interface configuration content from the transmission buffer.

[0020] In one embodiment, obtaining the interface configuration content of the UEFI firmware matching the input event based on the event instance and the graphics output protocol instance corresponding to the display device includes:

[0021] Obtain the interface buffer content from the frame buffer address of the graphics output protocol instance corresponding to the display device;

[0022] The interface display content corresponding to the event instance is generated using the graphics output protocol instance corresponding to the display device;

[0023] In the case where there is incremental content between the interface buffer content and the interface display content, the interface configuration content of the UEFI firmware that matches the input event is determined based on the incremental content.

[0024] In one embodiment, the input event includes a keyboard injection event; determining the event instance corresponding to the input event based on the input message includes:

[0025] Parse the input message to obtain the key code corresponding to the keyboard injection event and the key state corresponding to the key code;

[0026] When the key is in the pressed state, the key code is converted into a target code that matches the type of the key code;

[0027] The target code is filled into the constructed first structure to obtain the event instance corresponding to the input event.

[0028] In one embodiment, the input event includes a pointer injection event, and the display attribute information includes frame buffer resolution; determining the event instance corresponding to the input event based on the input message includes:

[0029] Parse the input message to obtain the position coordinates and key mask corresponding to the pointer injection event;

[0030] When the position coordinates are normalized according to the frame buffer resolution, the normalized position coordinates are mapped to absolute coordinates that match the UEFI firmware.

[0031] The button state corresponding to the key mask and the absolute coordinates are filled into the second structure to obtain the event instance corresponding to the input event.

[0032] Secondly, this application also provides a remote display control device based on UEFI firmware, comprising:

[0033] The acquisition module is used to acquire the protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device;

[0034] The first processing module is used to establish an authentication process between the computing device and the remote client based on a connection request sent by the remote client, when it is determined that the computing device has been connected to the display device, based on the fact that the protocol instance includes a graphics output protocol instance and the frame buffer address of the graphics output protocol instance is a valid address.

[0035] The second processing module is used to, upon successful authentication, respond to an input event triggered by the remote client and obtain the interface configuration content of the UEFI firmware that matches the input event based on the graphics output protocol instance corresponding to the display device.

[0036] The third processing module is used to display the interface configuration content through the remote client according to the display attribute information corresponding to the display device.

[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0038] Obtain the protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device;

[0039] If it is determined that the computing device is connected to the display device based on the fact that the protocol instance includes a graphics output protocol instance and the frame buffer address of the graphics output protocol instance is a valid address, an authentication process between the computing device and the remote client is established based on the connection request sent by the remote client.

[0040] If authentication is successful, in response to an input event triggered by the remote client, the interface configuration content of the UEFI firmware that matches the input event is obtained based on the graphics output protocol instance corresponding to the display device;

[0041] The interface configuration content is displayed through the remote client according to the display attribute information corresponding to the display device.

[0042] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0043] Obtain the protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device;

[0044] If it is determined that the computing device is connected to the display device based on the fact that the protocol instance includes a graphics output protocol instance and the frame buffer address of the graphics output protocol instance is a valid address, an authentication process between the computing device and the remote client is established based on the connection request sent by the remote client.

[0045] If authentication is successful, in response to an input event triggered by the remote client, the interface configuration content of the UEFI firmware that matches the input event is obtained based on the graphics output protocol instance corresponding to the display device;

[0046] The interface configuration content is displayed through the remote client according to the display attribute information corresponding to the display device.

[0047] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0048] Obtain the protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device;

[0049] If it is determined that the computing device is connected to the display device based on the fact that the protocol instance includes a graphics output protocol instance and the frame buffer address of the graphics output protocol instance is a valid address, an authentication process between the computing device and the remote client is established based on the connection request sent by the remote client.

[0050] If authentication is successful, in response to an input event triggered by the remote client, the interface configuration content of the UEFI firmware that matches the input event is obtained based on the graphics output protocol instance corresponding to the display device;

[0051] The interface configuration content is displayed through the remote client according to the display attribute information corresponding to the display device.

[0052] The aforementioned remote display control method, device, computer equipment, computer-readable storage medium, and computer program product based on UEFI firmware first obtains a protocol instance by calling the standard interface of the UEFI firmware. It then determines that the computing device is connected to the display device by verifying that the protocol instance contains a graphics output protocol instance and that the frame buffer address of the graphics output protocol instance is valid. This step requires no additional hardware; the firmware's own protocol query mechanism automatically and accurately detects the display device's presence, laying a reliable foundation for subsequent remote control and avoiding invalid operations and resource waste caused by blindly establishing connections. Next, based on the connection request sent by the remote client, an authentication process is established between the computing device and the remote client. Upon successful authentication, the interface configuration content of the UEFI firmware matching the input event is obtained based on the graphics output protocol instance corresponding to the display device. This achieves real-time capture of remote input events at the firmware stage, without relying on dedicated management hardware, allowing the remote client to interact with the UEFI firmware as if operating a local display device. Finally, based on the display attribute information of the display device itself, the content is configured through the remote client display interface, ensuring the consistency between the content presented by the remote client and the content of the local display device, which provides a seamless remote operation experience. At the same time, since all processing is completed based on UEFI firmware, the dependence on dedicated hardware is completely eliminated, and the cost of remote control is greatly reduced while maintaining out-of-band remote control capabilities. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating a remote display control method based on UEFI firmware in one embodiment;

[0055] Figure 2 This is a flowchart illustrating the processing flow of the detection and abstraction unit in one embodiment;

[0056] Figure 3 This is a flowchart of the virtual GOP generation and frame buffer management unit in one embodiment;

[0057] Figure 4 This is a flowchart of the processing of a UEFI input event injection unit in one embodiment;

[0058] Figure 5This is a flowchart illustrating the processing flow of the asynchronous event scheduling and lifecycle control unit in one embodiment.

[0059] Figure 6 This is a flowchart of the processing flow of a secure RFB protocol engine unit in one embodiment;

[0060] Figure 7 This is a structural block diagram of a remote display control system in one embodiment;

[0061] Figure 8 This is a flowchart illustrating remote display control based on various units in a remote display control system in one embodiment.

[0062] Figure 9 This is a flowchart illustrating a remote display control method based on UEFI firmware in another embodiment;

[0063] Figure 10 This is a structural block diagram of a remote display control device based on UEFI firmware in one embodiment;

[0064] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0067] Traditional technologies can achieve out-of-band remote control by integrating dedicated hardware or firmware into the motherboard. However, relying on dedicated hardware increases hardware costs and motherboard design complexity. Furthermore, the BMC runs its own independent firmware stack, which is prone to security vulnerabilities and is isolated from the host's Unified Extensible Firmware Interface (UEFI) secure boot trust chain. UEFI is a specification that defines the interface between the operating system and platform firmware, used to replace the traditional Basic Input / Output System (BIOS). Moreover, when implementing remote control based on UEFI firmware text, it mainly relies on a partial implementation of the EFI Development Kit II (EDK2) framework. This involves redirecting the firmware's text console output to the network via UEFI serial port redirection or a Teletype Network (Telnet) server. EDK2 is an open-source reference implementation framework for UEFI firmware. However, this method can only transmit text character streams or data packets and cannot achieve graphical interface remote interaction. In addition, remote control can be achieved by running a Virtual Network Computing (VNC) server or a Remote Desktop Protocol Service (RDP) service after the operating system is loaded. VNC is a remote desktop control system based on the Remote Frame Buffer (RFB) protocol. However, whether it is a general VNC / RDP or a lightweight remote tool, it runs as an application or service of the operating system. In scenarios such as the operating system not being installed, crashing, corrupting startup files, or failing secure boot verification, the remote control function is completely lost, and the administrator cannot perform any remote intervention.

[0068] Based on the above analysis, this application provides a remote display control method based on UEFI firmware. This method can be applied to the VNC component in the UEFI firmware or the UEFI scheduler in a computing device. The VNC component includes VNC driver software or a VNC server in the UEFI firmware. The VNC server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0069] In one exemplary embodiment, such as Figure 1 As shown, a remote display control method based on UEFI firmware is provided. Taking the application of this method to a VNC component or a UEFI scheduler as an example, the method includes the following steps:

[0070] S102, obtain the protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device.

[0071] The computing device refers to a device configured with UEFI firmware.

[0072] In an optional embodiment, obtaining a protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device includes: obtaining a protocol instance configured in the computing device from a protocol database through the standard interface of the UEFI firmware in the computing device.

[0073] In an optional embodiment, the VNC component or UEFI scheduler is communicatively connected to the remote display control system. The remote display control system can be integrated as a firmware module into the Queue Serial Peripheral Interface (QSPI) flash memory storing the UEFI firmware image, and is loaded and executed during the Driver Execution Environment (DXE) stage. The DXE stage is a crucial stage in the UEFI boot process, during which device drivers and applications are loaded and executed.

[0074] Specifically, the remote display control system includes a display detection and abstraction unit. Specifically, it obtains the protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device, including: when the driver entry function is called, obtaining the protocol instance configured in the computing device through the display detection and abstraction unit. Here, the driver entry function refers to InitializeVncDriver.

[0075] The standard interface refers to LocateProtocol(&gEfiGraphicsOutputProtocolGuid, NULL, &Gop). In other words, by calling the standard interface LocateProtocol(&gEfiGraphicsOutputProtocolGuid, NULL, &Gop) in the UEFI firmware, the protocol instances (i.e., EFI_GRAPHICS_OUTPUT_PROTOCOL instances) in the computing device can be enumerated.

[0076] In an optional embodiment, the protocol instance configured in the computing device is obtained through the standard interface of the UEFI firmware in the computing device. This includes: when the computing device is powered on, if the UEFI firmware has reached the DXE stage, the protocol instance configured in the computing device is obtained through the standard interface of the UEFI firmware in the computing device. Therefore, by obtaining the protocol instance early in the firmware startup process, a foundation is laid for subsequent remote control, display output, and input event processing, ensuring that remote functions are effective before the operating system starts, without relying on additional hardware or operating system environment.

[0077] S104, if it is determined that the computing device is connected to the display device based on the protocol instance including the graphics output protocol instance and the frame buffer address of the graphics output protocol instance being a valid address, an authentication process between the computing device and the remote client is established based on the connection request sent by the remote client.

[0078] Specifically, if the protocol instance in the protocol database includes the Graphics Output Protocol (GOP) and the frame buffer address of the GOP instance is a valid physical address, it is determined that the computing device is connected to a local physical display. In other words, if the GOP is not empty and the frame buffer address "Mode->FrameBufferBase" is a valid physical address, it is determined that the current computing device is connected to a local physical display.

[0079] The GOP protocol is a standard protocol defined by the UEFI specification, used to abstract linear frame buffer access in graphics display devices.

[0080] In an optional embodiment, the method further includes: when it is determined that the computing device is connected to the display device based on the fact that the protocol instance includes a graphics output protocol instance and the frame buffer address of the graphics output protocol instance is a valid address, marking the graphics output protocol instance as the main frame buffer source, and setting the flag bit of the computing device as a first flag bit, the first flag bit being used to characterize that the computing device is a device with a screen. For example, if the first flag bit is FALSE, then the flag bit IsHeadless of the computing device is set to FALSE.

[0081] In this context, "remote client" refers to a remote VNC client. Therefore, a remote client is a remote desktop control system based on the RFB protocol. The RFB protocol is the application layer protocol used by the VNC remote control system to transmit graphics frame updates and user input events. The necessary subset of the RFB protocol (i.e., the functions included in basic RFB communication) includes: protocol version handshake, security type negotiation (Type 2: VNC authentication), client initialization, server initialization, and pixel format and encoding negotiation.

[0082] In an optional embodiment, the method further includes: creating a timer event; and determining whether a connection request from a remote client has been received while the Transmission Control Protocol (TCP) port 5900 of the computing device is in a listening state via the timer event.

[0083] In an optional embodiment, the remote display control system further includes an asynchronous event scheduling and lifecycle control unit and a secure RFB protocol engine unit. The method further includes: upon initializing the network protocol stack of the secure RFB protocol engine unit, invoking the asynchronous event scheduling and lifecycle control unit to create a timer event. The network protocol stack can refer to a TCP protocol stack, an Internet Protocol (IP) protocol stack, or other types of protocol stacks.

[0084] Specifically, the process of creating a timer event includes: calling gBS->CreateEvent(), setting the event type to EVT_TIMER|EVT_NOTIFY_SIGNAL, setting the notification function priority (NotifyTpl) to TPL_CALLBACK (lower than most startup tasks in the UEFI priority system), and pointing the notification function pointer to the predefined VncTimerCallback function; then calling gBS->SetTimer() to set the event to periodic trigger mode, and setting the trigger interval TimerPeriod to a preset time. Therefore, the created timer event can determine whether a remote client has sent a connection request at preset intervals. The preset time can be set to 100 milliseconds (i.e., triggering 10 times per second) or other values.

[0085] Task Priority Level (TPL) is a priority mechanism used within UEFI to manage the execution order of event notification functions.

[0086] The VncTimerCallback function is scheduled to execute each time the timer expires and the UEFI kernel is idle at the TPL_CALLBACK level. Internally, it is implemented as a finite state machine, performing only one phase of finite work in a single call before returning immediately. Specifically, this includes:

[0087] State A (Listening State): If no remote client has connected yet, execute a non-blocking Transmission Control Protocol (TCP) accept() attempt. If a new connection is established, switch to State B.

[0088] State B (Authentication State): This step involves performing the Remote Frame Buffer (RFB) protocol handshake and password authentication (such as sending the protocol version or verifying the password). If authentication is successful, the system switches to State C (Service State). Specifically, the processing for State C includes: responding to an input event triggered by a remote client, and obtaining the UEFI firmware's interface configuration content matching the input event based on the graphical output protocol instance corresponding to the display device.

[0089] As can be seen from the above, when a connection request sent by a remote client is detected, authentication is required between the computing device and the remote client, that is, an authentication process is established between the computing device and the remote client.

[0090] S106, upon successful authentication, responds to an input event triggered by a remote client and obtains the UEFI firmware interface configuration content matching the input event based on the graphical output protocol instance corresponding to the display device.

[0091] In an optional embodiment, determining whether authentication between the computing device and the remote client is successful includes: if a TCP three-way handshake is completed between the computing device and the remote client, proceeding to the RFB protocol processing flow; if security authentication is completed in the RFB protocol processing flow, determining that the communication connection between the computing device and the remote client is successfully established.

[0092] Specifically, the authentication password, stored in encrypted form, is read from the variable area protected by UEFI Secure Boot; the authentication password is parsed using a decryption function to obtain the plaintext password; the password hash value of the plaintext password is determined using a preset function; and if the password hash value matches the password hash value passed by the remote client, the authentication between the computing device and the remote client is confirmed to be successful.

[0093] The default function is EFI_HASH_PROTOCOL or other types of functions.

[0094] S108, according to the display attribute information corresponding to the display device, displays the interface configuration content through the remote client.

[0095] Among them, display attribute information refers to configuration parameters that describe the graphics output capabilities of the computing device, including but not limited to: horizontal pixel resolution, vertical pixel resolution, number of pixels per scan line, and pixel format.

[0096] In an optional embodiment, the display attribute information and the framebuffer address of the GOP instance are stored in a global state structure. Specifically, the global state structure takes the following form:

[0097] FrameBufferBase: The physical base address of the linear frame buffer (i.e., the frame buffer address);

[0098] HorizontalResolution: Horizontal pixel resolution;

[0099] VerticalResolution: Vertical pixel resolution;

[0100] PixelsPerScanLine: Number of pixels per scan line.

[0101] PixelFormat: Pixel format (e.g., PixelRedGreenBlueReserved8BitPerColor)

[0102] Specifically, the remote display control system includes a display detection and abstraction unit. This unit acquires display attribute information corresponding to the display device and stores the display attribute information and the frame buffer address of the GOP instance into a global state structure.

[0103] In the above embodiment, a protocol instance is first obtained by calling the standard interface of the UEFI firmware. If the protocol instance contains a graphics output protocol instance and its frame buffer address is valid, it is determined that the computing device is connected to the display device. This step requires no additional hardware; the firmware's own protocol query mechanism can automatically and accurately detect the display device's presence, laying a reliable foundation for subsequent remote control and avoiding invalid operations and resource waste caused by blindly establishing connections. Next, based on the connection request sent by the remote client, an authentication process is established between the computing device and the remote client. If authentication is successful, the UEFI firmware's interface configuration content matching the input event is obtained based on the graphics output protocol instance corresponding to the display device. This achieves real-time capture of remote input events at the firmware stage without relying on dedicated management hardware, allowing the remote client to interact with the UEFI firmware as if operating a local display device. Finally, based on the display attribute information of the display device itself, the content is configured through the remote client display interface, ensuring the consistency between the content presented by the remote client and the content of the local display device, which provides a seamless remote operation experience. At the same time, since all processing is completed based on UEFI firmware, the dependence on dedicated hardware is completely eliminated, and the cost of remote control is greatly reduced while maintaining out-of-band remote control capabilities.

[0104] In one embodiment, the method further includes: sending a request to the memory allocation interface of the UEFI boot service when the protocol instance does not include a graphics output protocol instance; the request is used to request physical memory; filling the frame buffer address of the physical memory into a structure allocated in the UEFI memory pool to obtain a virtual graphics output protocol instance; and determining that the computing device is connected to a display device by installing the virtual graphics output protocol instance onto the UEFI handle of the computing device.

[0105] The protocol instances do not include graphics output protocol instances, indicating that the computing device is a headless device, meaning the computing device is not connected to a local physical display. The frame buffer address refers to the physical base address of the linear frame buffer; the frame buffer address in physical memory is the same as the frame buffer address of the virtual graphics output protocol instance.

[0106] In an optional embodiment, the remote display control system includes a display detection and abstraction unit and a virtual GOP generation and frame buffer management unit. The method further includes: if the protocol instance does not include a graphics output protocol instance, invoking the display detection and abstraction unit to set a flag bit of the computing device; if the flag bit of the computing device is set to a second flag bit, invoking the virtual GOP generation and frame buffer management unit to send a request to the memory allocation interface of the UEFI boot service.

[0107] For example, the IsHeadless flag of the computing device is the second flag (such as TRUE).

[0108] Specifically, if the protocol instance does not include a graphics output protocol instance, a request is sent to the memory allocation interface (i.e., AllocatePages()) of the UEFI BootServices; the request is used to request physical memory. Physical memory is a contiguous block of physical memory capable of accommodating a preset resolution (e.g., 1024x768x4 bytes x 2 = 6MB). The framebuffer address of the physical memory can be set to VirtualFbBase.

[0109] Specifically, the framebuffer address of physical memory is filled into a structure allocated in the UEFI memory pool to obtain a virtual graphics output protocol instance. This includes: when filling the framebuffer address of physical memory into the EFI_GRAPHICS_OUTPUT_PROTOCOL structure allocated in the UEFI memory pool, the virtual graphics output protocol instance is obtained by filling the structure with member function pointers (QueryMode, SetMode, Blt) and Mode information structures. Here, the virtual graphics output protocol instance refers to a virtually constructed GOP instance, and the structure can take the following form:

[0110] Set Mode->MaxMode to 1;

[0111] Set Mode->Mode to 0;

[0112] Set Mode->Info->Version to 0;

[0113] Set Mode->Info->HorizontalResolution to the preset width;

[0114] Set Mode->Info->VerticalResolution to the preset height;

[0115] Mode->Info->PixelFormat is set to PixelBlueGreenRedReserved8BitPerColor;

[0116] Set Mode->FrameBufferBase to the physical memory frame buffer address VirtualFbBase;

[0117] Set Mode->FrameBufferSize to 6MB.

[0118] Specifically, the UEFI handle includes the current UEFI driver image handle (gImageHandle) or a newly created control handle. Therefore, after installing the virtual graphics output protocol instance onto the computing device's UEFI handle, a "valid" graphics output protocol instance exists in the UEFI kernel's protocol database. In other words, the UEFI Graphics Console Driver in the UEFI firmware can, during subsequent startup, correctly find the created virtual GOP through the standard LocateProtocol call and treat it as a standard physical display device, normally drawing all graphical interfaces such as BIOS Setup (UEFI firmware settings) and the boot menu on the memory area pointed to by VirtualFbBase.

[0119] In an optional embodiment, the method further includes storing the frame buffer address and display attribute information of the virtual GOP instance in a global state structure for subsequent calls.

[0120] In an optional embodiment, the graphics output protocol instance corresponding to the display device includes a virtual GOP instance. Then, based on the graphics output protocol instance corresponding to the display device, obtaining the interface configuration content of the UEFI firmware matching the input event includes: obtaining the interface configuration content of the UEFI firmware matching the input event based on the virtual GOP instance.

[0121] Based on the above, the remote display control system includes: a display detection and abstraction unit, and a virtual GOP generation and frame buffer management unit. Specifically, when the protocol instance does not include a graphics output protocol instance, the display detection and abstraction unit is invoked to set the flag bit of the computing device; when the flag bit of the computing device is set to the second flag bit, the CreateVirtualGop() interface of the virtual GOP generation and frame buffer management unit is invoked to generate a virtual GOP instance and the frame buffer address VirtualFbBase of the virtual GOP instance.

[0122] like Figure 2 The diagram illustrates a processing flow chart for a display detection and abstraction unit. When the driver entry function InitializeVncDriver is called, the processing flow of the display detection and abstraction unit is initiated. The output of the display detection and abstraction unit includes: the frame buffer address when the computing device has a screen (i.e., a connected display device), and the virtual GOP creation request (i.e., a request request) when the computing device does not have a screen.

[0123] Specifically, the display detection and abstraction unit enumerates protocol instances through the UEFI standard interface LocateProtocol. If the protocol instance includes a GOP instance (i.e., a GOP instance is found), on-screen path processing is performed, specifically: reading the frame buffer address Mode->FrameBufferBase of the GOP instance; reading the resolution information Mode->Info; reading the pixel format; and setting the flag IsHeadless=FALSE. If the protocol instance does not include a GOP instance (i.e., a GOP instance is not found), on-screen path processing is performed, i.e., setting the flag IsHeadless=TRUE and sending a virtual GOP creation request (CreateVirtualGop request, also known as an application request) to the virtual GOP generation and frame buffer management unit, while waiting for the virtual GOP instance and frame buffer address to be returned by the virtual GOP generation and frame buffer management unit. Finally, regardless of whether the computing device has a screen or not, the display detection and abstraction unit updates the frame buffer address of the corresponding GOP instance, as well as the resolution parameters (such as horizontal and vertical pixel resolution) and pixel parameters (such as the number of pixels per scan line and pixel format) in the global state structure.

[0124] like Figure 3The diagram illustrates a processing flow chart for a virtual GOP generation and frame buffer management unit. The input to this unit includes a CreateVirtualGop call from the display detection and abstraction unit. Specifically, when the computing device is screenless, the display detection and abstraction unit sends a virtual GOP creation request (CreateVirtualGop request, or request request) to the virtual GOP generation and frame buffer management unit to generate a virtual GOP instance and its frame buffer address VirtualFbBase (i.e., the frame buffer address of the virtual frame buffer). Therefore, the output of the virtual GOP generation and frame buffer management unit includes the virtual GOP instance and its frame buffer address VirtualFbBase.

[0125] Specifically, the virtual GOP generation and frame buffer management unit requests physical memory and constructs an EFI_GRAPHICS_OUTPUT_PROTOCOL structure to obtain a virtual GOP instance. By installing the virtual GOP instance onto the current driver image handle (gImageHandle) or a newly created control handle of the UEFI firmware of the computing device, a "valid" graphics output protocol instance exists in the UEFI kernel's protocol database. Then, by returning the virtual GOP instance and its frame buffer address to the display detection and abstraction unit, the global frame buffer pointer executes VirtualFbBase. Thus, during subsequent startup, the UEFI graphics stack can correctly find the created virtual GOP instance through the standard LocateProtocol call and treat it as a standard physical display device, thereby enabling the normal drawing of all graphical interfaces, such as BIOS Setup (UEFI firmware settings) and the boot menu, on the memory area pointed to by VirtualFbBase.

[0126] In one embodiment, the computing device communicates with a remote client via a physical network interface card (NIC), which includes a transmit buffer and a receive buffer. In response to an input event triggered by the remote client, based on a graphics output protocol instance corresponding to the display device, the interface configuration content of the UEFI firmware matching the input event is obtained. This includes: in response to the input event triggered by the remote client, reading an input message corresponding to the input event from the receive buffer; determining an event instance corresponding to the input event based on the input message; obtaining the interface configuration content of the UEFI firmware matching the input event based on the event instance and the graphics output protocol instance corresponding to the display device; the interface configuration content is cached in the transmit buffer, and the remote client reads the interface configuration content from the transmit buffer.

[0127] Input events include keyboard injection events and pointer injection events.

[0128] In an optional embodiment, the remote display control system further includes a UEFI input event injection unit. The method further includes: by invoking the UEFI input event injection unit, in response to an input event triggered by a remote client, reading an input message corresponding to the input event from a receive buffer; and determining an event instance corresponding to the input event based on the input message.

[0129] In the above embodiments, by responding to input events from remote clients, messages are read from the receiving buffer and event instances are determined. Then, the interface configuration content is obtained by combining the graphics output protocol instance and the content is cached in the sending buffer for remote clients to read. This realizes a complete link for remote input to trigger interface content updates, ensuring that remote clients can obtain UEFI firmware interface screens that match the input operations in real time. Since it does not rely on dedicated hardware and only uses UEFI firmware to realize closed-loop processing of remote input and interface feedback, hardware costs and remote control costs are reduced.

[0130] In one embodiment, obtaining the interface configuration content of the UEFI firmware matching the input event based on the event instance and the graphics output protocol instance corresponding to the display device includes: obtaining interface buffer content from the frame buffer address of the graphics output protocol instance corresponding to the display device; generating interface display content corresponding to the event instance through the graphics output protocol instance corresponding to the display device; and determining the interface configuration content of the UEFI firmware matching the input event based on the incremental content when there is incremental content between the interface buffer content and the interface display content.

[0131] Among them, the graphics output protocol instance corresponding to the display device includes the GOP instance that exists in the computing device itself, or the virtual GOP instance.

[0132] In an optional embodiment, determining whether incremental content exists between the interface buffer content and the interface display content includes: determining whether incremental content exists between the interface buffer content and the interface display content based on a dirty rectangle method. Specifically, both the interface buffer content and the interface display content are divided into a preset number of macroblocks of fixed size; according to the macroblock division order, if at least one piece of data in the interface buffer content within a corresponding macroblock is different from the data in the interface buffer content within the corresponding macroblock, then the corresponding macroblock is determined to be a dirty rectangle, and it is determined that there is incremental content between the interface buffer content and the interface display content corresponding to the dirty rectangle.

[0133] For example, the remote display control system also includes an asynchronous event scheduling and lifecycle control unit. The method further includes: invoking the asynchronous event scheduling and lifecycle control unit to determine, based on a dirty rectangle approach, whether there is incremental content between the interface buffer content and the interface display content.

[0134] Furthermore, the method of obtaining incremental content includes: determining the data of the interface buffer content within the corresponding macroblock and the difference data between the data of the interface buffer content within the corresponding macroblock as update data; and combining the various update data to obtain incremental content.

[0135] The number of data to be updated must be at least one. The preset number can be 20 or other values, and the macroblock size can be 16×16 pixels or other values.

[0136] In an optional embodiment, both the interface buffer content and the interface display content are divided into a preset number of macroblocks of fixed size. According to the macroblock division order, if the data of the interface buffer content in any corresponding macroblock is the same as the data of the interface buffer content in any corresponding macroblock, it is determined that there is no incremental content between the interface buffer content and the interface display content.

[0137] Furthermore, the method also includes: when there is no incremental content between the interface buffer content and the interface display content, returning to the step of responding to the input event triggered by the remote client. This avoids repeatedly sending the same interface content due to no screen change, reduces network bandwidth usage and firmware processing overhead, and allows the remote control system to transmit data only when the interface actually changes, thus improving overall operating efficiency.

[0138] In an optional embodiment, determining the interface configuration content of the UEFI firmware that matches the input event based on the incremental content includes: encapsulating the incremental content into content in the form of RFB protocol and performing data encoding to obtain the interface configuration content of the UEFI firmware that matches the input event.

[0139] For example, the remote display control system includes a secure RFB protocol engine unit, which obtains incremental content from the asynchronous event scheduling and lifecycle control unit; encapsulates the incremental content into RFB protocol format to obtain the interface configuration content of the UEFI firmware matching the input event; and stores the interface configuration content in the transmission buffer through the TCP / IP protocol stack.

[0140] Specifically, the secure RFB protocol engine unit is used to encapsulate incremental content into a FramebufferUpdate message of the RFB protocol. This message includes: message type code, number of dirty rectangles, rectangle header (X coordinate, Y coordinate, width, height, encoding type=Raw) for each dirty rectangle, and incremental content corresponding to the dirty rectangle.

[0141] In the above embodiments, by triggering an incremental comparison between the interface buffer content and the interface display content based on event instances, the incremental content is output as the interface configuration content only when there is a change. This achieves accurate capture and on-demand transmission of UEFI firmware interface changes, avoids invalid or duplicate interface data updates, and thus reduces the amount of data transmitted remotely and the processing overhead.

[0142] In one embodiment, the input event includes a keyboard injection event. Determining the event instance corresponding to the input event based on the input message includes: parsing the input message to obtain the key code corresponding to the keyboard injection event and the key state corresponding to the key code; if the key state is pressed, converting the key code into a target code matching the type of the key code; and filling the target code into the constructed first structure to obtain the event instance corresponding to the input event.

[0143] The input message refers to the KeyEvent message based on the RFB protocol (i.e., the RFB KeyEvent message). Key states include pressed and released. The first structure refers to the EFI_KEY_DATA structure.

[0144] For example, if the key code is a printable character, the key code is converted to the corresponding Unicode code to obtain the target code. Specifically, the Unicode code is filled into Key.UnicodeChar in the first structure to obtain the event instance. Printable characters include letters, numbers, symbols, etc.

[0145] For example, if the key code is a function key, it is converted into the corresponding scan code to obtain the target code. Specifically, the scan code is filled into Key.ScanCode in the first structure to obtain the event instance. Function keys include arrow keys, F1~F12, Home, Del, etc.

[0146] In the above embodiments, by parsing the input message to obtain the key code and key state, and only when the key state is pressed, the key code is converted into a code value conforming to the UEFI specification and filled into the structure, which can ensure that the behavior of remote keyboard input at the firmware level conforms to the UEFI specification for keyboard input, and improves the compatibility between remote input and the firmware's native input mechanism.

[0147] In one embodiment, the input event includes a pointer injection event, and the display attribute information includes the frame buffer resolution. Based on the input message, determining the event instance corresponding to the input event includes: parsing the input message to obtain the position coordinates and key mask corresponding to the pointer injection event; normalizing the position coordinates according to the frame buffer resolution and mapping the normalized position coordinates to absolute coordinates matching the UEFI firmware; and filling the button state and absolute coordinates corresponding to the key mask into a second structure to obtain the event instance corresponding to the input event.

[0148] The frame buffer resolution includes both horizontal and vertical pixel resolution. Position coordinates include X and Y coordinates.

[0149] Specifically, the EFI_ABSOLUTE_POINTER_PROTOCOL.GetState() function is called to read the button state and absolute coordinates corresponding to the key mask, and the read button state and absolute coordinates are filled into the second structure to obtain the event instance corresponding to the input event.

[0150] In the above embodiments, by parsing the position coordinates of the pointer injection event and the key mask, and normalizing them with the frame buffer resolution, the coordinates are mapped to the absolute coordinates matched by the UEFI firmware and filled into the second structure. This achieves the accurate conversion of remote pointer events to firmware native absolute input events, improving the accuracy and compatibility of remote control.

[0151] It is understandable that, since the computing device and the remote client interact with each other through the send buffer and receive buffer in the physical network card, and the data format of the send buffer and receive buffer is the RFB protocol format, the input message corresponding to the input event can be called an RFB input message.

[0152] In one embodiment, the remote display control system further includes a UEFI input event injection unit. The method further includes: determining the event instance corresponding to the input event by invoking the UEFI input event injection unit.

[0153] like Figure 4 As shown, a flowchart of a UEFI input event injection unit is provided. The UEFI input event injection unit can obtain corresponding event instances by adopting different event processing procedures according to different input events, wherein:

[0154] Upon receiving an RFB-based input message, if the message type is determined to be a KeyEvent message (message type 4), it indicates that the input event is a keyboard-injected event. The corresponding event instance is then determined through the keyboard event handling process. Alternatively, if the message type is determined to be a PointerEvent message (message type 5), it indicates that the input event is a pointer-injected event. The corresponding event instance is then determined through the pointer event handling process. Furthermore, scenario adaptation is performed on the computing device. In screenless mode (i.e., without a connected display device), in response to remote input from a remote client, the local physical input device events of the computing device are masked. The event instance (keyboard instance or pointer instance) is then injected into the firmware input queue through the UEFI firmware input protocol stack, simulating local physical input behavior. This allows the UEFI application layer to respond to remote operations without any modifications, achieving complete remote interaction and control in screenless mode.

[0155] In one embodiment, the remote display control system further includes: an asynchronous event scheduling and lifecycle control unit, such as... Figure 5 As shown, a flowchart of the asynchronous event scheduling and lifecycle control unit is provided, wherein:

[0156] After the driver entry function InitializeVncDriver is called, low-priority timer events are created during the initialization phase through asynchronous event scheduling and the lifecycle control unit. State machine scheduling is then performed based on these timer events, and the state machine includes different states. State A represents the listening process, i.e., listening for connection requests from remote clients via the TCP port; State B represents the authentication process, i.e., verifying the connection between the computing device and the remote client when a connection request is sent; State C includes different sub-steps C1, C2, and C3. Sub-step C1 represents network input processing, i.e., responding to input events triggered by remote clients by reading the corresponding RFB input message from the receive buffer; sub-step C2 represents dirty rectangle block detection, i.e., determining whether there is incremental content between the content buffered on the interface and the content displayed on the interface; sub-step C3 represents RFB frame update encapsulation and transmission, i.e., encapsulating the incremental content into an RFB message and buffering it in the physical network card's transmit buffer.

[0157] Furthermore, when a remote client exit event is received, state D is executed. State D represents the destruction state and is used to perform secure erasure and resource release to release occupied memory, network ports and protocol instances, ensuring that the computing device is restored to a clean state when there is no remote connection, thereby improving system security and resource utilization efficiency.

[0158] Understandably, through the aforementioned state machine design and the execution time limit for each callback, the VNC service becomes a very lightweight "coroutine" in the firmware. The Central Processing Unit (CPU) time slice is immediately returned to the UEFI main loop after each callback. When high-priority startup tasks (such as Peripheral Component Interconnect (PCI) enumeration and file system initialization) need to be executed, they will be preempted due to their low callback priority, minimizing the impact on the startup process time.

[0159] In one embodiment, the remote display control system further includes: a secure RFB protocol engine unit, such as... Figure 6 As shown, a flowchart of the processing flow of a secure RFB protocol engine unit is provided. The processing flow of the secure RFB protocol engine unit is divided into a network-side processing flow and an encoding-side processing flow, specifically:

[0160] The network-side processing flow includes: calling `LocateProtocol(&gEfiSimpleNetworkProtocolGuid, NULL, &Snp)` to obtain a Simple Network Protocol (SNP) instance, and building a lightweight TCP / Internet Protocol (IP) stack based on this. SNP is the underlying network interface protocol defined by the UEFI specification, providing raw Ethernet packet transmission and reception capabilities. The protocol stack is only used to implement: Address Resolution Protocol Request (ARP) requests and responses, IP packet transmission and reception, TCP port 5900 listening, and TCP three-way handshake / timeout retransmission / keep-alive / four-way handshake. Specifically, the VNC component or UEFI scheduler calls the network-side processing flow of the secure RFB protocol engine unit to listen for connection requests sent by remote clients through port 5900, and to implement protocol handshake and authentication between the computing device and the remote client.

[0161] The encoding-side processing flow includes: receiving dirty rectangle data (i.e. incremental content) from the asynchronous event scheduling and lifecycle control unit; encapsulating the incremental content into a FramebufferUpdate message in the form of RFB protocol, including message type code, number of rectangles, and rectangle headers corresponding to the dirty rectangles (such as X coordinate, Y coordinate, width, height, encoding type=Raw, etc.); then encoding the FramebufferUpdate message and sending it via non-blocking TCP, i.e., storing the encoded FramebufferUpdate message in the transmit buffer of the physical network card.

[0162] Based on the above, it can be seen that, as Figure 7 The diagram illustrates the structural block diagram of various units in a remote display control system. Specifically, the remote display control system includes a display detection and abstraction unit, a virtual GOP generation and frame buffer management unit, an asynchronous event scheduling and lifecycle control unit, a secure RFB protocol engine unit, and a UEFI input event injection unit.

[0163] The display detection and abstraction unit is a front-end detection module that determines whether to enable it based on the detection results (i.e., whether the computing device is screenless). The asynchronous event scheduling and lifecycle control unit is the core of the entire system's scheduling, driving the secure RFB protocol engine unit and the UEFI input event injection unit to work in a time-sharing manner through registered timer callback functions. The secure RFB protocol engine unit communicates with remote clients externally through the physical network card; the UEFI input event injection unit interfaces with the UEFI firmware input protocol stack. Additionally, the physical graphics card and native GOP instance provide the main frame buffer data source for the display detection and abstraction unit in screen-equipped scenarios.

[0164] Specifically, the display detection and abstraction unit is used to detect the existence status of the graphics output protocol of the computing device during the driver initialization phase, and establish a unified graphics data source abstraction accordingly; the virtual GOP generation and frame buffer management unit is used to create a virtual frame buffer (i.e., physical memory) in the system memory when the computing device is screenless, and construct a virtual GOP instance that conforms to the UEFI standard to deceive the UEFI graphics stack to complete the initialization.

[0165] Specifically, the asynchronous event scheduling and lifecycle control unit is the scheduling core of the remote display control system. It is responsible for creating and managing low-priority timer callback events to realize the asynchronous time-sharing execution of VNC service business; at the same time, it manages the startup, operation and secure destruction of services. Its working principle includes: driver entry and fast exit. That is, unlike the regular UEFI driver, the driver entry function InitializeVncDriver is executed only in sequence: calling the display detection and abstraction unit, (when the computing device is screenless) calling the virtual GOP generation and frame buffer management unit, initializing the network protocol stack of the secure RFB protocol engine unit, and calling the asynchronous event scheduling and lifecycle control unit to create timers. Therefore, all operations are lightweight initializations, and return EFI_SUCCESS immediately after execution, without any infinite loop logic such as while(1). This ensures that driver loading does not block subsequent DXE driver scheduling.

[0166] Specifically, the secure RFB protocol engine unit is used to implement simplified TCP / IP communication and RFB protocol server functions based on the UEFI network protocol stack, completing the secure encoded transmission of frame buffer data. Its working principle is as follows: It abstracts the underlying network, calling `LocateProtocol(&gEfiSimpleNetworkProtocolGuid, NULL, &Snp)` to obtain an SNP instance, and builds a lightweight TCP / IP protocol stack based on this. Thus, the protocol stack only implements: ARP request and response, IP packet sending and receiving, TCP three-way handshake / timeout retransmission / keep-alive / four-way handshake. The UEFI input event injection unit is used to parse and convert keyboard and mouse input messages from remote clients into UEFI standard input protocol events, injecting them into the firmware input protocol stack to achieve remote control of the firmware interface.

[0167] When the computing device has a screen, that is, when the computing device is connected to a display device, the display device can refer to the local physical display. Thus, the content can be read from the frame buffer address of the graphics output protocol instance through the physical graphics card or video memory and displayed on the local physical display.

[0168] The physical network interface card (NIC) used by the computing device, whether it has a screen or not, can be the same NIC or different NICs. For example, the physical NIC used when the computing device has no screen is the first physical NIC, and the physical NIC used when the computing device has a screen is the second physical NIC.

[0169] Based on the various units in the aforementioned remote display control system, after the computing device is powered on, the VNC component or UEFI scheduler can realize the remote display control process between the remote client and the computing device by calling the various units in the remote display control system.

[0170] like Figure 8 As shown, a flowchart is provided for remote display control based on various units in a remote display control system. Taking the application of this method to a VNC component or a UEFI scheduler as an example, the flowchart illustrates the following:

[0171] After the computing device is powered on, the UEFI firmware sequentially goes through the Security (SEC) phase, the Pre-EFI Initialization Phase (PEI), the DXE phase, and the Boot Device Selection (BDS) phase, ultimately booting the operating system. The BDS phase is responsible for enumerating boot options and booting the operating system in the UEFI boot process. The method provided in this application is embedded in the above phases, specifically as follows:

[0172] When the UEFI firmware reaches the DXE stage, the display detection and abstraction unit is invoked to obtain the protocol instances configured in the computing device. It is then determined whether a GOP instance is found, i.e., whether the protocol instances include a graphics output protocol instance. Specifically, if the protocol instances include a GOP instance and its frame buffer address is valid, the computing device is determined to be in a screen-enabled scenario, meaning it is connected to a display device. If the protocol instances do not include a GOP instance, the computing device is determined to be in a screenless scenario, meaning it is not connected to a display device. Further, when the computing device is in a screenless scenario, a virtual GOP instance and its frame buffer address are constructed by invoking the virtual GOP generation and frame buffer management unit. Subsequently, regardless of whether the computing device is in a screen-enabled or screenless scenario, information such as the frame buffer address, resolution, pixel format, and scene marker (screenless or screen-enabled) of the corresponding GOP instance is stored in a global state structure.

[0173] Further, the following steps are performed sequentially: The network-side processing flow of the secure RFB protocol engine is invoked to initialize the simplified TCP / IP network protocol stack; the asynchronous event scheduling and lifecycle control unit is invoked to create timer events and register the callback function VncTimerCallback. Through the callback function, the secure RFB protocol engine unit and the UEFI input event injection unit are invoked to perform listening, authentication, and service (i.e., VNC listening, VNC authentication, and VNC service) for the remote VNC client. Thus, through the above processing, after the VNC driver is loaded, EFI_SUCCESS is returned, and the UEFI firmware continues with subsequent driver scheduling in the DXE stage. When the UEFI firmware reaches the BDS stage, the device is enumerated to perform subsequent selection and load the operating system bootloader. Finally, when the remote VNC client is detected to have exited, the remote VNC client is destroyed (i.e., VNC destruction), thus completing the remote display control process based on the UEFI firmware.

[0174] based on Figure 8As illustrated in the embodiments, taking the UEFI scheduler as an example, the remote display control method provided in this application can be loaded through the UEFI scheduler during the DXE stage. Specifically, by calling the display detection and abstraction unit, if it is determined that the computing device has a screen, the GOP instance and its frame buffer address are obtained; if the computing device does not have a screen, the virtual GOP generation and frame buffer management unit is called to create a virtual GOP instance and its frame buffer address to complete unified display abstraction. Further, by calling the asynchronous event scheduling and lifecycle control unit to initialize the network SNP interface and the simplified TCP / IP stack, a TPL_CALLBACK level timer event is created and a callback function VncTimerCallback is registered. Thus, through the above settings, the UEFI main loop continues to execute subsequent DXE stage tasks and BDS stage tasks, with the timer triggering at a 100ms period. Among them, during the first few triggers of the callback function VncTimerCallback, it is in state A, performing TCP listening. If a remote client connection is detected and the handshake authentication is successful, it enters the service loop in state C. Each callback is executed sequentially: the input message corresponding to the input event is read from the physical network card's receive buffer; the UEFI input event injection unit is called to determine the event instance corresponding to the input event; the asynchronous event scheduling and lifecycle control unit is called to perform dirty rectangle detection; the secure RFB protocol engine unit is called to perform incremental data encoding; and the interface configuration content is stored in the physical network card's transmit buffer via the network. Thus, the interface configuration content can be displayed through a remote client according to the display attribute information corresponding to the display device. The UEFI boot process completes all operations such as POST self-test, PCI enumeration, hard disk identification, and system booting without interference. Furthermore, before booting the operating system, the UEFI firmware calls ExitBootServices, triggering the asynchronous event scheduling and lifecycle control unit to execute state D (i.e., destruction), thereby performing TCP session closure, memory clearing, and resource release for a safe exit.

[0175] In summary, such as Figure 9 As shown, a remote display control method based on UEFI firmware is provided. Taking the application of this method to a VNC component and a UEFI scheduler as an example, the method includes the following steps:

[0176] S901 obtains the protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device.

[0177] S902, if it is determined that the computing device is connected to the display device based on the protocol instance including the graphics output protocol instance and the frame buffer address of the graphics output protocol instance being a valid address, an authentication process between the computing device and the remote client is established based on the connection request sent by the remote client.

[0178] S903, upon successful authentication, responds to an input event triggered by a remote client by reading the input message corresponding to the input event from the receive buffer.

[0179] S904, Based on the input message, determine the event instance corresponding to the input event.

[0180] S905 retrieves the interface buffer content from the frame buffer address of the graphics output protocol instance corresponding to the display device.

[0181] S906 generates the interface display content corresponding to the event instance by using the graphics output protocol instance corresponding to the display device.

[0182] S907: In the case where there is incremental content between the interface buffer content and the interface display content, the interface configuration content of the UEFI firmware that matches the input event is determined based on the incremental content.

[0183] S908 displays the interface configuration content through a remote client according to the display attribute information corresponding to the display device.

[0184] The specific content of S901 to S908 can be found in the aforementioned description and will not be repeated here.

[0185] In summary, this application provides a remote display control method for UEFI environments that is independent of the operating system, compatible with both devices with and without local displays (headless devices), and does not affect the normal firmware boot process. Specifically, the method provided in this application integrates a VNC server conforming to the RFB protocol entirely within the UEFI firmware in a purely software manner, without relying on any dedicated management hardware or independent BMC chip, thus achieving remote graphical control functionality. Furthermore, it runs a long-term VNC network service at the firmware layer without disrupting the original UEFI execution flow, achieving non-blocking graphical remote control. Moreover, by constructing a GOP unified display abstraction mechanism, it enables fully graphical remote control of both display-equipped and headless devices at the firmware layer.

[0186] Therefore, the method provided in this application can improve operational capabilities. By loading the method provided in this application during the UEFI DXE stage and continuously running it until the operating system boots, it achieves graphical remote control of the entire process from device store to operating system loading. Administrators can remotely perform operations such as BIOS configuration modification, boot option selection, firmware diagnosis, and system installation in any scenario, including operating system crashes, boot corruption, or system not being installed. In addition, the method of this application can also overcome the graphical blind spot of headless devices. By dynamically creating a virtual GOP and using the UEFI standard protocol installation mechanism, it tricks the firmware graphics stack into initializing in pure memory, enabling computing devices without GPUs or physical display interfaces to have a remotely accessible BIOS graphical interface for the first time, filling the technical gap in firmware-level graphical remote control of headless devices. Furthermore, the method of this application also features universality and low-cost remote control. The method provided in this application is a pure firmware software solution, requiring no additional dedicated BMC chips or other hardware, and directly utilizes the device's existing network card and standard UEFI firmware environment. At the same time, the server is compatible with the RFB protocol and any standard VNC Viewer client, making deployment convenient and highly versatile.

[0187] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0188] Based on the same inventive concept, this application also provides a UEFI firmware-based remote display control device for implementing the aforementioned UEFI firmware-based remote display control method. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations in one or more embodiments of the UEFI firmware-based remote display control device provided below can be found in the limitations of the UEFI firmware-based remote display control method described above, and will not be repeated here.

[0189] In one exemplary embodiment, such as Figure 10As shown, a remote display control device based on UEFI firmware is provided, comprising: an acquisition module 1001, a first processing module 1002, a second processing module 1003, and a third processing module 1004, wherein:

[0190] The acquisition module 1001 is used to acquire the protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device; the first processing module 1002 is used to establish an authentication process between the computing device and the remote client based on the connection request sent by the remote client, when it is determined that the computing device is connected to the display device based on the protocol instance including a graphics output protocol instance and the frame buffer address of the graphics output protocol instance being a valid address; the second processing module 1003 is used to, in response to an input event triggered by the remote client when authentication is successful, obtain the interface configuration content of the UEFI firmware matching the input event based on the graphics output protocol instance corresponding to the display device; the third processing module 1004 is used to display the interface configuration content through the remote client according to the display attribute information corresponding to the display device.

[0191] In one embodiment, the first processing module 1002 is further configured to: send a request to the memory allocation interface of the UEFI boot service if the protocol instance does not include a graphics output protocol instance; the request is used to request physical memory; fill the frame buffer address of the physical memory into a structure allocated in the UEFI memory pool to obtain a virtual graphics output protocol instance; and determine that the computing device is connected to a display device by installing the virtual graphics output protocol instance onto the UEFI handle of the computing device.

[0192] In one embodiment, the computing device communicates with the remote client via a physical network interface card (NIC), the NIC including a transmit buffer and a receive buffer; the second processing module 1003 is further configured to: in response to an input event triggered by the remote client, read an input message corresponding to the input event from the receive buffer; determine an event instance corresponding to the input event based on the input message; obtain the interface configuration content of the UEFI firmware matching the input event based on the event instance and the graphics output protocol instance corresponding to the display device; the interface configuration content is cached in the transmit buffer, and the remote client reads the interface configuration content from the transmit buffer.

[0193] In one embodiment, the second processing module 1003 is further configured to: obtain interface buffer content from the frame buffer address of the graphics output protocol instance corresponding to the display device; generate interface display content corresponding to the event instance through the graphics output protocol instance corresponding to the display device; and, in the case that there is incremental content between the interface buffer content and the interface display content, determine the interface configuration content of the UEFI firmware that matches the input event based on the incremental content.

[0194] In one embodiment, the input event includes a keyboard injection event; the second processing module 1003 is further configured to: parse the input message to obtain the key code corresponding to the keyboard injection event and the key state corresponding to the key code; when the key state is a pressed state, convert the key code into a target code that matches the type of the key code; fill the target code into the constructed first structure to obtain the event instance corresponding to the input event.

[0195] In one embodiment, the input event includes a pointer injection event, and the display attribute information includes the frame buffer resolution; the second processing module 1003 is further configured to: parse the input message to obtain the position coordinates and key mask corresponding to the pointer injection event; normalize the position coordinates according to the frame buffer resolution, and map the normalized position coordinates to absolute coordinates matching the UEFI firmware; fill the button state corresponding to the key mask and the absolute coordinates into a second structure to obtain the event instance corresponding to the input event.

[0196] The modules in the aforementioned UEFI firmware-based remote display and control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0197] In one exemplary embodiment, a computer device is provided, which may be a remote display and control system, and its internal structure diagram may be as follows: Figure 11As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores XX data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a remote display control method based on UEFI firmware.

[0198] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0199] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0200] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0201] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0202] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0203] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0204] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0205] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A remote display control method based on UEFI firmware, characterized in that, The method includes: Obtain the protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device; If it is determined that the computing device is connected to the display device based on the fact that the protocol instance includes a graphics output protocol instance and the frame buffer address of the graphics output protocol instance is a valid address, an authentication process between the computing device and the remote client is established based on the connection request sent by the remote client. If authentication is successful, in response to an input event triggered by the remote client, the interface configuration content of the UEFI firmware that matches the input event is obtained based on the graphics output protocol instance corresponding to the display device; The interface configuration content is displayed through the remote client according to the display attribute information corresponding to the display device.

2. The method according to claim 1, characterized in that, The method further includes: If the protocol instance does not include a graphics output protocol instance, a request is sent to the memory allocation interface of the UEFI boot service; the request is used to request physical memory. The frame buffer address of the physical memory is filled into the structure allocated in the UEFI memory pool to obtain a virtual graphics output protocol instance; By installing the virtual graphics output protocol instance onto the UEFI handle of the computing device, it is determined that the computing device is connected to the display device.

3. The method according to claim 1 or 2, characterized in that, The computing device communicates with the remote client via a physical network interface card (NIC), which includes a transmit buffer and a receive buffer. In response to an input event triggered by the remote client, the interface configuration content of the UEFI firmware matching the input event is obtained based on the graphics output protocol instance corresponding to the display device, including: In response to an input event triggered by the remote client, the input message corresponding to the input event is read from the receive buffer; Based on the input message, determine the event instance corresponding to the input event; Based on the event instance and the graphics output protocol instance corresponding to the display device, the interface configuration content of the UEFI firmware matching the input event is obtained; the interface configuration content is cached in the transmission buffer, and the remote client reads the interface configuration content from the transmission buffer.

4. The method according to claim 3, characterized in that, The step of obtaining the interface configuration content of the UEFI firmware matching the input event based on the event instance and the graphics output protocol instance corresponding to the display device includes: Obtain the interface buffer content from the frame buffer address of the graphics output protocol instance corresponding to the display device; The interface display content corresponding to the event instance is generated using the graphics output protocol instance corresponding to the display device; In the case where there is incremental content between the interface buffer content and the interface display content, the interface configuration content of the UEFI firmware that matches the input event is determined based on the incremental content.

5. The method according to claim 3, characterized in that, The input event includes a keyboard injection event; determining the event instance corresponding to the input event based on the input message includes: Parse the input message to obtain the key code corresponding to the keyboard injection event and the key state corresponding to the key code; When the key is in the pressed state, the key code is converted into a target code that matches the type of the key code; The target code is filled into the constructed first structure to obtain the event instance corresponding to the input event.

6. The method according to claim 3, characterized in that, The input event includes a pointer injection event, and the display attribute information includes the frame buffer resolution; determining the event instance corresponding to the input event based on the input message includes: Parse the input message to obtain the position coordinates and key mask corresponding to the pointer injection event; When the position coordinates are normalized according to the frame buffer resolution, the normalized position coordinates are mapped to absolute coordinates that match the UEFI firmware. The button state corresponding to the key mask and the absolute coordinates are filled into the second structure to obtain the event instance corresponding to the input event.

7. A remote display control device based on UEFI firmware, characterized in that, The device includes: The acquisition module is used to acquire the protocol instance configured in the computing device through the standard interface of the UEFI firmware in the computing device; The first processing module is used to establish an authentication process between the computing device and the remote client based on a connection request sent by the remote client, when it is determined that the computing device has been connected to the display device, based on the fact that the protocol instance includes a graphics output protocol instance and the frame buffer address of the graphics output protocol instance is a valid address. The second processing module is used to, upon successful authentication, respond to an input event triggered by the remote client and obtain the interface configuration content of the UEFI firmware that matches the input event based on the graphics output protocol instance corresponding to the display device. The third processing module is used to display the interface configuration content through the remote client according to the display attribute information corresponding to the display device.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images