A picture display method, device and storage medium

Abstract

The application discloses a picture display method and device and a storage medium, relates to the field of remote control, and can effectively solve the problems of mouse image following, dragging and residual image on the display of a control device, and improves user experience. The method comprises the following steps: a control device receives non-mouse layer data of a first picture of a computing device sent by the computing device, wherein the non-mouse layer data is data separated from the first picture by the computing device; and the control device draws and displays a first remote picture based on the non-mouse layer data and first local mouse data of the control device.

Description

Technical Field

[0001] This application relates to the field of remote control, and more particularly to a screen display method, device and storage medium. Background Technology

[0002] With the development of data centers, users have an increasing demand for remote control and management of computing equipment. Keyboard-video-mouse (KVM) technology is a technology used for remote control of computing devices. Its main function is to control and manage one or more computing devices through a set of input / output devices (keyboard, mouse, and monitor). When the control device uses KVM technology to remotely control and manage computing devices, the control device displays the mouse image on both the control device and the computing device. However, when the mouse moves too fast or the network connection is poor, undesirable effects such as mouse tracking, dragging, and ghosting may appear on the control device's monitor, resulting in a poor user experience. Summary of the Invention

[0003] This application provides a screen display method, device, and storage medium, which can effectively solve problems such as mouse image following, dragging, and ghosting on the monitor of the control device, thereby improving the user experience.

[0004] To achieve the above technical objectives, this application adopts the following technical solution:

[0005] In a first aspect, embodiments of this application provide a screen display method, which is applied to a control device. The method includes: receiving non-mouse layer data of a first screen of a computing device sent by a computing device; wherein the non-mouse layer data is data separated from the first screen by the computing device; and drawing and displaying a first remote screen based on the non-mouse layer data and the first local mouse data of the control device. The first remote screen is a remote screen displayed by the control device based on the first screen of the remotely controlled computing device.

[0006] It is understandable that in this first remote screen, since the first local mouse data comes from the control device and the non-mouse layer data comes from the computing device, only one mouse image is displayed on the screen. This method effectively solves the undesirable effects such as mouse following, dragging and ghosting caused by the overlapping display of mouse images of the control device and the computing device in the traditional method.

[0007] In one possible implementation, before receiving the non-mouse layer data of the first screen of the computing device sent by the computing device, the method further includes: sending the first mouse position information of the computing device to the computing device, wherein the mouse image in the first screen is drawn based on the first mouse position information.

[0008] It is understandable that, since the computing device is remotely controlled by the controlled device, the position of the mouse image on the screen of the computing device can be controlled by the controlled device, and this method enables the controlled device to perform control functions.

[0009] In another possible implementation, before sending the first mouse position information to the computing device, the method further includes: determining the first mouse position of the control device; wherein the first mouse position of the control device is the mouse position indicated by the first local mouse data; and determining the first mouse position information of the computing device based on the first mouse position of the control device and the resolution of the computing device.

[0010] It is understandable that by obtaining the resolution of the computing device to determine the first mouse position information of the computing device, the position information of the first position calculated by the control device can be more accurate.

[0011] In another possible implementation, after drawing and displaying the first remote screen, the method further includes: receiving a first operation on the first remote screen; wherein the first operation is used to display a second remote screen; the first operation indicates a second mouse position of the control device; in response to the first operation, sending second mouse position information of the computing device to the computing device; the second mouse position information is used to update the first screen of the computing device to the second screen, wherein the mouse image in the second screen is drawn based on the second mouse position information.

[0012] It is understandable that, since the first remote screen is a remote screen of the first screen, when the first operation is performed on the first remote screen, the first remote screen switches to the second remote screen, and the first screen also switches to the second screen at the same time, so that the computing device can complete the precise movement of the mouse image under the control of the control device.

[0013] In another possible implementation, the method further includes: in response to the first operation, receiving non-mouse layer data of a second screen sent by the computing device; wherein the non-mouse layer data is data separated from the second screen by the computing device; drawing and displaying a second remote screen based on the non-mouse layer data of the second screen and the second local mouse data of the control device; wherein the second remote screen is a remote screen displayed by the control device based on the second screen of the remotely controlled computing device; and the second mouse data of the control device is data indicating the second mouse position of the control device.

[0014] It is understandable that since the non-mouse layer data of the remote screen displayed by the control device comes from the computing device, when the first screen switches to the second screen under the control of the first operation, the computing device will also send the updated non-mouse layer data to the control device, so that the control device switches to the second remote screen accordingly.

[0015] In another possible implementation, the time interval between receiving the first operation and receiving the non-mouse layer data of the second screen is less than a preset threshold.

[0016] Understandably, the preset threshold value is a very small value. In actual implementation, since network processing speed is very fast, this value does not need to be set in advance. The most recent first operation and non-mouse layer data can be selected.

[0017] Secondly, embodiments of this application provide a screen display method applied to a computing device. The method includes: updating current image data based on first mouse position information of the computing device to obtain first image data; using the first image data to draw a first screen; separating non-mouse layer data from the first image data; and sending the non-mouse layer data to a control device; wherein the non-mouse layer data is used to instruct the control device to display a first remote screen; and the first remote screen is drawn based on the first local mouse data of the control device and the non-mouse layer data of the computing device.

[0018] Understandably, the computing device separates the non-mouse layer data from the first image data and sends this non-mouse layer data to the control device, while not sending the mouse layer data to the control device. This ensures that the first remote screen drawn by the control device does not contain the mouse image from the computing device, and only displays the mouse image from the control device's local display. This method effectively solves the undesirable effects such as mouse following, dragging, and ghosting caused by the overlapping display of mouse images from the control device and the computing device in traditional methods.

[0019] In one possible implementation, before drawing the first screen based on the first mouse position information of the computing device, the method further includes: receiving the first mouse position information of the computing device sent by the control device, wherein the first mouse position information is determined by the control device based on the first mouse position of the control device and the resolution of the computing device.

[0020] It is understandable that, since the computing device is remotely controlled by the control device, the computing device receives the first mouse position information sent by the control device in order to complete the precise control of the mouse position on the computing device by the control device.

[0021] In another possible implementation, after separating the non-mouse layer data from the first image data, the method further includes storing the first image data in a first storage space and storing the non-mouse layer data in a second storage space.

[0022] It is understandable that the first and second storage spaces are cache spaces used to temporarily store the first image data and non-mouse layer data. The computing device stores this image data and non-mouse layer data separately, so that when retrieving these two types of data subsequently, they can be retrieved from different storage spaces, avoiding confusion and improving retrieval efficiency.

[0023] In another possible implementation, before sending the non-mouse layer data to the control device, the method further includes: obtaining the non-mouse layer data from a second storage space.

[0024] Understandably, since the non-mouse layer data is temporarily stored in the second storage space in the above method, the computing device retrieves the data from the second storage space when sending the data to the control device. Because the receiving rate of the control device and the sending rate of the computing device may differ, storing the data in the cache space can resolve the contradiction of inconsistent communication rates between the control device and the computing device.

[0025] In another possible implementation, the above method further includes: obtaining first image data from a first storage space, and drawing and displaying a first screen based on the first image data.

[0026] It is understandable that the first image data includes mouse data and non-mouse data, with the mouse data provided by the control device to enable the control device to remotely control the computing device.

[0027] Thirdly, embodiments of this application provide a screen display device, wherein the screen display device is applied to various modules of the screen display method of the first aspect or any possible implementation of the first aspect; or, the screen display device is applied to various modules of the screen display method of the second aspect or any possible implementation of the second aspect.

[0028] For example, the aforementioned screen display device may include, but is not limited to, devices such as servers, tablets, desktops, laptops, and netbooks.

[0029] Fourthly, embodiments of this application provide a screen display device, including a memory and a processor. The memory and the processor are coupled; the memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the screen display device performs a screen display method as described in the first aspect and any possible implementation thereof.

[0030] Fifthly, this application provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on a display device, the display device performs the method as described in the first aspect and any possible implementation thereof; or, when the computer instructions are executed on the display device, the display device performs a display method as described in the second aspect and any possible implementation thereof.

[0031] Sixthly, this application provides a computer program product comprising computer instructions. When the computer instructions are executed on a display device, they cause the display device to perform the method as described in the first aspect and any possible implementation thereof; or, when the computer instructions are executed on the display device, they cause the display device to perform a display method as described in the second aspect and any possible implementation thereof.

[0032] For a detailed description of aspects three through six and their various implementations in this application, please refer to the detailed description in aspect one or two and their various implementations; and for a detailed analysis of the beneficial effects of aspects three through six and their various implementations in aspect one or two and their various implementations, please refer to the beneficial effect analysis in aspect one or two and their various implementations, which will not be repeated here.

[0033] These or other aspects of this application will become more readily apparent in the following description. Attached Figure Description

[0034] Figure 1 A flowchart illustrating the mouse synchronization effect in a control device and a computing device, as proposed in an embodiment of the application;

[0035] Figure 2 This is a schematic diagram of an implementation environment related to a screen display method provided in an embodiment of this application;

[0036] Figure 3 A flowchart of a screen display method provided in an embodiment of this application;

[0037] Figure 4 A flowchart illustrating another screen display method provided in this application embodiment;

[0038] Figure 5 A schematic diagram of a mouse image on a first remote screen provided in an embodiment of this application;

[0039] Figure 6 A flowchart illustrating another screen display method provided in this application embodiment;

[0040] Figure 7 A flowchart illustrating another screen display method provided in this application embodiment;

[0041] Figure 8 This application provides a diagram illustrating the processing of first image data by each chip in a computing device.

[0042] Figure 9 This is a schematic diagram of the structure of a screen display device provided in an embodiment of this application;

[0043] Figure 10 This is a schematic diagram of another screen display device provided in an embodiment of this application. Detailed Implementation

[0044] The terms “first,” “second,” and “third,” etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with “first,” “second,” or “third,” etc., may explicitly or implicitly include one or more of that feature.

[0045] With the development of data centers, the demand for remote control and management of data center computing equipment is increasing. When using KVM technology to remotely control and manage computing equipment, the control device uses this technology to overlay the mouse images on both the control device and the computing device to achieve synchronized mouse operation. However, when the mouse moves too fast or the network connection is poor, undesirable effects such as mouse image following, dragging, and ghosting may appear on the control device's monitor, resulting in a poor user experience.

[0046] In one example, such as Figure 1 As shown, Figure 1 A flowchart illustrating the mouse synchronization effect between a control device and a computing device in related technologies is shown. The process includes steps 1-6: 1. The control device acquires the target position of the mouse image locally (i.e., where the control device is located) and calculates the mouse image position on the computing device. 2. The computing device sends the mouse image position information to the computing device. 3. Upon receiving the mouse image position information, the computing device updates the image data. This image data is used to draw a screen containing both mouse and non-mouse images. The position of the mouse image on the screen is determined based on the target position information. 4. The computing device updates the screen using the updated image data. 5. The computing device sends the updated image data to the control device. 6. Based on the image data and the target position of the local mouse, the control device displays the local mouse image and the updated screen from the computing device on the screen.

[0047] As described above, the control device's monitor displays both the local mouse image and the computing device's screen, while the computing device's screen also contains the computing device's mouse image. Therefore, the control device's monitor displays two mouse images. These two mouse images are positioned at the same coordinates, resulting in them overlapping on the control device's monitor. If the network connection between the control device and the computing device is poor, the image data transmitted from the computing device to the control device will be delayed, potentially leading to undesirable effects such as mouse tracking, dragging, and ghosting.

[0048] Based on this, this application provides a screen display method. In this method, a computing device separates non-mouse layer data from a first screen and sends the non-mouse layer data to a control device. The control device draws and displays a first remote screen based on the non-mouse layer data sent by the computing device and the first local mouse data of the control device. In this first remote screen, since the first local mouse data comes from the control device and the non-mouse layer data comes from the computing device, only one mouse image is displayed in the first remote screen. This method effectively solves the undesirable effects such as mouse following, dragging, and ghosting caused by the overlapping display of mouse images of the control device and the computing device in traditional methods, thus improving the user experience.

[0049] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0050] Please refer to Figure 2 This diagram illustrates an implementation environment related to the screen display method provided in this application. Figure 2 As shown, the implementation environment may include: control device 100 and computing device 200.

[0051] The control device 100 includes a display 110 and an input device 120; the computing device 200 includes a display 210.

[0052] Control device 100 includes a central processing unit (CPU) 101, a baseboard management controller (BMC) chip 102, a graphics processing unit (GPU) 103, a digital-to-analog converter (DAC) 104, a universal serial bus (USB) 105, and a video graphics array (VGA) 106. Computing device 200 includes a CPU 201, a BMC chip 202, a GPU 203, a DAC 204, a video compression module 205, a USB 206, and a VGA 207.

[0053] The control device 100 communicates with the computing device 200 via a network connection, and the control device 100 can remotely control the computing device 200.

[0054] For example, the control device 100 may include, but is not limited to, servers, tablets, desktops, laptops, netbooks, etc.

[0055] For example, computing device 200 may include, but is not limited to, servers, tablets, desktops, laptops, netbooks, etc.

[0056] The display 110 is used to display the remote screen shown on the display 210 when the control device 100 remotely controls the computing device 200. The display 110 and the control device 110 can be integrated or set up independently, and this application embodiment does not limit this.

[0057] The display 210 is used to display the screen output by the computing device 200. The display 210 and the computing device 200 can be integrated or set up independently; this embodiment does not limit this.

[0058] Input device 120 is used to control the mouse image of control device 100 under user control.

[0059] For example, input device 120 may include, but is not limited to, devices such as mouse, keyboard and touchpad.

[0060] CPU 101 and CPU 201 are the computation and control cores of control device 100 and computing device 200, respectively, and are the final execution units for information processing and program execution. In this embodiment, CPU 101 is used to receive a first operation input from input device 120 and send the second mouse position information indicated by the first operation to BMC 201 of computing device 200 through BMC; at the same time, it is also used to drive GPU to display a first remote screen and a second remote screen.

[0061] BMC chip 102 and BMC chip 202 are remote management chips for control device 100 and computing device 200, respectively. These management chips can collect various information from their respective devices and provide it to upper-layer network management software. In this embodiment, BMC chip 102 enables control device 100 to remotely control computing device 200 by running a KVM program. Simultaneously, control device 100 transmits mouse image mapping location information and non-mouse layer data to BMC chip 202 of computing device 102.

[0062] When the control device 100 remotely controls the computing device 200 through the BMC chip 102, it uses KVM technology.

[0063] The KVM program is a program that runs in the BMC chip 102. KVM technology is a technology that provides motherboard-level access and supports multi-platform servers and serial devices. The control device 100 can access and control the computing device 200 through the KVM port provided by the BMC chip 102.

[0064] In some embodiments, the BMC chip has different names in different computing devices. For example, the BMC chip can be called an out-of-band controller, which is an out-of-band processor independent of the central processing unit (CPU). The out-of-band controller may include a monitoring and management unit outside the computer device, a management system in a management chip outside the processor, a baseboard management controller (BMC), a system management mode (SMM), etc.

[0065] GPU 103 and GPU 203 are the graphics processors of the control device 100 and the computing device 200, respectively, and are microprocessors used for image and graphics-related operations. In this embodiment, GPU 103 can be used to drive the display 110 connected to the control device 100 to display a first remote image and a second remote image; GPU 203 can be used to separate the image data of the computing device 200 and drive the display 210 connected to the computing device 200 to display the first image and the second image.

[0066] DAC104 and DAC204 are chips that convert digital signals into analog signals. In this embodiment, GPU103 converts image data using DAC104 and sends it to display 110 to display a first remote image and a second remote image; GPU203 converts image data using DAC204 and sends it to display 210 to display the first image and the second image.

[0067] The video compression module 205 is used to compress the image data of the computing device 200 and then send it to the BMC chip 102 of the control device 100 via the BMC chip 202.

[0068] USB105 is used for communication and data transmission between BMC chip 102 and CPU 101, and USB206 is used for communication and data transmission between BMC202 and CPU201.

[0069] VGA106 is used by GPU103 to send image data to display 110; VGA207 is used by GPU203 to send image data to display 210.

[0070] The control device 100 also includes a memory 107, and the computing device 200 also includes a memory 208.

[0071] Memory 107 and memory 208 include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or optical memory.

[0072] Optionally, CPU 101 implements the screen display method provided in this application embodiment by reading instructions stored in memory 107 or CPU 201 by reading instructions stored in memory 208; alternatively, CPU 101 or CPU 201 implements the screen display method provided in this application embodiment by internally stored instructions. When CPU 101 implements the screen display method in the above embodiment by reading instructions stored in memory 107 or CPU 201 by reading instructions stored in memory 208, the memory 107 or memory 208 stores instructions for implementing the screen display method provided in this application embodiment.

[0073] The BMC chip 102, GPU 103, and DAC 104 described above can be integrated together or set up independently; the BMC chip 202, GPU 203, DAC 204, and video compression module 205 can be integrated together or set up independently. This application embodiment does not limit the independent or integrated configuration of the above chips.

[0074] The following describes a screen display method provided by an embodiment of this application:

[0075] Please refer to Figure 3 This is a flowchart illustrating a screen display method provided in an embodiment of this application. Figure 3 As shown, the method may include S101-S108.

[0076] S101: The control device determines the first mouse position information of the computing device based on the first mouse position of the control device.

[0077] The first mouse position of the control device is determined by the control device based on the display position of the current mouse image. The data for drawing and displaying the mouse image of the control device at the first mouse position is called the first local mouse data.

[0078] Since the computing device is a device that is remotely controlled by the control device, the mouse position of the computing device is determined based on the mouse position of the control device.

[0079] The initial mouse position of the control device can be represented in screen coordinates. For example, the screen coordinates of the initial mouse position of the control device are (x1, y1).

[0080] Optionally, the initial mouse position information of the computing device can be calculated based on the resolution of the control device and the resolution of the computing device. Specific calculation steps may include steps 1-2.

[0081] Step 1: The control device obtains the resolution of the control device and calculates the resolution of the calculation device.

[0082] For example, a method for the control device to obtain the resolution of the computing device could be as follows: the control device sends an instruction to the computing device to obtain the resolution. After receiving the instruction, the computing device sends the resolution information back to the control device.

[0083] Step 2: The control device determines the first mouse position information of the computing device based on the resolution of the control device, the resolution of the computing device, and the first mouse position of the control device.

[0084] The first mouse position of the control device and the first mouse position of the calculation device can be represented by screen coordinates.

[0085] When the resolution of the computing device is the same as that of the control device, the position indicated by the first mouse position information of the computing device is the same as that of the first mouse position of the control device. For example, if the screen coordinates of the first mouse position of the control device are (x1, y1), then the screen coordinates included in the first mouse position information of the computing device are (x1, y1).

[0086] When the resolution of the computing device differs from that of the control device, the first mouse position information of the computing device needs to be determined based on the resolution of the control device, the resolution of the computing device, and the first mouse position of the control device. For example, if the resolution of the control device is a*b, the resolution of the computing device is c*d, and the screen coordinates of the first mouse position of the control device are (x1, y1), then the screen coordinates included in the first mouse position information of the computing device are (x1 / a*c, y1 / b*d).

[0087] By obtaining the resolution of the computing device, the first mouse position information of the computing device is determined, making the first mouse position information of the computing device calculated by the control device more accurate.

[0088] S102: The control device sends the first mouse position information of the computing device to the computing device.

[0089] The first mouse position information of the computing device is used to draw the mouse image in the first screen.

[0090] The first screen of a computing device is the screen displayed after the current screen on the computing device. The first screen can be the screen displayed when the computing device is initially connected to the control device, or it can be any screen displayed during the process of the computing device being remotely controlled by the control device.

[0091] Correspondingly, the computing device receives the first mouse position information of the computing device sent by the aforementioned control device.

[0092] S103: The computing device updates the current image data based on the first mouse position information to obtain the first image data.

[0093] The current image data is the image data contained in the frame currently displayed on the computing device. This currently displayed frame is the frame preceding the first frame.

[0094] The first image data is the data used to draw the first screen of the computing device. The first image data includes mouse data and non-mouse data of the computing device.

[0095] Non-mouse data is the data used to draw the non-mouse image of the first screen, while mouse data is the data used to draw the mouse image of the first screen.

[0096] In one example, the screen coordinates of the mouse position in the currently displayed image are (x0, y0), meaning the screen coordinates of the mouse position included in the current image data are (x0, y0). If the screen coordinates of the mouse position included in the received first mouse position information are (x1 / a*c, y1 / b*d), then the screen coordinates of the mouse position included in the current image data are updated from (x0, y0) to (x1 / a*c, y1 / b*d), meaning the screen coordinates of the mouse position in the obtained first image data are (x1 / a*c, y1 / b*d).

[0097] S104: The computing device separates the non-mouse layer data from the first image data.

[0098] Since the first image data contains mouse data and non-mouse data, the computing device separates the mouse data and non-mouse data in the first image data. The separated non-mouse data can be drawn as a separate image layer, and the separated non-mouse data can be called non-mouse layer data.

[0099] In one example, the computing device can label mouse data and non-mouse data in the first image data with different labels, for example, labeling mouse data as A and non-mouse data as B. When the computing device separates the first image data, it draws the data labeled B into a separate image layer, which is denoted as the non-mouse layer data.

[0100] By using the method of labeling different data in the above example, non-mouse layer data can be quickly separated from the first image data.

[0101] Optionally, after the computing device separates the non-mouse layer data from the first image data, it stores the first image data in the first storage space and the non-mouse layer data in the second storage space.

[0102] The first and second storage spaces are cache spaces used to temporarily store the first image data and non-mouse layer data.

[0103] In one example, mouse layer data is labeled A, and non-mouse layer data is labeled B. Data labeled A and B is stored in the first storage space, and data labeled B is stored in the second storage space.

[0104] In the aforementioned optional methods, after the computing device separates the non-mouse layer data from the first image data, it stores the first image data and the non-mouse layer data separately. When retrieving these two types of data subsequently, they can be retrieved from different storage spaces, avoiding confusion and improving retrieval efficiency. Simultaneously, since the receiving rate of the control device and the sending rate of the computing device may be inconsistent, storing the data in a cache space can resolve the contradiction of inconsistent communication rates between the control device and the computing device.

[0105] S105 (optional): The computing device draws and displays a first screen based on the first image data.

[0106] Optionally, the computing device obtains first image data from the first storage space, and draws and displays the first image based on the first image data.

[0107] S106: The computing device sends the non-mouse layer data of the first screen to the control device.

[0108] Among them, the non-mouse layer data is used to control the device to display the first remote screen; the first remote screen is drawn based on the first local mouse data of the control device and the non-mouse layer data of the computing device.

[0109] The computing device can periodically send non-mouse layer data to the control device, or send non-mouse layer data to the control device when the non-mouse layer data changes.

[0110] Situations where non-mouse layer data changes include: a user modifying the non-mouse layer data on the computing device; a change in the computing device's program modifying the non-mouse layer data; or a control device sending a control command to the computing device, causing the non-mouse layer data to change.

[0111] Optionally, the computing device retrieves non-mouse layer data from the second storage space and sends the non-mouse layer data to the control device.

[0112] Because the non-mouse layer data is temporarily stored in the second storage space in the above method, the computing device retrieves the data from the second storage space when sending it to the control device. Storing data in the cache space can resolve the contradiction of inconsistent communication speeds between the control device and the computing device.

[0113] S107: The control device receives non-mouse layer data of the first screen of the computing device sent by the computing device.

[0114] S108: The control device draws and displays the first remote screen based on the non-mouse layer data of the first screen and the first local mouse data of the control device.

[0115] The first remote screen is the remote screen displayed on the first screen of the remote control computing device.

[0116] In S105 and S106 above, the computing device may execute S105 first, or S106 first, or both simultaneously. This application embodiment does not limit the order in which the computing device executes S105 and S106.

[0117] In the screen display method proposed in the above embodiments, the mouse data in the image displayed on the first remote screen of the control device comes from the local control device, while the non-mouse layer data comes from the computing device. This method ensures that only one mouse image is displayed in the first remote screen of the control device, which can effectively solve the undesirable effects such as mouse following, dragging, and ghosting caused by the overlapping display of mouse images of the control device and the computing device in traditional methods, thereby improving the user experience.

[0118] The above embodiment describes a method for displaying a first remote screen on a control device. This first remote screen can be a screen displayed when the control device and the computing device initially connect, or it can be any remote screen displayed during the process of the control device controlling the computing device. This application also provides an interaction process between the control device and the computing device when the control device switches from a first remote screen to a second remote screen; please refer to [reference needed]. Figure 4 The process includes: S201-S209.

[0119] S201: The control device receives a first operation for the first remote screen.

[0120] The first operation is used to instruct the control device to display a second remote screen, and the first operation indicates the second mouse position of the control device.

[0121] The first action could be moving the mouse.

[0122] In one example, the user moves the mouse image in the first remote screen from a first mouse position to a second mouse position by performing a first operation, such as moving the mouse. The second mouse position is the position of the mouse image in the second remote screen.

[0123] For example, the second mouse position of the control device can be represented in the form of screen coordinates. For instance, the second mouse position of the control device is (x2, y2).

[0124] The second mouse position of the control device indicated by the first operation can be the same as or different from the first mouse position of the control device.

[0125] For control devices, in addition to the first remote screen, there is also a local screen. The mouse cursor can be in either the first remote screen or the local screen. Only when the mouse cursor is in the first remote screen is the operation that indicates the second mouse position called the first operation.

[0126] In one example, such as Figure 5 As shown, Figure 5 Figure A shows a schematic diagram of the mouse cursor on the first remote screen. Figure 5Figure B illustrates a schematic diagram of the mouse cursor on the local screen. When the mouse cursor is in the first remote screen, the operation indicating its second mouse position is called the first operation.

[0127] S202: In response to the first operation, the control device determines the second mouse position information of the computing device based on the second mouse position indicated by the first operation.

[0128] The second mouse position information of the computing device is used to update the first screen of the computing device to the second screen, and the mouse image in the second screen is drawn based on the second mouse position information of the computing device.

[0129] For a detailed description, please refer to S101.

[0130] S203: The control device sends the second mouse position information of the computing device to the computing device.

[0131] For a detailed description, please refer to S102.

[0132] Since the second mouse position of the computing device corresponds to the second mouse position of the control device, the position of the mouse image on the computing device changes based on the position of the mouse image on the control device. For example, when the mouse image on the control device moves under the instruction of the first operation, the control device sends the position information of the mouse image on the first screen to the computing device so that the mouse image on the first screen of the computing device moves to the corresponding position.

[0133] The control device can periodically send the position information of the mouse image to the computing device, or it can send the position information of the mouse image to the computing device when the position of the mouse image on the control device changes.

[0134] It is understandable that the method of periodically sending control devices can improve the sensitivity of mouse image movement on computing devices; the method of triggering sending when the position of the mouse image on the control device changes can save the operating resources of the control device.

[0135] S204: The computing device updates the first image data based on the second mouse position information of the computing device to obtain the second image data.

[0136] The second image data is the image data used to draw the second screen. The second image data includes both non-mouse data and mouse data.

[0137] For a detailed description, please refer to S103.

[0138] S205: The computing device separates the non-mouse layer data from the second image data.

[0139] For a detailed description, please refer to S104.

[0140] S206: The computing device draws and displays a second screen based on the second image data.

[0141] The updated second image data includes non-mouse data and mouse data. The computing device generates a second screen based on the updated image data. The position of the mouse image in the second screen corresponds to the second mouse position of the mouse image in the first remote screen of the first operation instruction in S201.

[0142] S207: The computing device sends non-mouse layer data of the second screen to the control device to control the device to draw and display the second remote screen.

[0143] For a detailed description here, please refer to S106.

[0144] In S206 and S207 above, the computing device may execute S206 first, or S207 first, or both simultaneously. This application embodiment does not limit the execution order of the computing device in S206 and S207 above.

[0145] S208: In response to the first operation, the control device receives non-mouse layer data of the second screen sent by the computing device.

[0146] S209: The control device draws and displays the second remote screen based on the non-mouse layer data of the second screen and the second local mouse data of the control device.

[0147] The second remote screen is a remote screen displayed on the second screen of the remote control computing device.

[0148] The second mouse data of the control device is the data indicating the position of the second mouse of the control device, that is, the data of the second mouse position indicated by the first operation in S201.

[0149] The time interval between the moment when the control device receives the first operation in S201 and the moment when the control device receives the non-mouse layer data of the second screen in S208 is less than a preset threshold.

[0150] The preset threshold value is a very small value. In actual implementation, since the network processing speed is very fast, this value does not need to be set in advance. It is only necessary to select the most recent first operation and non-mouse layer data.

[0151] In the above embodiments, under the remote control of the first operation of the control device, the screen displayed on the computing device switches from the first screen to the second screen, and the control device completes the remote control function of the computing device.

[0152] The following is based on Figure 2The internal structures of the control device and computing device shown are used to illustrate a screen display method provided in embodiments S101-S108 and S201-S209 above. This method may include S301-S323. Wherein, as... Figure 6 As shown, Figure 6 The diagram illustrates the processing steps of each chip within the control and computing devices in S301-S311 for displaying the first remote screen. Figure 7 As shown, Figure 7 The diagram shows the processing flow of each chip inside the control device and computing device in S312-S323 when the control device switches from the first remote screen to the second remote screen.

[0153] S301: The CPU of the control device determines the first mouse position information of the computing device based on the first mouse position of the control device.

[0154] The calculation steps for the first mouse position information include S301a-S301b:

[0155] S301a: The CPU of the control device obtains the resolution of the control device and calculates the resolution of the device.

[0156] S301b: The CPU of the control device determines the first mouse position information of the computing device based on the resolution of the control device, the resolution of the computing device, and the first mouse position of the control device.

[0157] S302: The CPU of the control device sends the first mouse position information of the computing device to the BMC chip of the computing device through the BMC chip.

[0158] Specifically, the CPU of the control device sends the first mouse position information of the computing device to the BMC chip through the USB interface, and the BMC chip sends the first mouse position information of the computing device to the BMC chip of the computing device.

[0159] S303: The CPU of the computing device obtains the first mouse position information from the BMC of the computing device.

[0160] Specifically, the CPU of the computing device obtains the first mouse position information from the BMC of the computing device via the USB interface.

[0161] S304: The CPU of the computing device updates the current image data based on the first mouse position information to obtain the first image data.

[0162] S305: The CPU of the computing device sends the first image data to the GPU.

[0163] S306: The GPU of the computing device separates the non-mouse layer data from the first image data.

[0164] Optionally, the GPU of the computing device stores the first image data in the first storage space and the non-mouse layer data in the second storage space.

[0165] like Figure 8 As shown, Figure 8 The diagram illustrates the processing of first image data by the CPU and GPU of a computing device.

[0166] S307: The GPU of the computing device drives the display to show the first image based on the first image data.

[0167] Optionally, after the GPU of the computing device obtains the first image data from the first storage space, it sends it to the DAC for processing, and then sends it to the display through the VGA interface.

[0168] S308: The GPU of the computing device sends the non-mouse layer data of the first screen to the BMC chip of the control device through the BMC chip.

[0169] Optionally, after the GPU of the computing device obtains the non-mouse layer data of the first screen from the second storage space, it sends it to the video compression module for processing, and then sends it to the BMC chip of the control device through the BMC chip.

[0170] S309: The CPU of the control device obtains the non-mouse layer data of the first screen from the BMC chip.

[0171] S310: The CPU of the control device sends the first local mouse data and the first screen's non-mouse layer data to the GPU.

[0172] S311: The GPU of the control device drives the display to show the first remote screen based on the non-mouse layer data of the first screen and the first local mouse data.

[0173] Specifically, the GPU of the control device converts the non-mouse layer data of the first screen and the first local mouse data through the DAC, and then sends them to the monitor through the VGA interface, so that the monitor displays the first remote screen.

[0174] S312: The CPU of the control device receives a first operation for the first remote screen.

[0175] The first operation indicates the second mouse position in the mouse image of the control device within the first remote screen.

[0176] S313: The CPU of the control device responds to the first operation and determines the second mouse position information of the computing device based on the second mouse position indicated by the first operation.

[0177] S314: The CPU of the control device sends the second mouse position information to the BMC chip of the computing device through the BMC chip.

[0178] S315: The CPU of the computing device obtains the second mouse position information from the BMC of the computing device.

[0179] S316: The CPU of the computing device updates the first image data based on the second mouse position information to obtain the second image data.

[0180] S317: The CPU of the computing device sends the second image data to the GPU.

[0181] S318: The GPU of the computing device separates the non-mouse layer data from the second image data.

[0182] S319: The GPU of the computing device drives the display to show a second image based on the second image data.

[0183] S320: The GPU of the computing device sends non-mouse layer data of the second screen to the BMC chip of the control device through the BMC chip.

[0184] S321: The CPU of the control device obtains the non-mouse layer data of the second screen from the BMC chip.

[0185] S322: The CPU of the control device sends the second local mouse data and the non-mouse layer data of the second screen to the GPU.

[0186] S323: The GPU of the control device drives the display to show a second remote screen based on the non-mouse layer data of the second screen and the second local mouse data.

[0187] The GPU of the control device converts the non-mouse layer data of the second screen and the second local mouse data through the DAC, and then sends them to the monitor through the VGA interface, so that the monitor displays the second remote screen.

[0188] For detailed descriptions of S301-S311 above, please refer to S101-S108; for detailed descriptions of S312-S323, please refer to S201-S209.

[0189] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0190] This application also provides a screen display device 300, such as a control device. Figure 9 The diagram shown is a structural schematic of a screen display device 300 provided in an embodiment of this application.

[0191] The display device 300 includes: a receiving unit 301, configured to receive non-mouse layer data of the first screen of the computing device sent by the computing device; wherein the non-mouse layer data is data separated from the first screen by the computing device; and a display unit 302, configured to draw and display a first remote screen based on the non-mouse layer data and first local mouse data of the control device, wherein the first remote screen is a remote screen displayed by the control device based on the first screen of the remotely controlled computing device. For example, combined with... Figure 3 The receiving unit 301 is used in S107 of the method embodiment, and the display unit 302 is used in S108 of the method embodiment.

[0192] Optionally, the display device 300 further includes a sending unit 303, configured to send first mouse position information of the computing device to the computing device before receiving non-mouse layer data of the first screen of the computing device sent by the computing device, wherein the mouse image in the first screen is drawn based on the first mouse position information. For example, combined with Figure 3 The sending unit 303 is used in S102 of the method embodiment.

[0193] Optionally, the display device 300 further includes a determining unit 304, configured to determine the first mouse position of the control device before sending the first mouse position information to the computing device; wherein the first mouse position of the control device is the mouse position indicated by the first local mouse data; and the first mouse position information of the computing device is determined based on the first mouse position of the control device and the resolution of the computing device. For example, in combination with Figure 3 The determining unit 304 is used in S101 of the method embodiment.

[0194] Optionally, after drawing and displaying the first remote screen, the receiving unit 301 is further configured to receive a first operation on the first remote screen; wherein the first operation is used to display a second remote screen; the first operation indicates a second mouse position of the control device; the sending unit 303 is further configured to, in response to the first operation, send second mouse position information of the computing device to the computing device; the second mouse position information is used to update the first screen of the computing device to the second screen, and the mouse image in the second screen is drawn based on the second mouse position information. For example, combined with Figure 4 The receiving unit 301 is used in S201 of the method embodiment, and the sending unit 303 is used in S202 of the method embodiment.

[0195] Optionally, the receiving unit 301 is further configured to, in response to the first operation, receive non-mouse layer data of the second screen sent by the computing device; wherein the non-mouse layer data is data separated from the second screen by the computing device; the display unit 302 is further configured to, based on the non-mouse layer data of the second screen and the second local mouse data of the control device, draw and display a second remote screen; wherein the second remote screen is a remote screen displayed by the control device based on the second screen of the remotely controlled computing device; the second mouse data of the control device is data indicating the second mouse position of the control device. For example, combined with Figure 4 The receiving unit 301 is used in S208 of the method embodiment, and the display unit 302 is used in S209 of the method embodiment.

[0196] Optionally, the time interval between receiving the first operation and receiving the non-mouse layer data of the second screen is less than a preset threshold.

[0197] Of course, the screen display device 300 provided in this application embodiment includes, but is not limited to, the above-described modules.

[0198] For example, combining Figure 6 The functions of the receiving unit 301 and the determining unit 304 can be achieved through... Figure 6 The CPU of the central control device is implemented, and the function of the display unit 302 can be achieved through... Figure 6 The control device is implemented using a GPU and a display, and the function of the transmitting unit 303 can be achieved through... Figure 6 The CPU and BMC implementation of the control device.

[0199] This application embodiment also provides another screen display device 400, such as a computing device. Figure 10 The diagram shown is a structural schematic of a screen display device 400 provided in an embodiment of this application.

[0200] The display device 400 includes: an update unit 401, configured to update current image data based on first mouse position information of a computing device to obtain first image data; the first image data is used to draw a first screen; a separation unit 402, configured to separate non-mouse layer data from second image data; and a sending unit 403, configured to send the non-mouse layer data to a control device; wherein the non-mouse layer data is used to instruct the control device to display a first remote screen; the first remote screen is drawn based on the first local mouse data of the control device and the non-mouse layer data of the computing device. For example, combined with... Figure 3 The update unit 401 is used in S103 of the method embodiment, the separation unit 402 is used in S104 of the method embodiment, and the sending unit 403 is used in S106 of the method embodiment.

[0201] Optionally, the display device 400 further includes a receiving unit 404, configured to receive the first mouse position information of the computing device sent by the control device before drawing the first screen based on the first mouse position information of the computing device. The first mouse position information is determined by the control device based on the first mouse position of the control device and the resolution of the computing device. For example, the receiving unit 404 is used in S102 of the method embodiment.

[0202] Optionally, the display device 400 further includes a storage unit 405, used to separate the non-mouse layer data from the first image data, store the first image data in a first storage space, and store the non-mouse layer data in a second storage space. For example, the storage unit 405 is used in S104 of the method embodiment.

[0203] Optionally, the display device 400 further includes an acquisition unit 406, used to acquire non-mouse layer data from a second storage space before sending the non-mouse layer data to the control device. For example, the acquisition unit 401 is used in S106 of the method embodiment.

[0204] Optionally, the acquisition unit 406 is further configured to acquire first image data from the first storage space, and the screen display device 400 further includes a display unit 407 configured to draw and display a first screen based on the first image data. For example, the acquisition unit 406 and the display unit 404 are used in S105 of the method embodiment.

[0205] Of course, the screen display device 400 provided in this application embodiment includes, but is not limited to, the above-described modules.

[0206] For example, combining Figure 6 The function of updating unit 401 can be achieved through... Figure 6 The CPU implementation of the computing device allows the functions of the separation unit 402, storage unit 405, and acquisition unit 406 to be achieved through... Figure 6The GPU implementation of the computing device allows the functions of the transmitting unit 403 and the receiving unit 404 to be achieved through... Figure 6 The BMC implementation of the computing device allows the display unit 407 to function through... Figure 6 The implementation of GPUs and displays in computing devices.

[0207] Another embodiment of this application provides a screen display device, which can be a server, tablet computer, desktop computer, laptop computer, netbook, or similar device. The screen display device includes a memory and a processor. The memory and processor are coupled; the memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, the screen display device performs the various steps of the screen display method shown in the above-described method embodiment.

[0208] Another embodiment of this application provides a computer-readable storage medium storing computer instructions that, when executed on a screen display device, cause the screen display device to perform each step of the screen display method flow shown in the above method embodiment.

[0209] Another embodiment of this application provides a chip system applied to a screen display device. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The interface circuits are used to receive signals from the memory of the screen display device or computing device and send signals to the processor, the signals including computer instructions stored in the memory. When the screen display device processor executes the computer instructions, the screen display device performs each step of the screen display method flow shown in the above method embodiment.

[0210] In another embodiment of this application, a computer program product is also provided, which includes computer instructions that, when executed on a screen display device, cause the screen display device to perform each step of the screen display method flow shown in the above method embodiment.

[0211] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented using software programs, the above embodiments can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a server, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0212] The above description is merely a specific embodiment of this application. Any variations or substitutions conceived by those skilled in the art based on the specific embodiments provided in this application should be covered within the protection scope of this application.

Claims

1. A method for displaying a screen, characterized in that, Applied to control equipment, the method includes: The computing device receives non-mouse layer data of its first screen, which is data extracted from the first screen by the computing device. Based on the non-mouse layer data and the first local mouse data of the control device, a first remote screen is drawn and displayed. The first remote screen is a remote screen displayed by the control device based on the first screen of the remotely controlled computing device. After drawing and displaying the first remote view, the method further includes: Receive a first operation for the first remote screen; wherein the first operation is used to display a second remote screen; the first operation indicates a second mouse position of the control device; In response to the first operation, the computing device sends second mouse position information of the computing device; the second mouse position information is used to update the first screen of the computing device to a second screen, and the mouse image in the second screen is drawn based on the second mouse position information.

2. The method according to claim 1, characterized in that, Before receiving the non-mouse layer data of the first screen of the computing device sent by the computing device, the method further includes: The computing device sends the first mouse position information of the computing device to the computing device, and the mouse image in the first screen is drawn based on the first mouse position information.

3. The method according to claim 2, characterized in that, Before sending the first mouse position information to the computing device, the method further includes: Determine the first mouse position of the control device; wherein, the first mouse position of the control device is the mouse position indicated by the first local mouse data; The first mouse position information of the computing device is determined based on the first mouse position of the control device and the resolution of the computing device.

4. The method according to claim 1, characterized in that, The method further includes: In response to the first operation, the non-mouse layer data of the second screen sent by the computing device is received; wherein the non-mouse layer data is data separated from the second screen by the computing device; Based on the non-mouse layer data of the second screen and the second local mouse data of the control device, a second remote screen is drawn and displayed; wherein, the second remote screen is a remote screen displayed by the control device based on the second screen of the remotely controlled computing device; the second mouse data of the control device is data indicating the second mouse position of the control device.

5. The method according to claim 4, characterized in that, The time interval between receiving the first operation and receiving the non-mouse layer data of the second screen is less than a preset threshold.

6. A method for displaying a screen, characterized in that, Applied to a computing device, the method includes: The control device receives first mouse position information of the computing device sent by the control device, wherein the first mouse position information is determined by the control device based on the first mouse position of the control device and the resolution of the computing device; Based on the first mouse position information of the computing device, the current image data is updated to obtain the first image data; the first image data is used to draw the first screen. Separate the non-mouse layer data from the first image data; The non-mouse layer data is sent to the control device; wherein the non-mouse layer data is used to instruct the control device to display a first remote screen; the first remote screen is drawn based on the first local mouse data of the control device and the non-mouse layer data.

7. The method according to claim 6, characterized in that, After separating the non-mouse layer data from the first image data, the method further includes: The first image data is stored in the first storage space, and the non-mouse layer data is stored in the second storage space.

8. The method according to claim 7, characterized in that, Before sending the non-mouse layer data to the control device, the method further includes: The non-mouse layer data is obtained from the second storage space.

9. The method according to claim 7, characterized in that, The method further includes: The first image data is obtained from the first storage space, and a first screen is drawn and displayed based on the first image data.

10. A screen display device, characterized in that, The device includes a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code including computer instructions; wherein, when the processor executes the computer instructions, the display device performs the method as described in any one of claims 1-9.

Images