Screen capture method, device and electronic device based on virtual display

Abstract

A virtual display-based screenshot method, device, and electronic device, applied to a cloud desktop server, are characterized by comprising the following steps: an operating system generates a video frame and stores it in a ring buffer, the ring buffer being located in the operating system's graphics card cache memory; the virtual display obtains and locks the current video frame, obtains an encoding configuration in real time to encode the current video frame, and upon completion of encoding, releases the current video frame and stores the encoded video frame in the ring buffer; an application periodically or upon notification obtains the encoded video frame from the ring buffer and transmits it over a network. The present invention eliminates memory copying from video generation to encoding, reducing the performance overhead and loss caused by memory copying and improving screenshot efficiency.

Description

Technical Field

[0001] The present invention relates to the field of cloud desktops, and in particular to a screenshot method, device and electronic equipment based on a virtual display. Background Art

[0002] With the development of Internet technology, cloud technology and cloud storage are attracting more and more attention from the IT industry. Their applications are gradually being promoted and have received widespread acclaim.

[0003] Existing cloud desktop systems are based on virtual desktop technology. In a cloud desktop system, cloud desktop terminals rarely require application installation; they communicate with a cloud desktop server via a unique connection protocol. The cloud desktop terminal displays the user's interactive desktop. The cloud desktop terminal transmits data such as mouse movements, clicks, and keyboard operations to the cloud desktop server. The cloud desktop server then renders the corresponding cloud desktop image based on the data transmitted by the cloud desktop terminal. The server then takes a screenshot of the rendered cloud desktop image and transmits the screenshot back to the cloud desktop terminal for display.

[0004] At present, a screenshot solution using a virtual display has emerged, which is as follows: a virtual display is installed on the cloud desktop server, and the virtual display is used to render the received data to obtain a cloud desktop image; when it is detected that the cloud desktop image currently rendered by the virtual display has changed, the cloud desktop image currently rendered by the virtual display is captured; the captured cloud desktop image is written to the graphics card memory corresponding to the virtual display; when it is detected that the cloud desktop image is written to the graphics card memory, the cloud desktop image is found from the graphics card memory and encoded; and the encoded cloud desktop image is sent to the cloud desktop terminal.

[0005] However, this solution has the following shortcomings: after the virtual display generates the cloud desktop image, the application copies the cloud desktop to the memory for encoding. Therefore, each cloud desktop image output will generate at least one copy, which will have a significant impact on the computer's performance overhead and loss. In addition, after the screenshot is taken, the system notification is made to inform the application of the event, and the application then encodes it, binding the entire behavior to business operations and system events. There are many links, so the robustness is relatively weak. In addition, when the application encoding time is usually longer than the interval between desktop image changes, frame skipping will occur. For example, the second frame of the cloud desktop image is generated before the first frame is encoded. When the encoding is completed, only the third frame of the cloud desktop image can be obtained. There may also be continuous delays, such as buffering all desktop images, but due to slow encoding, the image being encoded is much earlier than the current latest desktop image. Therefore, this solution is not suitable for displays with higher refresh rates such as 144fps and 165. Summary of the Invention

[0006] The main purpose of the present invention is to overcome the above-mentioned defects in the prior art and to propose a screenshot method, device and electronic device based on a virtual display, which can reduce the performance overhead and loss caused by memory copying and improve the screenshot efficiency.

[0007] The present invention adopts the following technical solutions:

[0008] A screenshot method based on a virtual display, applied to a cloud desktop server, is characterized by comprising the following steps:

[0009] The operating system generates video frames and stores them in a ring buffer, which is located in the graphics card cache memory of the operating system;

[0010] The virtual display obtains and locks the current video frame, obtains the encoding configuration in real time to encode the current video frame, and after the encoding is completed, releases the current video frame and saves the encoded video frame in the ring buffer;

[0011] The application obtains the encoded video frames from the ring buffer periodically or after being notified, and sends them through the network.

[0012] Preferably, if the virtual display fails to obtain the current video frame, the virtual display is destroyed and a new virtual display is created to obtain the current video frame.

[0013] Preferably, the virtual display calls a system function to lock or release the video frame; the system function includes a locking function and a releasing function.

[0014] Preferably, the step of obtaining the encoding configuration to encode the current video frame specifically includes the following steps:

[0015] When obtaining the current video frame, the latest encoding configuration is obtained from the shared memory of the operating system and compared with the previous encoding configuration to determine whether the encoder instance needs to be recreated. If so, the encoder instance is initialized. After that, the encoder instance object is obtained and encoding is performed in combination with the latest encoding configuration.

[0016] Preferably, if the ring buffer is full of the encoded video frames that have not been taken away, the encoded video frames are locked, and the operating system stops generating the video frames until at least one encoded video frame is taken away, and then releases the encoded video frames.

[0017] Preferably, the operating system manages the status of the unencoded video frames and the encoded video frames in the ring buffer.

[0018] Preferably, the operating system generates video frames at a set time interval, where the set time interval is determined by the frame rate or screen refresh rate or network speed for transmitting the video; or, the operating system generates video frames when the desktop changes.

[0019] Preferably, the application is a business program transmitted over the network.

[0020] A screenshot device based on a virtual display, applied to a cloud desktop server, is characterized by comprising:

[0021] A generation module generates video frames and stores them in a ring buffer, where the ring buffer is located in a graphics card cache memory of an operating system;

[0022] Acquisition module, obtains the current video frame for locking;

[0023] An encoding module, which obtains an encoding configuration and encodes the current video frame according to the encoding configuration, and after encoding is completed, releases the current video frame and stores the encoded video frame in the ring buffer;

[0024] The sending module obtains the encoded video frame from the ring buffer periodically or after being notified, and sends it through the network.

[0025] An electronic device, characterized in that it includes a processor and a memory, wherein the processor is used to execute a screenshot method program stored in the memory to implement the screenshot method based on a virtual display.

[0026] From the above description of the present invention, it can be seen that compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. In this invention, encoded video frames are stored in a ring buffer. The application periodically or upon notification retrieves the encoded video frames from the ring buffer and transmits them over the network. From video generation to encoding, no memory copying is performed, reducing the performance overhead and loss caused by memory copying and improving screenshot efficiency.

[0028] 2. In the present invention, since the operating system can generate video frames according to the set time interval, and the virtual display obtains the current video frame, the video output time interval is fixed, so the video data output frequency is relatively smooth and is not affected by system events.

[0029] 3. In the present invention, when the virtual display obtains the currently generated video frame, it locks the video frame for encoding, and releases the video frame after encoding to be stored in the ring buffer. The purpose of locking the video frame is to prevent the video frame from being overwritten by a new video frame before it is encoded. By releasing the desktop image texture in the ring buffer, the refresh time and refresh frequency of the operating system cloud desktop image are controlled.

[0030] 4. In the present invention, if the ring buffer is full of unremoved encoded video frames, the encoded video frames will be locked until at least one encoded video frame is removed. The operating system stops generating video frames and no new data will be generated. The virtual display will also pause the acquisition of video frames, the frame rate will drop, and the refresh time will naturally drop. The purpose is to more freely control the frequency of screenshots to meet subsequent functional business requirements.

[0031] 5. In the present invention, screenshot and encoding are achieved through a virtual display, and the entire behavior is not bound to business operations and system events. There are fewer links, so the robustness is stronger. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the main flow chart of the method of the present invention;

[0033] Figure 2 Specific flow chart of the method of the present invention;

[0034] Figure 3 It is a composition diagram of the device of the present invention;

[0035] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0036] The present invention is further described below through specific embodiments.

[0037] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0038] Currently, cloud desktop systems typically include cloud desktop servers and cloud desktop terminals. These servers and terminals establish a network connection and communicate via a specific connection protocol. One cloud desktop server can correspond to multiple cloud desktop terminals, and the cloud desktop server can receive operational information from these terminals. A cloud desktop server isn't strictly a server; it can also be a computer or other device. For example, when User A controls User B's computer via a remote client on their computer, User B's computer acts as the cloud desktop server, while User A's computer acts as the cloud desktop terminal. Cloud desktop terminals can be terminals with display screens, including but not limited to smartphones, laptops, tablets, and desktop computers.

[0039] See also Figure 1 、 Figure 2 The present invention proposes a screenshot method based on a virtual display, which is applied to a cloud desktop server. The cloud desktop server is equipped with a virtual display, which is used to obtain video frames generated by the operating system for display. By using the virtual display to "display" the cloud desktop image, the cloud desktop server no longer needs to be connected to a physical display. The screenshot method of the present invention specifically includes the following steps:

[0040] The operating system generates video frames and stores them in a ring buffer, which is located in the graphics cache memory of the operating system;

[0041] The virtual display obtains and locks the current video frame, obtains the encoding configuration in real time to encode the current video frame, and after the encoding is completed, releases the current video frame and saves the encoded video frame in the ring buffer;

[0042] The application obtains the encoded video frames from the ring buffer periodically or after being notified and sends them over the network.

[0043] In the present invention, the operating system generates video frames at set intervals. The set interval can be determined by the frame rate, screen refresh rate, or network speed of the video transmission network. Typically, the operating system outputs video frames at fixed intervals. For example, if the frame rate is set to 60Hz, the interval is 1000ms / 60, which is approximately 16ms (milliseconds), meaning that a video frame is output every 16 milliseconds. The video frames in the present invention can also be referred to as cloud desktop images. Because the video frame output interval is fixed, the video frame data output frequency is relatively smooth and is not affected by operating system events or network speed. In practical applications, the operating system can also be configured to generate video frames when the desktop changes, which can be set according to needs and is flexible and controllable. The virtual display captures the current video frame and locks it. In other words, as soon as the operating system generates a new video frame, the new video frame is captured by the virtual display. The acquisition interval can be the same as the video frame generation interval. In the present invention, if the virtual display fails to capture a new video frame, the virtual display can be destroyed and a new virtual display can be created to recapture the current video frame.

[0044] Furthermore, the video frame is locked for encoding, released after encoding, and the encoded video frame is saved in a ring buffer. The purpose of locking the video frame is to prevent the video frame from being overwritten by a new video frame before it is encoded. The virtual display calls a system function to lock or release the video frame; the system function includes a locking function and a releasing function. For example, the locking function is IddCxSwapChainReleaseAndAcquireBuffer, and the releasing function is IddCxSwapChainFinishedProcessingFrame. See Figure 2 The real-time acquisition encoding configuration of the present invention encodes the current video frame, specifically including the following:

[0045] When acquiring the current video frame, the latest encoding configuration is retrieved from the operating system's shared memory or elsewhere and compared with the previous encoding configuration to determine whether a new encoder instance needs to be created. If the latest encoding configuration matches the previous one, there's no need to recreate the encoder instance; the encoder instance object is directly retrieved and encoding is performed using the latest encoding configuration. If they are inconsistent, the encoder instance must be recreated, initially initialized, and then the encoder instance object is retrieved and encoding is performed using the latest encoding configuration. After encoding is complete, the result is returned to the virtual display. After encoding is complete, the virtual display can also notify the application to remove the encoded video frame, using events or other methods. The video frame can also be released after notifying the application.

[0046] The virtual display can perform encoding operations by invoking the graphics card's interface. The encoding configuration is stored in a designated location in the operating system's shared memory. Because the encoding configuration is subject to change, the latest encoding configuration must be retrieved for each encoding operation. For example, the encoding configuration might be h264, 20 Mbps, and 1080p at 60Hz. By storing the encoding configuration in shared memory, the virtual display can access the configuration more quickly, and the configuration takes effect in real time.

[0047] Furthermore, the ring buffer of the present invention is located in the operating system's memory and, through shared memory, allows specific processes to read and modify it. When the virtual reality device stores encoded video frames in the ring buffer, it can also notify the operating system of the status of several encoded video frames in the ring buffer. The operating system can then manage the status of the video frames in the ring buffer. Video frame status includes displayable video frames, overwritten video frames, and ready-to-display video frames. Displayable video frames are encoded video frames that can be removed, ready-to-display video frames are video frames that will be encoded later, and overwritten video frames are video frames that can be overwritten by new video frames.

[0048] In the present invention, if the ring buffer is full of unremoved encoded video frames, the encoded video frames are locked to prevent them from being overwritten by new video frames. The operating system stops generating video frames until at least one encoded video frame is removed, at which point the encoded video frames are released. For example, assuming the ring buffer has two parameters, one for the written position and the other for the read position, the written position is incremented by 1 after writing, and the read position is incremented by 1 after reading. If the written position minus the read position equals the size of the ring buffer, the ring buffer is full and no more data can be written. Otherwise, data will be written to the position currently being read, resulting in dirty data.

[0049] Specifically, the present invention controls the refresh time and frequency of the operating system's cloud desktop image by releasing desktop image textures in a ring buffer. When the operating system stops generating video frames, no new data is generated, and the virtual display pauses capturing video frames, reducing the frame rate and naturally the refresh time. This allows for more flexible control of the frequency of screenshots to meet subsequent functional and business needs.

[0050] The operating system of the present invention is an operating system in a narrow sense, similar to Windows, Linux, MacOS, etc. The application is a business program transmitted over the network. The application will periodically or upon notification, obtain video frames in a ring buffer at a specified location that are in a displayable state, and then send the video frames over the network, for example, to a cloud desktop client.

[0051] See also Figure 3The present invention also proposes a screenshot device based on a virtual display, which is applied to a cloud desktop server and includes:

[0052] The generation module generates video frames and stores them in a ring buffer. It can generate video frames at a set interval. This interval can be determined by the resolution, frame rate, or screen refresh rate. The generation module can then output video frames at a fixed interval. It can also be configured to generate video frames whenever the desktop changes. This flexibility is controllable and can be customized to your needs.

[0053] The acquisition module acquires the current video frame for locking. That is, under normal circumstances, as soon as the generation module generates a new video frame, the new video frame is acquired, and the acquisition interval can be the same as the video frame generation interval, or can be set to be different.

[0054] The encoding module obtains the encoding configuration and encodes the current video frame according to the encoding configuration. After encoding is completed, the current video frame is released and the encoded video frame is saved in the ring buffer. Specifically, the latest encoding configuration is obtained from the shared memory of the operating system and compared with the previous encoding configuration to determine whether the encoder instance needs to be recreated. If the latest encoding configuration is consistent with the previous encoding configuration, there is no need to recreate the encoder instance. The encoder instance object is directly obtained and encoding is performed based on the latest encoding configuration. If they are inconsistent, the encoder instance needs to be recreated, that is, the encoder instance is initialized. After that, the encoder instance object is obtained and encoding is performed based on the latest encoding configuration.

[0055] The sending module obtains the encoded video frames from the ring buffer periodically or upon notification and sends them over the network. Specifically, the sending module obtains the video frames to be displayed and sends them.

[0056] An electronic device of the present invention includes a processor and a memory, wherein the processor is used to execute a screenshot method program stored in the memory to implement the above-mentioned screenshot method based on a virtual display.

[0057] The memory in the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0058] In some embodiments, the memory stores the following elements, executable units or data structures, or a subset or an extended set thereof: an operating system and application programs, etc.

[0059] The operating system includes various system programs, such as a framework layer, a core library layer, and a driver layer, which are used to implement various basic services and handle hardware-based tasks. It also includes various application programs, such as a media player (MediaPlayer) and a browser (Browser), which are used to implement various application services. In an embodiment of the present invention, the processor is used to execute the method steps provided by the method of the present invention by calling programs or instructions stored in the memory.

[0060] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited to this. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A screenshot method based on a virtual display, applied to a cloud desktop server, characterized in that: The steps include: The operating system generates video frames and stores them in a ring buffer, which is located in the graphics card cache memory of the operating system; The virtual display obtains and locks the current video frame, obtains the encoding configuration in real time to encode the current video frame, and after the encoding is completed, releases the current video frame and saves the encoded video frame in the ring buffer; The application obtains the encoded video frames from the ring buffer periodically or after being notified, and sends them through the network.

2. A screenshot method based on a virtual display according to claim 1, characterized in that: If the virtual display fails to obtain the current video frame, the virtual display is destroyed and a new virtual display is created to obtain the current video frame.

3. A screenshot method based on a virtual display according to claim 2, characterized in that: The virtual display calls a system function to lock or release the video frame; the system function includes a locking function and a releasing function.

4. A screenshot method based on a virtual display according to claim 1, characterized in that: The obtaining of the encoding configuration to encode the current video frame specifically includes the following steps: When obtaining the current video frame, the latest encoding configuration is obtained from the shared memory of the operating system and compared with the previous encoding configuration to determine whether the encoder instance needs to be recreated. If so, the encoder instance is initialized. After that, the encoder instance object is obtained and encoding is performed in combination with the latest encoding configuration.

5. The screenshot method based on a virtual display according to claim 1, wherein: If the ring buffer is full of the encoded video frames that have not been taken away, the encoded video frames are locked, and the operating system stops generating the video frames until at least one encoded video frame is taken away, and then releases the encoded video frames.

6. The screenshot method based on a virtual display according to claim 1, wherein: The operating system manages the states of the unencoded video frames and the encoded video frames in the ring buffer.

7. The screenshot method based on a virtual display according to claim 1, wherein: The operating system generates video frames at a set time interval, where the set time interval is determined by a frame rate or a screen refresh rate or a network speed for transmitting the video; or, when the desktop changes, the operating system generates video frames.

8. The method for capturing a screenshot based on a virtual display according to claim 1, wherein: The application program is a service program transmitted over the network.

9. A screenshot device based on a virtual display, applied to a cloud desktop server, characterized in that: include: A generation module, wherein the operating system generates video frames and stores the frames in a ring buffer, wherein the ring buffer is located in a graphics card cache memory of the operating system; Acquisition module, obtains the current video frame for locking; The encoding module, the virtual display obtains the encoding configuration and encodes the current video frame according to the encoding configuration, and after the encoding is completed, releases the current video frame and stores the encoded video frame in the ring buffer; The sending module, the application obtains the encoded video frame from the ring buffer periodically or after being notified, and sends it through the network.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the processor is used to execute a screenshot method program stored in the memory to implement the screenshot method based on a virtual display according to any one of claims 1 to 8.

Images