Cloud desktop display method, cloud desktop terminal, and storage medium

By performing backward interpolation frame prediction and displaying predicted frames in the cloud desktop terminal, the problem of insufficient display fluency caused by network blockage of cloud desktop is solved, and the display frame rate is improved when the transmission frame rate of the cloud desktop server is insufficient, and the improvement is achieved. User experience.

WO2024244587A1PCT designated stage expired Publication Date: 2024-12-05CLOUD INTELLIGENCE ASSETS HOLDING (SINGAPORE) PTE LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/080639
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-30
Filing Date
2024-03-07
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

The display fluency of cloud desktop is insufficient, which is mainly due to the decrease in the frame rate of the back-end transmission due to network blockage, which in turn causes the front-end display frame rate to decrease simultaneously, resulting in the display of cloud desktop not being smooth.

Method used

In the cloud desktop terminal, backward interpolation frame prediction is performed based on the received image frame and the previous frame, predicted frames are constructed, and predicted frames are displayed before receiving the next frame, so that the cloud desktop server will independently behave when the transmission frame rate is insufficient. Improve display frame rate and improve fluency.

Benefits of technology

By actively performing backward insertion frames, cloud desktop terminals can increase display frame rate without increasing bandwidth and delay, improve the display fluency of cloud desktops, and ensure user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024080639_05122024_PF_FP_ABST
    Figure CN2024080639_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present application provide a cloud desktop display method, a cloud desktop terminal, and a storage medium. In the embodiments of the present application, the cloud desktop terminal is reconstructed. After an image frame sent by a cloud desktop server is received, the image frame can be displayed; backward frame interpolation prediction can be performed on the basis of the image frame and a preceding reference frame, to construct a prediction frame that can be interposed before the next image frame; and on this basis, the prediction frame can be displayed before the next image frame is received. In this way, the cloud desktop terminal can actively perform backward frame interpolation while normally displaying the image frame acquired from the cloud desktop server, so that the display frame rate is autonomously increased in the cloud desktop terminal when the transmission frame rate of the cloud desktop server is insufficient, thereby improving the display fluency of the cloud desktop without increasing a bandwidth and a delay.
Need to check novelty before this filing date? Find Prior Art

Description

A cloud desktop display method, cloud desktop terminal and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 30, 2023, with application number 202310630891.8 and invention name “A cloud desktop display method, cloud desktop terminal and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of cloud computing technology, and in particular to a cloud desktop display method, a cloud desktop terminal, and a storage medium. Background Art

[0003] Cloud desktops, also known as desktop virtualization or cloud computing, are a new alternative to traditional computers. With cloud desktops, users no longer need to purchase a mainframe computer. All of the mainframe's components are virtualized on a backend server, while the frontend devices are primarily used for interactive operations. Through the collaboration of the frontend devices and backend servers, the user experience is consistent with that of a computer.

[0004] Currently, the front-end display frame rate and back-end transmission frame rate of the cloud desktop are consistent. When the cloud desktop undergoes drastic changes, the back-end transmission frame rate often drops due to network congestion, and the front-end display frame rate will also drop synchronously, resulting in insufficient display smoothness of the cloud desktop.

[0005] Summary of the Invention

[0006] Various aspects of the present application provide a cloud desktop display method, a cloud desktop terminal, and a storage medium to improve the display smoothness of the cloud desktop.

[0007] The present invention provides a cloud desktop display method applicable to a cloud desktop terminal, the method comprising:

[0008] After receiving the Nth image frame sent by the cloud desktop server, displaying the Nth image frame;

[0009] Constructing a prediction frame preceding the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server;

[0010] Before receiving the N+1th image frame, displaying the predicted frame;

[0011] Wherein, N is an integer greater than 1.

[0012] The embodiment of the present application also provides a cloud desktop terminal, including a memory, a processor and a communication component;

[0013] The memory is used to store one or more computer instructions;

[0014] The processor is coupled to the memory and the communication component, and is configured to execute the one or more computer instructions to execute the aforementioned cloud desktop display method.

[0015] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the aforementioned cloud desktop display method.

[0016] In an embodiment of the present application, the cloud desktop terminal is modified. After receiving the image frame sent by the cloud desktop server, the image frame can be displayed; and backward interpolation prediction can be performed based on the image frame and its previous reference frame to construct a predicted frame that can be inserted before the next image frame; based on this, the predicted frame can be displayed before the next image frame. In this way, the cloud desktop terminal can actively perform backward interpolation in addition to the normal display of the image frame obtained from the cloud desktop server, so that when the transmission frame rate of the cloud desktop server is insufficient, the display frame rate can be autonomously increased in the cloud desktop terminal to improve the display smoothness of the cloud desktop without increasing bandwidth and delay.

[0017] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0019] FIG1 is a flow chart of a cloud desktop display method provided by an exemplary embodiment of the present application;

[0020] FIG2 is a logical diagram of a cloud desktop display method provided by an exemplary embodiment of the present application;

[0021] FIG3a to FIG3c are schematic diagrams of an application scenario provided by an exemplary embodiment of the present application;

[0022] FIG4 is a schematic structural diagram of a cloud desktop terminal provided by another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0023] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] At present, due to the instability of the transmission frame rate, the display smoothness of the cloud desktop is insufficient. For this reason, in some embodiments of the present application: the cloud desktop terminal is modified. After receiving the image frame sent by the cloud desktop server, the image frame can be displayed; and backward interpolation prediction can be performed based on the image frame and its previous reference frame to construct a predicted frame that can be inserted before the next image frame; based on this, the predicted frame can be displayed before the next image frame. In this way, the cloud desktop terminal can actively perform backward interpolation in addition to the normal display of the image frame obtained from the cloud desktop server, so that when the transmission frame rate of the cloud desktop server is insufficient, the display frame rate can be autonomously increased in the cloud desktop terminal to improve the display smoothness of the cloud desktop without increasing bandwidth and delay.

[0025] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0026] Figure 1 is a flow chart of a cloud desktop display method provided by an exemplary embodiment of the present application. The method can be executed by a data processing device, which can be implemented as software, hardware, or a combination of software and hardware. The data processing device can be integrated into a cloud desktop terminal. Referring to Figure 1, the method may include:

[0027] Step 100: After receiving the Nth image frame sent by the cloud desktop server, display the Nth image frame;

[0028] Step 101: construct a prediction frame before the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server;

[0029] Step 102: Display the predicted frame before receiving the N+1th image frame;

[0030] Wherein, N is an integer greater than 1.

[0031] The cloud desktop display method provided in this embodiment can be applied to various application scenarios using cloud desktops, and this embodiment does not limit the application scenarios.

[0032] Cloud desktop technology primarily involves cloud desktop servers and cloud desktop terminals. Data exchange between the cloud desktop server and the cloud desktop terminal relies on a network. The cloud desktop terminal can transmit keyboard and mouse operation data to the cloud desktop server. The cloud desktop server, based on this keyboard and mouse operation data, simulates the cloud desktop screen and transmits the screen back to the cloud desktop terminal. The cloud desktop server typically transmits the cloud desktop screen to the cloud desktop terminal via a video stream. In this embodiment, the cloud desktop screen is described as an image frame.

[0033] During the research process, the applicant found that the transmission frame rate of the cloud desktop server is unstable. Especially when a window is dragged or a page is scrolled, the sudden changes in the cloud desktop will cause a large amount of data to need to be encoded and transmitted over the network. The transmission frame rate of the cloud desktop server will decrease, and the display frame rate in the cloud desktop terminal will also decrease synchronously, resulting in insufficient display smoothness of the cloud desktop.

[0034] To this end, this embodiment innovatively proposes a cloud desktop display solution, in which the cloud desktop terminal performs frame insertion at an appropriate time to improve the display smoothness of the cloud desktop.

[0035] FIG2 is a logical diagram of a cloud desktop display method provided by an exemplary embodiment of the present application. Referring to FIG2 , a frame insertion engine can be added to the cloud desktop terminal to execute the relevant logical steps of the cloud desktop display method provided by this embodiment. The frame insertion engine can communicate with the existing client in the cloud desktop terminal. In this embodiment, the client in the cloud desktop terminal can trigger the frame insertion engine to execute the cloud desktop display method provided by this embodiment each time it receives an image frame sent by the cloud desktop server.

[0036] Referring to Figure 1, in step 100, after receiving the current image frame sent by the cloud desktop server, the current image frame can be displayed. For ease of description, in this embodiment, the current frame is described as the Nth image frame. It should be understood that N can be any integer greater than 1. For the image frames received by the cloud desktop terminal from the cloud desktop server, these image frames can be displayed according to the traditional display scheme in this embodiment, that is, after receiving these image frames, the display processing can be directly performed. In addition, in this embodiment, the transmission frame rate of the cloud desktop server is not interfered with, and the cloud desktop terminal can increase its own display frame rate as needed.

[0037] Continuing to refer to FIG. 1 , in step 101 , a prediction frame preceding the N+1 th image frame may be constructed based on the N th image frame and the N−1 th image frame received from the cloud desktop server.

[0038] In this embodiment, the Nth image frame and the N-1th image frame can be used as references for backward frame prediction. In this way, the predicted frames can be used to supplement the display of the intermediate state of the relevant desktop objects between the Nth image frame and the N+1th image frame. The relevant desktop objects here refer to the desktop objects appearing in the Nth image frame and the desktop objects newly added in the N+1th image frame.

[0039] In a preferred implementation, it is proposed that: after receiving the Nth image frame sent by the cloud desktop, it is also possible to predict whether the desktop objects in the Nth image frame will only undergo linear motion. Here, the desktop objects in the image frame may include but are not limited to various display windows and display icons, etc., which are not limited here. There may be multiple desktop objects contained in the image frame, and in this implementation, the above-mentioned prediction operation can be performed on all desktop objects contained in the image frame. What is predicted here is whether each desktop object in the Nth image frame will only undergo linear motion next. Linear motion can also be understood as straight-line motion, which means that the motion trajectory of the object is a straight line. In this embodiment, a variety of implementation methods can be used to complete the above-mentioned prediction operation, and the specific implementation method will be described in detail later.

[0040] In this implementation, if the desktop object in the Nth image frame is predicted to undergo only linear motion, the aforementioned backward interpolation operation is performed. Since the desktop object in the Nth image frame is predicted to undergo only linear motion, backward frame prediction can be accurately performed based on the linear motion trend of the desktop object. The constructed predicted frame conforms to the linear motion trend of the desktop object in the Nth image frame and can be well connected to the N+1th image frame, ensuring smooth image processing.

[0041] Of course, in addition to the preferred implementation described above, other triggering conditions can also be used in this embodiment to trigger the backward frame interpolation operation in step 101, such as a user proactively initiating a frame interpolation start instruction in the cloud desktop terminal. It is also possible that the cloud desktop terminal performs the aforementioned backward frame interpolation operation for each image frame received from the cloud desktop server without configuring a triggering condition. Both of these can effectively improve the display frame rate in the cloud desktop terminal, and this embodiment is not limited thereto.

[0042] The specific implementation method of constructing the prediction frame will be described in detail later.

[0043] In addition, in this embodiment, the prediction frame constructed in step 101 may be one or more, and this embodiment does not limit the number of prediction frames.

[0044] On this basis, referring to FIG1 , in step 102, the predicted frame can be displayed before the N+1th image frame is received. In practical applications, the display timing of the predicted frame can be determined based on the motion moment represented by the predicted frame. For example, if the predicted frame represents an intermediate state 0.1ms after the Nth image frame, the predicted frame can be displayed 0.1ms after the Nth image frame is displayed. If multiple predicted frames are constructed in step 101, the display timing corresponding to each of the multiple predicted frames can also be determined accordingly, so that the multiple predicted frames can be displayed sequentially.

[0045] After receiving the N+1th image frame sent by the cloud desktop server, the N+1th image frame can be displayed. In this way, from the perspective of sensory experience, the cloud desktop terminal can insert one or more predicted frames between the received Nth image frame and the N+1th image frame to present the intermediate motion state of the desktop objects in the Nth image frame and the N+1th image frame, thereby making the display from the Nth image frame to the N+1th image frame smoother.

[0046] In summary, in this embodiment, the cloud desktop terminal is modified. After receiving the image frame sent by the cloud desktop server, the image frame can be displayed; and backward interpolation prediction can be performed based on the image frame and its previous reference frame to construct a predicted frame that can be inserted before the next image frame; based on this, the predicted frame can be displayed before the next image frame. In this way, the cloud desktop terminal can actively perform backward interpolation in addition to the normal display of the image frame obtained from the cloud desktop server. Therefore, when the transmission frame rate of the cloud desktop server is insufficient, the display frame rate can be autonomously increased in the cloud desktop terminal to improve the display smoothness of the cloud desktop without increasing bandwidth and delay.

[0047] In the above or below embodiments, various implementations may be used to predict whether the desktop object in the Nth image frame will only move linearly.

[0048] In an optional implementation, an evaluation condition may be predefined, based on which, if the Nth image frame satisfies the specified evaluation condition, it is predicted that the desktop object in the Nth image frame will only move linearly.

[0049] An exemplary evaluation condition may include a first condition and a second condition. The first condition includes the consistency of the image position of the desktop object that has changed in the Nth image frame and the N-1th image frame; the second condition includes no operation events other than scrolling and dragging occurring within a specified period of time before the Nth image frame is received. The presence of a change refers to differences between a desktop object in the current image frame and the previous image frame, including differences in content and / or position.

[0050] It's worth noting that in actual applications, since the time interval between two adjacent image frames is typically short, only a single keyboard or mouse operation event typically occurs. Accordingly, the desktop object that changes in both the Nth and N-1th image frames is typically a single object. Of course, in extremely rare cases, multiple desktop objects may change in both the Nth and N-1th image frames. In this case, each of the multiple changed desktop images can be evaluated separately. If all of these multiple desktop images meet the evaluation criteria, it can be determined that the desktop object in the Nth image frame undergoes only linear motion.

[0051] Based on this, referring to Figure 2, after receiving the Nth image frame, the interpolation engine in the cloud desktop terminal can first evaluate the Nth image frame according to the evaluation conditions. If the image positions of the changed desktop objects in the Nth image frame and the N-1th image frame are consistent; and within the specified time before receiving the Nth image frame, no other operation events except scrolling and dragging operations occur, then it can be determined that the desktop object in the Nth image frame will only move linearly.

[0052] In actual applications, the cloud desktop server can determine the image position of the desktop object that has changed in each image frame and send it to the cloud desktop terminal. In this way, the cloud desktop terminal can receive the Nth image frame sent by the cloud desktop server and the image position of the desktop object that has changed in the Nth image frame. Here, the image position refers to the pixel position of the desktop object in the image frame. Generally, the pixel position can be represented in the form of pixel coordinates, which is not limited here. For example, when the desktop object is a browser window, the image position refers to the pixel position of the browser window in the entire desktop screen. Of course, the cloud desktop terminal itself can also perform image comparison between image frames to independently determine the desktop object that has changed in the image frame, and is not limited to the above-mentioned implementation scheme of providing the image position by the cloud desktop server.

[0053] In this exemplary evaluation condition, if the Nth image frame satisfies the aforementioned exemplary evaluation condition, then the linear motion of the desktop object in the Nth image frame is characterized as page scrolling. In this case, the desktop object that changes is typically a window, such as a folder window, a browser window, etc.

[0054] To further ensure the accuracy of the evaluation, the aforementioned exemplary evaluation conditions may further include the requirement that multiple consecutive image frames prior to the Nth image frame meet both the first and second conditions. The multiple consecutive image frames prior to the Nth image frame refer to multiple consecutive image frames counted from the Nth image frame onward. For example, it may be determined whether the Nth image frame, the N-1th image frame, and the N-2th image frame all meet the aforementioned first and second conditions. If so, it may be determined that the desktop object in the Nth image frame only experiences linear motion.

[0055] Based on the above exemplary evaluation conditions, when the desktop object in the Nth image frame undergoes only linear motion such as page scrolling, interpolation may be performed to improve the display smoothness of the cloud desktop.

[0056] In addition to the above exemplary evaluation conditions, this embodiment may also support other evaluation conditions to discover other types of linear motions of the desktop object in the Nth image frame.

[0057] For example, the evaluation conditions may include a third condition and a fourth condition. The third condition may include that the desktop objects that have changed in the Nth image frame and the N-1th image frame are consistent; the second condition may include that no other operation events other than a single-direction movement operation have occurred within a specified period of time before the Nth image frame is received. Among them, a single-direction drag operation may include the movement operation of a window or icon performed by the direction keys of the keyboard. Of course, this exemplary evaluation condition may further include that multiple consecutive image frames before the Nth image frame all meet the third and fourth conditions. If the Nth image frame meets this exemplary evaluation condition, the linear movement of the desktop object in the Nth image frame is characterized as a unidirectional drag, and the desktop object that has changed is usually a window or icon. Based on this exemplary evaluation condition, when the desktop object in the Nth image frame only undergoes linear movement such as a unidirectional drag, interpolation can be performed to improve the display smoothness of the cloud desktop.

[0058] In this embodiment, the Nth image frame can be evaluated under multiple evaluation conditions at the same time. When the Nth image frame contains multiple desktop objects that are changing, if the evaluation conditions met by the multiple desktop objects are not exactly the same, it can also be determined that the desktop objects in the Nth image frame will only undergo linear motion.

[0059] Through the above optional implementation method, it is possible to predict whether the desktop objects in the image frame will only move linearly by analyzing some easily available information without performing motion analysis on the image frame, which can effectively reduce the computing pressure on the cloud desktop terminal.

[0060] In this embodiment, in addition to using a preset evaluation condition implementation method to predict whether the desktop object in the Nth image frame will only experience linear motion, other implementation methods may also be used. For example, the cloud desktop terminal may directly perform motion analysis on a limited number of image frames prior to the Nth image frame and, in combination with keyboard and mouse operations, predict whether the desktop object in the Nth image frame will only experience linear motion. This embodiment is not limited to this.

[0061] In the above or below embodiments, a variety of implementations may be used to construct a prediction frame.

[0062] In an optional implementation, motion estimation can be performed based on the N-1th image frame and the Nth image frame to determine the motion vector corresponding to the target desktop object that has changed in the Nth image frame; according to the motion vector, the display data corresponding to the target desktop object after linear motion of a specified length of time is calculated to construct a predicted frame.

[0063] It is worth noting that what is constructed here is a prediction frame that can be inserted between the Nth image frame and the N+1th image frame. The prediction frame reflects the intermediate motion state of the desktop object that undergoes linear motion in the Nth image frame, during the process of moving from "its position in the Nth image frame" to "its position in the N+1th image frame". To ensure this, optionally, in this embodiment, the time interval between the reception of the Nth image frame and the N+1th image frame can be estimated based on the transmission frame rate of the cloud desktop server. When constructing the prediction frame, it is sufficient to ensure that the motion duration referenced in the calculation does not exceed the time interval. To this end, the aforementioned specified duration may be less than the interval between the receipt moment corresponding to the Nth image frame and the receipt moment corresponding to the N+1th image frame.

[0064] In this implementation, the required predicted frames can be constructed based on motion estimation technology. The basic principle of motion estimation technology is to estimate the direction and distance of the object's motion based on the position change of the same object in two image frames, thereby generating a motion vector for the object. In this embodiment, various existing or future motion estimation algorithms can be used to perform the motion estimation operation herein, and the motion estimation algorithm used in this embodiment is not limited.

[0065] Among them, in this implementation, the desktop objects in two adjacent image frames can be compared. If a desktop object is different in the two image frames, it can be determined as the target desktop object in the subsequent image frame. This process can be completed in the cloud desktop server. Since the cloud desktop server can know the content in each image frame, this comparison operation has a sufficient execution basis for the cloud desktop server. Moreover, the cloud desktop server can provide the description information of the target desktop object that has changed in the Nth image frame to the cloud desktop terminal. The description information here may include but is not limited to the image position and / or object identification of the desktop object in the image frame. Of course, this process can also be completed in the cloud desktop terminal, which is not limited here.

[0066] In one exemplary approach, the Nth image frame is divided into several image blocks; motion estimation is performed on each of these image blocks. Specifically, for each image block in the Nth image frame, its position in the N-1th image frame is searched, and the relative offset between the positions of the same image block in the Nth and N-1th image frames is calculated; the resulting relative offset is commonly referred to as a motion vector. The size of the image blocks can be flexibly set as needed and is not limited here.

[0067] In this exemplary embodiment, motion estimation can be used to generate a motion vector corresponding to each image block in the Nth image frame. If multiple motion vectors are generated by motion estimation, a target motion vector that meets preset requirements can be selected from the multiple motion vectors and used as the motion vector corresponding to the target desktop object in the Nth image frame. Here, the target desktop object only undergoes linear motion, so the motion vector of the target desktop object should be single. In this exemplary implementation, a suitable single motion vector can be determined for the target desktop object.

[0068] This exemplary solution further proposes that the motion vectors corresponding to each image block contained in the Nth image frame can be deduplicated, and a suitable motion vector can be selected for the target desktop object in the Nth image frame from the candidate motion vectors remaining after deduplication. As mentioned above, the time interval between two adjacent image frames is very short, so the number of target desktop objects in the Nth image frame is typically one. Based on this, starting with any candidate motion vector remaining after deduplication, the number of image blocks suitable for the candidate motion vector surrounding the image block corresponding to the candidate motion vector can be detected to determine the number of image blocks corresponding to each candidate motion vector. From each candidate motion vector, the candidate motion vector corresponding to the largest number of image blocks is selected as the motion vector corresponding to the target desktop object. Of course, in extremely special cases, if there are multiple target desktop objects in the Nth image frame, the image blocks contained in each target desktop object can be determined separately, and the motion vectors associated with the image blocks contained in a single target desktop object can be used as candidate motion vectors for that target desktop object. The above-mentioned screening logic is then executed to determine a suitable motion vector for each target desktop object.

[0069] It is worth noting that, in this optional implementation, other solutions may be used to select a suitable motion vector for the target desktop object in the Nth image frame, and are not limited to the above exemplary solution.

[0070] In this optional implementation, after selecting an appropriate motion vector for the target desktop object in the Nth image frame, the display data corresponding to the target desktop object after linear motion for a specified duration can be calculated based on the motion vector to construct a predicted frame. In this way, motion simulation can be performed on the target desktop object based on the target desktop object's motion vector to determine the display data corresponding to the target desktop object in the predicted frame.

[0071] Here, for different types of linear motion, different calculation schemes may be used to calculate the display data. Taking the linear motion corresponding to the target desktop object as page scrolling as an example, an exemplary calculation scheme may be: determining the motion area from the image area corresponding to the target desktop object in the Nth image frame; calculating the motion position of each image block in the motion area after a specified time period according to the motion vector; determining the image block that is not completely blocked according to the calculated motion position; and using the image data in the image block that is not completely blocked as the display data. The motion area refers to the area of ​​the desktop object that is different between the Nth image frame and the N-1th image frame. The motion area is usually a part of the area in the desktop object. For example, when the desktop object is a browser window, the motion area is usually the page display area therein.

[0072] It can be seen that, according to the motion trend of the target desktop object, each image block in the moving area will continue to move linearly after the Nth image frame. However, since other areas of the target desktop object do not change, these areas may obstruct certain image blocks after the movement. To this end, in this exemplary calculation scheme, image blocks that are not completely obscured can be determined, and the image data in these blocks can be used as display data. Here, when the motion vector is known, different prediction frames can be constructed by adjusting the specified duration used. In practical applications, the motion vector may be represented by direction and distance. In this case, the specified duration can be converted into a distance coefficient to calculate the motion position of the image block. For example, if the motion vector is mv(x, y), and the specified duration is half the interval between the reception time corresponding to the Nth image frame and the reception time corresponding to the N+1th image frame, the motion position of the image block after the specified duration can be calculated based on the motion vector being mv(x / 2, y / 2).

[0073] For other types of linear motion, exemplary calculation schemes can also be provided. For example, for the aforementioned unidirectional dragging, the motion position of each image block corresponding to the target desktop object in the Nth image frame after a specified time length can be calculated according to the motion vector of the target desktop object, and the image data in all image blocks corresponding to the target desktop image will be retained, thereby constructing a predicted frame after the specified time length.

[0074] As mentioned above, the linear motion type of the target desktop object can be determined in step 101. Therefore, a calculation scheme suitable for the linear motion type of the target desktop object can be selected for constructing the prediction frame.

[0075] In summary, in this embodiment, multiple implementation methods can be used to construct a predicted frame for the Nth image frame, and the constructed predicted frame conforms to the linear motion trend of the target desktop object that will undergo linear motion in the Nth image frame. Therefore, the predicted frame can accurately display the intermediate motion state of the target desktop object. In this way, the predicted frame can be inserted into the Nth image frame and the N+1th image frame, thereby improving the display smoothness of the cloud desktop.

[0076] 3a to 3c are schematic diagrams of an application scenario provided by an exemplary embodiment of the present application. In this application scenario, the image frame frame2 received by the cloud desktop terminal from the cloud desktop server displays a browser window in full screen mode.

[0077] 1) The cloud desktop terminal receives frame 2 and can decode and display it immediately without increasing delay;

[0078] 2) The interpolation engine in the cloud desktop terminal can start working. The interpolation engine determines whether the image position of the desktop object (i.e., the browser window in the figure) that has changed in frame2 is consistent with that in frame1; determines whether no other operation events other than scrolling and dragging occurred between the reception of frame1 and the reception of frame2; and determines whether frame2, frame1, and frame0 all meet the above two conditions. If so, the engine proceeds to step 3).

[0079] 3) The interpolation engine can use frame1 and frame2 to predict and generate frame2.5, which is displayed before frame3 arrives.

[0080] 4) The cloud desktop terminal receives frame 3 and immediately decodes and displays it without increasing delay;

[0081] 5) Execute the judgment logic in step 2) here, if yes, continue to step 6).

[0082] 6) The interpolation engine can use frame2 and frame3 to predict and generate frame3.5, which is displayed before frame4 arrives.

[0083] 7) Repeat the above process.

[0084] Referring to Figure 3a, the browser windows in frame2 and frame1 are both full screen, so the image positions are consistent. Combined with the judgments under the other two conditions, in step 2) above, the judgment result is assumed to be yes. Continuing with Figure 3b, Figure 3b shows the motion vector determined by motion estimation between frameN and frameN-1, with the motion distance mv(x, y) and the motion direction indicated by the arrow in Figure 3b. Based on this, referring to the motion region determined in the browser window in Figure 3a (the area outlined by the black line), the motion vector can be used to calculate that the movement distance of this motion region relative to frameN in the predicted frame frameN+0.5 should be mv(x / 2, y / 2), with the motion direction remaining unchanged. Referring to Figure 3c, frameN+0.5 is obtained as shown. In frameN+0.5, there is a portion of the browser window with no display data. In this embodiment, this portion can be filled with white to serve as a fill area. However, it should be understood that in actual applications, the presence of this small white region in the predicted frame does not affect the user's visual experience.

[0085] In summary, interpolation can be implemented between image frames generated by page scrolling in the cloud desktop. This can autonomously increase the display frame rate in the cloud desktop terminal when the cloud desktop server reduces the transmission frame rate due to page scrolling, thereby improving the display smoothness of the cloud desktop. During the entire process, there is no need to increase any bandwidth, nor will it cause any display delay in the original image frame. In the application scenarios of Figures 3a to 3c, the display frame rate in the cloud desktop terminal can be doubled. It should be understood that when the number of single interpolated frames is greater, the display frame rate in the cloud desktop terminal can be further improved.

[0086] It should be noted that in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The sequence numbers of the operations, such as 101, 102, etc., are merely used to distinguish between different operations, and the sequence numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that descriptions such as "first" and "second" in this article are used to distinguish different conditions, etc., and do not represent a sequential order, nor do they limit "first" and "second" to being different types.

[0087] FIG4 is a schematic diagram of the structure of a cloud desktop terminal provided by another exemplary embodiment of the present application. As shown in FIG4 , the cloud desktop terminal includes: a memory 40 , a processor 41 , and a communication component 42 .

[0088] The processor 41 is coupled to the memory 40 and the communication component 42 and is configured to execute the computer program in the memory 40 to:

[0089] After receiving the Nth image frame sent by the cloud desktop server, displaying the Nth image frame;

[0090] Constructing a prediction frame preceding the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server;

[0091] Before receiving the N+1th image frame, displaying the predicted frame;

[0092] Wherein, N is an integer greater than 1.

[0093] In an optional embodiment, after receiving the Nth image frame sent by the cloud desktop server, the processor 41 may further be configured to:

[0094] Predict whether the desktop object in the Nth image frame will only move linearly;

[0095] If so, the operation of constructing a predicted frame located before the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server is performed.

[0096] In an optional embodiment, the processor 41 may also be configured to:

[0097] If the Nth image frame satisfies a specified evaluation condition, then predicting that the desktop object in the Nth image frame will only undergo linear motion;

[0098] Among them, the evaluation conditions include a first condition and a second condition, the first condition includes that the image positions of the desktop objects that have changed in the Nth image frame and the N-1th image frame are consistent; the second condition includes that no other operation events other than scrolling and dragging operations occur within a specified time period before receiving the Nth image frame.

[0099] In an optional embodiment, the evaluation condition further includes that a plurality of consecutive image frames before the Nth image frame all meet the first condition and the second condition.

[0100] In an optional embodiment, when the processor 41 constructs a predicted frame located before the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server, it can be specifically configured to:

[0101] performing motion estimation based on the N-1th image frame and the Nth image frame to determine a motion vector corresponding to a target desktop object that has changed in the Nth image frame;

[0102] Calculating display data corresponding to the target desktop object after linear motion for a specified period of time according to the motion vector to construct the prediction frame;

[0103] The specified duration is smaller than the interval between the time instant at which the Nth image frame is received and the time instant at which the N+1th image frame is received.

[0104] In an optional embodiment, when the processor 41 performs motion estimation based on the N-1th image frame and the Nth image frame to determine the motion vector corresponding to the target desktop object that has changed in the Nth image frame, it can be specifically configured to:

[0105] Dividing the Nth image frame into a plurality of image blocks;

[0106] performing motion estimation on each of the plurality of image blocks;

[0107] If there are multiple motion vectors generated by the motion estimation, a target motion vector that meets the preset requirements is selected from the multiple motion vectors as the motion vector corresponding to the target desktop object.

[0108] In an optional embodiment, if the Nth image frame contains only one target desktop object, the processor 41 may be configured to select a target motion vector that meets preset requirements from the multiple motion vectors as the motion vector corresponding to the target desktop object by:

[0109] Deduplication of motion vectors corresponding to respective image blocks contained in the Nth image frame;

[0110] Starting from any candidate motion vector remaining after deduplication, detecting the number of image blocks applicable to the candidate motion vector around the image block corresponding to the candidate motion vector, so as to determine the number of image blocks corresponding to each candidate motion vector;

[0111] From each candidate motion vector, a target candidate motion vector corresponding to the largest number of image blocks is selected as the motion vector corresponding to the target desktop object.

[0112] In an optional embodiment, if the linear motion corresponding to the target desktop object is page scrolling, the processor 41 may be specifically configured to calculate the display data corresponding to the target desktop object after the target desktop object has moved for a specified period of time according to the motion vector:

[0113] determining a motion area from an image area corresponding to the target desktop object in the Nth image frame;

[0114] Calculating the motion position of each image block in the motion area after the specified time period according to the motion vector;

[0115] According to the calculated motion position, determine the image block that is not completely blocked;

[0116] The image data in the image block that is not completely blocked is used as the display data.

[0117] In an optional embodiment, the desktop object is a window or an icon, and the linear motion includes page scrolling or unidirectional dragging.

[0118] In an optional embodiment, the cloud desktop server determines the image position of the desktop object with changes in each image frame and sends it to the cloud desktop terminal.

[0119] Furthermore, as shown in Figure 4, the cloud desktop terminal may also include other components such as a display 43, a power supply component 44, and an audio component 45. Figure 4 only schematically shows some components, which does not mean that the cloud desktop terminal only includes the components shown in Figure 4.

[0120] It is worth noting that the technical details in the above-mentioned embodiments of the cloud desktop terminal can be referred to the relevant description in the aforementioned method embodiment. In order to save space, they will not be repeated here, but this should not cause any loss of the protection scope of this application.

[0121] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which can implement the steps in the above method embodiment when the computer program is executed.

[0122] The memory in Figure 4 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, images, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0123] The communication component in Figure 4 above is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0124] The display in FIG4 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0125] The power supply assembly in Figure 4 provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0126] The audio component in Figure 4 above can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0127] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0128] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0129] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0131] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0132] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0133] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.

Claims

1. A cloud desktop display method, applicable to a cloud desktop terminal, the method include: After receiving the Nth image frame sent by the cloud desktop server, displaying the Nth image frame; Constructing a prediction frame located before the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server; Before receiving the N+1th image frame, displaying the predicted frame; Wherein, N is an integer greater than 1.

2. The method according to claim 1, after receiving the Nth image frame sent by the cloud desktop server, further include: Predicting whether the desktop object in the Nth image frame will only move linearly; If so, the operation of constructing a predicted frame located before the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server is performed.

3. The method according to claim 2, further comprising: include: If the Nth image frame satisfies the specified evaluation condition, it is predicted that the desktop object in the Nth image frame will only move linearly; Among them, the evaluation conditions include a first condition and a second condition, the first condition includes that the image positions of the desktop objects that have changed in the Nth image frame and the N-1th image frame are consistent; the second condition includes that no other operation events except scrolling and dragging operations occur within a specified time period before receiving the Nth image frame. 4 . The method according to claim 3 , wherein the evaluation condition further comprises that a plurality of consecutive image frames before the Nth image frame all meet the first condition and the second condition.

5. The method according to claim 1, wherein the predicted frame located before the N+1th image frame is constructed based on the Nth image frame and the N-1th image frame received from the cloud desktop server, include: Performing motion estimation based on the N-1th image frame and the Nth image frame to determine a motion vector corresponding to a target desktop object that has changed in the Nth image frame; According to the motion vector, the display data corresponding to the target desktop object after the linear motion of the specified duration is calculated to construct the prediction frame; The specified duration is smaller than the interval between the time instant at which the Nth image frame is received and the time instant at which the N+1th image frame is received.

6. The method according to claim 5, wherein the motion estimation is performed based on the N-1th image frame and the Nth image frame to determine the motion vector corresponding to the target desktop object that has changed in the Nth image frame, include: Dividing the Nth image frame into a plurality of image blocks; Performing motion estimation on each of the plurality of image blocks; If there are multiple motion vectors generated by motion estimation, a target motion vector that meets preset requirements is selected from the multiple motion vectors as the motion vector corresponding to the target desktop object.

7. The method according to claim 6, if the target desktop object contained in the Nth image frame is one, then selecting a target motion vector that meets preset requirements from the multiple motion vectors as the motion vector corresponding to the target desktop object, include: Deduplication of motion vectors corresponding to respective image blocks contained in the Nth image frame; Starting from any candidate motion vector remaining after deduplication, detecting the number of image blocks applicable to the candidate motion vector around the image block corresponding to the candidate motion vector, so as to determine the number of image blocks corresponding to each candidate motion vector; From the candidate motion vectors, the target candidate motion vector corresponding to the maximum number of image blocks is selected as the motion vector corresponding to the target desktop object.

8. The method according to claim 5, if the linear motion corresponding to the target desktop object is page scrolling, then according to the motion vector, the display data corresponding to the target desktop object after the motion for a specified time is calculated, include: Determine a motion area from an image area corresponding to the target desktop object in the Nth image frame; According to the motion vector, respectively calculating the motion position of each image block in the motion area after the specified time length; According to the calculated motion position, determine the image block that is not completely blocked; The image data in the image block that is not completely blocked is used as the display data.

9. The method according to claim 2, wherein the desktop object is a window or an icon, and the linear motion comprises page scrolling or unidirectional dragging.

10. The method according to claim 3, wherein the cloud desktop server determines the image position of the desktop object with changes in each image frame and sends it to the cloud desktop terminal.

11. A cloud desktop terminal, comprising a memory, a processor and a communication component; The memory is used to store one or more computer instructions; The processor is coupled to the memory and the communication component, and is used to execute the one or more computer instructions to execute the cloud desktop display method according to any one of claims 1-10.

12. A computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes the one or more processors to execute the cloud desktop display method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Image processing method and related product

    CN111083417A

  • Image processing method and device, electronic equipment and computer readable storage medium

    CN112596843A

  • Display device and gesture track recognition method

    CN114637439A

  • Frame insertion method in interface display process and terminal equipment

    CN114764357A

  • Cloud desktop display method, cloud desktop terminal and storage medium

    CN116719592A