Cloud desktop display method, cloud desktop terminal and storage medium

Abstract

The embodiment of the present application provides a cloud desktop display method, a cloud desktop terminal and a storage medium. In the embodiment of the present application, the cloud desktop terminal is modified. After receiving the image frame sent by the cloud desktop server, the image frame can be displayed; and backward interpolation prediction can be performed based on the image frame and its previous reference frame to construct a predicted frame that can be inserted before the next image frame; based on this, the predicted frame can be displayed before the next image frame. In this way, the cloud desktop terminal can actively perform backward interpolation in addition to the normal display of the image frame obtained from the cloud desktop server, so that when the transmission frame rate of the cloud desktop server is insufficient, the display frame rate can be autonomously increased in the cloud desktop terminal to improve the display smoothness of the cloud desktop without increasing bandwidth and delay.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a cloud desktop display method, a cloud desktop terminal, and a storage medium. Background Art

[0002] Cloud desktops, also known as desktop virtualization or cloud computing, are a new alternative to traditional computers. With cloud desktops, users no longer need to purchase a mainframe computer. All of the mainframe's components are virtualized on a backend server, while the frontend devices are primarily used for interactive operations. Through the collaboration of the frontend devices and backend servers, the user experience is consistent with that of a computer.

[0003] Currently, the front-end display frame rate and back-end transmission frame rate of the cloud desktop are consistent. When the cloud desktop undergoes drastic changes, the back-end transmission frame rate often drops due to network congestion, and the front-end display frame rate will also drop synchronously, resulting in insufficient display smoothness of the cloud desktop. Summary of the Invention

[0004] Various aspects of the present application provide a cloud desktop display method, a cloud desktop terminal, and a storage medium to improve the display smoothness of the cloud desktop.

[0005] The present invention provides a cloud desktop display method applicable to a cloud desktop terminal, the method comprising:

[0006] After receiving the Nth image frame sent by the cloud desktop server, displaying the Nth image frame;

[0007] Constructing a prediction frame preceding the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server;

[0008] Before receiving the N+1th image frame, displaying the predicted frame;

[0009] Wherein, N is an integer greater than 1.

[0010] The embodiment of the present application also provides a cloud desktop terminal, including a memory, a processor and a communication component;

[0011] The memory is used to store one or more computer instructions;

[0012] The processor is coupled to the memory and the communication component, and is configured to execute the one or more computer instructions to execute the aforementioned cloud desktop display method.

[0013] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the aforementioned cloud desktop display method.

[0014] In an embodiment of the present application, the cloud desktop terminal is modified. After receiving an image frame sent by a cloud desktop server, the image frame can be displayed; and backward interpolation prediction can be performed based on the image frame and its previous reference frame to construct a predicted frame that can be inserted before the next image frame; based on this, the predicted frame can be displayed before the next image frame. In this way, the cloud desktop terminal can actively perform backward interpolation in addition to the normal display of the image frame obtained from the cloud desktop server, so that when the transmission frame rate of the cloud desktop server is insufficient, the display frame rate can be autonomously increased in the cloud desktop terminal to improve the display smoothness of the cloud desktop without increasing bandwidth and delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 A flowchart of a cloud desktop display method provided by an exemplary embodiment of the present application;

[0017] Figure 2 A logical diagram of a cloud desktop display method provided by an exemplary embodiment of the present application;

[0018] Figure 3a to Figure 3c A schematic diagram of an application scenario provided by an exemplary embodiment of the present application;

[0019] Figure 4 A schematic structural diagram of a cloud desktop terminal provided in another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] At present, due to the instability of the transmission frame rate, the display smoothness of the cloud desktop is insufficient. For this reason, in some embodiments of the present application: the cloud desktop terminal is modified. After receiving the image frame sent by the cloud desktop server, the image frame can be displayed; and backward interpolation prediction can be performed based on the image frame and its previous reference frame to construct a predicted frame that can be inserted before the next image frame; based on this, the predicted frame can be displayed before the next image frame. In this way, the cloud desktop terminal can actively perform backward interpolation in addition to the normal display of the image frame obtained from the cloud desktop server, so that when the transmission frame rate of the cloud desktop server is insufficient, the display frame rate can be autonomously increased in the cloud desktop terminal to improve the display smoothness of the cloud desktop without increasing bandwidth and delay.

[0022] The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0023] Figure 1 This is a flow chart of a cloud desktop display method provided by an exemplary embodiment of the present application. The method can be executed by a data processing device. The data processing device can be implemented as software, hardware, or a combination of software and hardware. The data processing device can be integrated into a cloud desktop terminal. Figure 1 , the method may include:

[0024] Step 100: After receiving the Nth image frame sent by the cloud desktop server, display the Nth image frame;

[0025] Step 101: construct a prediction frame before the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server;

[0026] Step 102: Display the predicted frame before receiving the N+1th image frame;

[0027] Wherein, N is an integer greater than 1.

[0028] The cloud desktop display method provided in this embodiment can be applied to various application scenarios using cloud desktops, and this embodiment does not limit the application scenarios.

[0029] Cloud desktop technology primarily involves cloud desktop servers and cloud desktop terminals. Data exchange between the cloud desktop server and the cloud desktop terminal relies on a network. The cloud desktop terminal can transmit keyboard and mouse operation data to the cloud desktop server. The cloud desktop server, based on this keyboard and mouse operation data, simulates the cloud desktop screen and transmits the screen back to the cloud desktop terminal. The cloud desktop server typically transmits the cloud desktop screen to the cloud desktop terminal via a video stream. In this embodiment, the cloud desktop screen is described as an image frame.

[0030] During the research process, the inventor found that the transmission frame rate of the cloud desktop server is unstable. Especially when a window is dragged or a page is scrolled, the cloud desktop changes drastically, which will cause a large amount of data to need to be encoded and transmitted over the network. The transmission frame rate of the cloud desktop server will decrease, and the display frame rate in the cloud desktop terminal will also decrease synchronously, resulting in insufficient display smoothness of the cloud desktop.

[0031] To this end, this embodiment innovatively proposes a cloud desktop display solution, in which the cloud desktop terminal performs frame insertion at an appropriate time to improve the display smoothness of the cloud desktop.

[0032] Figure 2 This is a logical diagram of a cloud desktop display method provided by an exemplary embodiment of the present application. Figure 2 A frame insertion engine can be added to the cloud desktop terminal to execute the relevant logical steps of the cloud desktop display method provided in this embodiment. The frame insertion engine can communicate with the existing client in the cloud desktop terminal. In this embodiment, the client in the cloud desktop terminal can trigger the frame insertion engine to execute the cloud desktop display method provided in this embodiment each time it receives an image frame from the cloud desktop server.

[0033] refer to Figure 1 In step 100, after receiving the current image frame sent by the cloud desktop server, the current image frame can be displayed. For ease of description, in this embodiment, the current frame is described as the Nth image frame. It should be understood that N can be any integer greater than 1. For the image frames received by the cloud desktop terminal from the cloud desktop server, these image frames can be displayed according to the traditional display scheme in this embodiment, that is, after receiving these image frames, the display processing can be directly performed. In addition, in this embodiment, the transmission frame rate of the cloud desktop server is not interfered with, and the cloud desktop terminal can increase its own display frame rate as needed.

[0034] Continue to refer Figure 1 In step 101, a prediction frame located before the N+1th image frame may be constructed based on the Nth image frame and the N-1th image frame received from the cloud desktop server.

[0035] In this embodiment, the Nth image frame and the N-1th image frame can be used as references for backward frame prediction. In this way, the predicted frames can be used to supplement the display of the intermediate state of the relevant desktop objects between the Nth image frame and the N+1th image frame. The relevant desktop objects here refer to the desktop objects appearing in the Nth image frame and the desktop objects newly added in the N+1th image frame.

[0036] In a preferred implementation, it is proposed that: after receiving the Nth image frame sent by the cloud desktop, it is also possible to predict whether the desktop objects in the Nth image frame will only undergo linear motion. Here, the desktop objects in the image frame may include but are not limited to various display windows and display icons, etc., which are not limited here. There may be multiple desktop objects contained in the image frame, and in this implementation, the above-mentioned prediction operation can be performed on all desktop objects contained in the image frame. What is predicted here is whether each desktop object in the Nth image frame will only undergo linear motion next. Linear motion can also be understood as straight-line motion, which means that the motion trajectory of the object is a straight line. In this embodiment, a variety of implementation methods can be used to complete the above-mentioned prediction operation, and the specific implementation method will be described in detail later.

[0037] In this implementation, if the desktop object in the Nth image frame is predicted to undergo only linear motion, the aforementioned backward interpolation operation is performed. Since the desktop object in the Nth image frame is predicted to undergo only linear motion, backward frame prediction can be accurately performed based on the linear motion trend of the desktop object. The constructed predicted frame conforms to the linear motion trend of the desktop object in the Nth image frame and can be well connected to the N+1th image frame, ensuring smooth image processing.

[0038] Of course, in addition to the preferred implementation described above, other triggering conditions can also be used in this embodiment to trigger the backward frame interpolation operation in step 101, such as a user proactively initiating a frame interpolation start instruction in the cloud desktop terminal. It is also possible that the cloud desktop terminal performs the aforementioned backward frame interpolation operation for each image frame received from the cloud desktop server without configuring a triggering condition. Both of these can effectively improve the display frame rate in the cloud desktop terminal, and this embodiment is not limited thereto.

[0039] The specific implementation method of constructing the prediction frame will be described in detail later.

[0040] In addition, in this embodiment, the prediction frame constructed in step 101 may be one or more, and this embodiment does not limit the number of prediction frames.

[0041] On this basis, reference Figure 1 In step 102, the predicted frame can be displayed before the N+1th image frame is received. In practical applications, the display timing of the predicted frame can be determined based on the motion moment represented by the predicted frame. For example, if the predicted frame represents an intermediate state 0.1ms after the Nth image frame, the predicted frame can be displayed 0.1ms after the Nth image frame is displayed. If multiple predicted frames are constructed in step 101, the display timings corresponding to the multiple predicted frames can also be determined accordingly, so that the multiple predicted frames can be displayed sequentially.

[0042] After receiving the N+1th image frame sent by the cloud desktop server, the N+1th image frame can be displayed. In this way, from the perspective of sensory experience, the cloud desktop terminal can insert one or more predicted frames between the received Nth image frame and the N+1th image frame to present the intermediate motion state of the desktop objects in the Nth image frame and the N+1th image frame, thereby making the display from the Nth image frame to the N+1th image frame smoother.

[0043] In summary, in this embodiment, the cloud desktop terminal is modified. After receiving the image frame sent by the cloud desktop server, the image frame can be displayed; and backward interpolation prediction can be performed based on the image frame and its previous reference frame to construct a predicted frame that can be inserted before the next image frame; based on this, the predicted frame can be displayed before the next image frame. In this way, the cloud desktop terminal can actively perform backward interpolation in addition to the normal display of the image frame obtained from the cloud desktop server. Therefore, when the transmission frame rate of the cloud desktop server is insufficient, the display frame rate can be autonomously increased in the cloud desktop terminal to improve the display smoothness of the cloud desktop without increasing bandwidth and delay.

[0044] In the above or below embodiments, various implementations may be used to predict whether the desktop object in the Nth image frame will only move linearly.

[0045] In an optional implementation, an evaluation condition may be predefined, based on which, if the Nth image frame satisfies the specified evaluation condition, it is predicted that the desktop object in the Nth image frame will only move linearly.

[0046] An exemplary evaluation condition may include a first condition and a second condition. The first condition includes the consistency of the image position of the desktop object that has changed in the Nth image frame and the N-1th image frame; the second condition includes no operation events other than scrolling and dragging occurring within a specified period of time before the Nth image frame is received. The presence of a change refers to differences between a desktop object in the current image frame and the previous image frame, including differences in content and / or position.

[0047] It's worth noting that in actual applications, since the time interval between two adjacent image frames is typically short, only a single keyboard or mouse operation event typically occurs. Accordingly, the desktop object that changes in both the Nth and N-1th image frames is typically a single object. Of course, in extremely rare cases, multiple desktop objects may change in both the Nth and N-1th image frames. In this case, each of the multiple changed desktop images can be evaluated separately. If all of these multiple desktop images meet the evaluation criteria, it can be determined that the desktop object in the Nth image frame undergoes only linear motion.

[0048] Based on this, reference Figure 2 After receiving the Nth image frame, the interpolation engine in the cloud desktop terminal can first evaluate the Nth image frame according to the evaluation conditions. If the image positions of the desktop objects that have changed in the Nth image frame and the N-1th image frame are consistent; and no other operation events except scrolling and dragging operations occur within the specified time before receiving the Nth image frame, it can be determined that the desktop objects in the Nth image frame will only move linearly.

[0049] In actual applications, the cloud desktop server can determine the image position of the desktop object that has changed in each image frame and send it to the cloud desktop terminal. In this way, the cloud desktop terminal can receive the Nth image frame sent by the cloud desktop server and the image position of the desktop object that has changed in the Nth image frame. Here, the image position refers to the pixel position of the desktop object in the image frame. Generally, the pixel position can be represented in the form of pixel coordinates, which is not limited here. For example, when the desktop object is a browser window, the image position refers to the pixel position of the browser window in the entire desktop screen. Of course, the cloud desktop terminal itself can also perform image comparison between image frames to independently determine the desktop object that has changed in the image frame, and is not limited to the above-mentioned implementation scheme of providing the image position by the cloud desktop server.

[0050] In this exemplary evaluation condition, if the Nth image frame satisfies the aforementioned exemplary evaluation condition, then the linear motion of the desktop object in the Nth image frame is characterized as page scrolling. In this case, the desktop object that changes is typically a window, such as a folder window, a browser window, etc.

[0051] To further ensure the accuracy of the evaluation, the aforementioned exemplary evaluation conditions may further include the requirement that multiple consecutive image frames prior to the Nth image frame meet both the first and second conditions. The multiple consecutive image frames prior to the Nth image frame refer to multiple consecutive image frames counted from the Nth image frame onward. For example, it may be determined whether the Nth image frame, the N-1th image frame, and the N-2th image frame all meet the aforementioned first and second conditions. If so, it may be determined that the desktop object in the Nth image frame only experiences linear motion.

[0052] Based on the above exemplary evaluation conditions, when the desktop object in the Nth image frame undergoes only linear motion such as page scrolling, interpolation may be performed to improve the display smoothness of the cloud desktop.

[0053] In addition to the above exemplary evaluation conditions, this embodiment may also support other evaluation conditions to discover other types of linear motions of the desktop object in the Nth image frame.

[0054] For example, the evaluation conditions may include a third condition and a fourth condition. The third condition may include that the desktop objects that have changed in the Nth image frame and the N-1th image frame are consistent; the second condition may include that no other operation events other than a single-direction movement operation have occurred within a specified period of time before the Nth image frame is received. Among them, a single-direction drag operation may include the movement operation of a window or icon performed by the direction keys of the keyboard. Of course, this exemplary evaluation condition may further include that multiple consecutive image frames before the Nth image frame all meet the third and fourth conditions. If the Nth image frame meets this exemplary evaluation condition, the linear movement of the desktop object in the Nth image frame is characterized as a unidirectional drag, and the desktop object that has changed is usually a window or icon. Based on this exemplary evaluation condition, when the desktop object in the Nth image frame only undergoes linear movement such as a unidirectional drag, interpolation can be performed to improve the display smoothness of the cloud desktop.

[0055] In this embodiment, the Nth image frame can be evaluated under multiple evaluation conditions at the same time. When the Nth image frame contains multiple desktop objects that are changing, if the evaluation conditions met by the multiple desktop objects are not exactly the same, it can also be determined that the desktop objects in the Nth image frame will only undergo linear motion.

[0056] Through the above optional implementation method, it is possible to predict whether the desktop objects in the image frame will only move linearly by analyzing some easily available information without performing motion analysis on the image frame, which can effectively reduce the computing pressure on the cloud desktop terminal.

[0057] In this embodiment, in addition to using a preset evaluation condition implementation method to predict whether the desktop object in the Nth image frame will only experience linear motion, other implementation methods may also be used. For example, the cloud desktop terminal may directly perform motion analysis on a limited number of image frames prior to the Nth image frame and, in combination with keyboard and mouse operations, predict whether the desktop object in the Nth image frame will only experience linear motion. This embodiment is not limited to this.

[0058] In the above or below embodiments, a variety of implementations may be used to construct a prediction frame.

[0059] In an optional implementation, motion estimation can be performed based on the N-1th image frame and the Nth image frame to determine the motion vector corresponding to the target desktop object that has changed in the Nth image frame; according to the motion vector, the display data corresponding to the target desktop object after linear motion of a specified length of time is calculated to construct a predicted frame.

[0060] It is worth noting that what is constructed here is a prediction frame that can be inserted between the Nth image frame and the N+1th image frame. The prediction frame reflects the intermediate motion state of the desktop object that undergoes linear motion in the Nth image frame, during the process of moving from "its position in the Nth image frame" to "its position in the N+1th image frame". To ensure this, optionally, in this embodiment, the time interval between the reception of the Nth image frame and the N+1th image frame can be estimated based on the transmission frame rate of the cloud desktop server. When constructing the prediction frame, it is sufficient to ensure that the motion duration referenced in the calculation does not exceed the time interval. To this end, the aforementioned specified duration may be less than the interval between the receipt moment corresponding to the Nth image frame and the receipt moment corresponding to the N+1th image frame.

[0061] In this implementation, the required predicted frames can be constructed based on motion estimation technology. The basic principle of motion estimation technology is to estimate the direction and distance of the object's motion based on the position change of the same object in two image frames, thereby generating a motion vector for the object. In this embodiment, various existing or future motion estimation algorithms can be used to perform the motion estimation operation herein, and the motion estimation algorithm used in this embodiment is not limited.

[0062] Among them, in this implementation, the desktop objects in two adjacent image frames can be compared. If a desktop object is different in the two image frames, it can be determined as the target desktop object in the subsequent image frame. This process can be completed in the cloud desktop server. Since the cloud desktop server can know the content in each image frame, this comparison operation has a sufficient execution basis for the cloud desktop server. Moreover, the cloud desktop server can provide the description information of the target desktop object that has changed in the Nth image frame to the cloud desktop terminal. The description information here may include but is not limited to the image position and / or object identification of the desktop object in the image frame. Of course, this process can also be completed in the cloud desktop terminal, which is not limited here.

[0063] In one exemplary approach, the Nth image frame is divided into several image blocks; motion estimation is performed on each of these image blocks. Specifically, for each image block in the Nth image frame, its position in the N-1th image frame is searched, and the relative offset between the positions of the same image block in the Nth and N-1th image frames is calculated; the resulting relative offset is commonly referred to as a motion vector. The size of the image blocks can be flexibly set as needed and is not limited here.

[0064] In this exemplary embodiment, motion estimation can be used to generate a motion vector corresponding to each image block in the Nth image frame. If multiple motion vectors are generated by motion estimation, a target motion vector that meets preset requirements can be selected from the multiple motion vectors and used as the motion vector corresponding to the target desktop object in the Nth image frame. Here, the target desktop object only undergoes linear motion, so the motion vector of the target desktop object should be single. In this exemplary implementation, a suitable single motion vector can be determined for the target desktop object.

[0065] This exemplary solution further proposes that the motion vectors corresponding to each image block contained in the Nth image frame can be deduplicated, and a suitable motion vector can be selected for the target desktop object in the Nth image frame from the candidate motion vectors remaining after deduplication. As mentioned above, the time interval between two adjacent image frames is very short, so the number of target desktop objects in the Nth image frame is typically one. Based on this, starting with any candidate motion vector remaining after deduplication, the number of image blocks suitable for the candidate motion vector surrounding the image block corresponding to the candidate motion vector can be detected to determine the number of image blocks corresponding to each candidate motion vector. From each candidate motion vector, the candidate motion vector corresponding to the largest number of image blocks is selected as the motion vector corresponding to the target desktop object. Of course, in extremely special cases, if there are multiple target desktop objects in the Nth image frame, the image blocks contained in each target desktop object can be determined separately, and the motion vectors associated with the image blocks contained in a single target desktop object can be used as candidate motion vectors for that target desktop object. The above-mentioned screening logic is then executed to determine a suitable motion vector for each target desktop object.

[0066] It is worth noting that, in this optional implementation, other solutions may be used to select a suitable motion vector for the target desktop object in the Nth image frame, and are not limited to the above exemplary solution.

[0067] In this optional implementation, after selecting a suitable motion vector for the target desktop object in the Nth image frame, the display data corresponding to the target desktop object after linear motion for a specified duration can be calculated based on the motion vector to construct a predicted frame. In this way, motion simulation can be performed on the target desktop object based on the motion vector to determine the display data corresponding to the target desktop object in the predicted frame.

[0068] Here, for different types of linear motion, different calculation schemes may be used to calculate the display data. Taking the linear motion corresponding to the target desktop object as page scrolling as an example, an exemplary calculation scheme may be: determining the motion area from the image area corresponding to the target desktop object in the Nth image frame; calculating the motion position of each image block in the motion area after a specified time period according to the motion vector; determining the image block that is not completely blocked according to the calculated motion position; and using the image data in the image block that is not completely blocked as the display data. The motion area refers to the area of ​​the desktop object that is different between the Nth image frame and the N-1th image frame. The motion area is usually a part of the area in the desktop object. For example, when the desktop object is a browser window, the motion area is usually the page display area therein.

[0069] It can be seen that, according to the motion trend of the target desktop object, each image block in the moving area will continue to move linearly after the Nth image frame. However, since other areas of the target desktop object do not change, these areas may obstruct certain image blocks after the movement. To this end, in this exemplary calculation scheme, image blocks that are not completely obscured can be determined, and the image data in these blocks can be used as display data. Here, when the motion vector is known, different prediction frames can be constructed by adjusting the specified duration used. In practical applications, the motion vector may be represented by direction and distance. In this case, the specified duration can be converted into a distance coefficient to calculate the motion position of the image block. For example, if the motion vector is mv(x, y), and the specified duration is half the interval between the reception time corresponding to the Nth image frame and the reception time corresponding to the N+1th image frame, the motion position of the image block after the specified duration can be calculated based on the motion vector being mv(x / 2, y / 2).

[0070] For other types of linear motion, exemplary calculation schemes can also be provided. For example, for the aforementioned unidirectional dragging, the motion position of each image block corresponding to the target desktop object in the Nth image frame after a specified time length can be calculated according to the motion vector of the target desktop object, and the image data in all image blocks corresponding to the target desktop image will be retained, thereby constructing a predicted frame after the specified time length.

[0071] As mentioned above, the linear motion type of the target desktop object can be determined in step 101. Therefore, a calculation scheme suitable for the linear motion type of the target desktop object can be selected for constructing the prediction frame.

[0072] In summary, in this embodiment, multiple implementation methods can be used to construct a predicted frame for the Nth image frame, and the constructed predicted frame conforms to the linear motion trend of the target desktop object that will undergo linear motion in the Nth image frame. Therefore, the predicted frame can accurately display the intermediate motion state of the target desktop object. In this way, the predicted frame can be inserted into the Nth image frame and the N+1th image frame, thereby improving the display smoothness of the cloud desktop.

[0073] Figure 3a to Figure 3c This is a schematic diagram of an application scenario provided by an exemplary embodiment of the present application. In this application scenario, the image frame frame2 received by the cloud desktop terminal from the cloud desktop server displays a browser window in full screen mode.

[0074] 1) The cloud desktop terminal receives frame 2 and can decode and display it immediately without increasing delay;

[0075] 2) The interpolation engine in the cloud desktop terminal can start working. The interpolation engine determines whether the image position of the desktop object (i.e., the browser window in the figure) that has changed in frame2 is consistent with that in frame1; determines whether no other operation events other than scrolling and dragging occurred between the reception of frame1 and the reception of frame2; and determines whether frame2, frame1, and frame0 all meet the above two conditions. If so, the engine proceeds to step 3).

[0076] 3) The interpolation engine can use frame1 and frame2 to predict and generate frame2.5, which is displayed before frame3 arrives.

[0077] 4) The cloud desktop terminal receives frame 3 and immediately decodes and displays it without increasing delay;

[0078] 5) Execute the judgment logic in step 2) here, if yes, continue to step 6).

[0079] 6) The interpolation engine can use frame2 and frame3 to predict and generate frame3.5, which is displayed before frame4 arrives.

[0080] 7) Repeat the above process.

[0081] refer to Figure 3a , the browser windows in frame2 and frame1 are both full screen, so the image positions are consistent. Combined with the judgment under the other two conditions, in the above step 2), assuming that the judgment result is yes. Then continue to refer to Figure 3b , Figure 3bThe motion vector determined by the motion estimation between frameN and frameN-1 is shown in FIG. 1 , where the motion distance is mv(x, y) and the motion direction is as follows: Figure 3b As shown by the arrow in the middle, based on this, you can refer to Figure 3a The moving area (the area framed by black lines) determined in the browser window can be calculated based on the motion vector. The moving distance of the moving area in the predicted frame frameN+0.5 relative to frameN should be mv(x / 2, y / 2), and the moving direction remains unchanged. Figure 3c , we can obtain frame N+0.5 as shown in the figure. In frame N+0.5, there will be a small area in the browser window with no display data. In this embodiment, this area can be filled with white as a fill area. However, it should be understood that in actual applications, the presence of this small white area in the predicted frame does not affect the user's visual experience.

[0082] In summary, interpolation can be implemented between image frames generated by page scrolling in the cloud desktop. This can automatically increase the display frame rate in the cloud desktop terminal when the cloud desktop server reduces the transmission frame rate due to page scrolling, thereby improving the display smoothness of the cloud desktop. During the entire process, no bandwidth needs to be increased, and no display delay will occur in the original image frames. Figure 3a to Figure 3c In the application scenario, the display frame rate in the cloud desktop terminal can be doubled. It should be understood that when the number of single inserted frames is larger, the display frame rate in the cloud desktop terminal can be further improved.

[0083] It should be noted that in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The sequence numbers of the operations, such as 101, 102, etc., are merely used to distinguish between different operations, and the sequence numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that descriptions such as "first" and "second" in this article are used to distinguish different conditions, etc., and do not represent a sequential order, nor do they limit "first" and "second" to being different types.

[0084] Figure 4 This is a schematic diagram of the structure of a cloud desktop terminal provided by another exemplary embodiment of the present application. Figure 4 As shown, the cloud desktop terminal includes: a memory 40, a processor 41 and a communication component 42.

[0085] The processor 41 is coupled to the memory 40 and the communication component 42 and is configured to execute the computer program in the memory 40 to:

[0086] After receiving the Nth image frame sent by the cloud desktop server, displaying the Nth image frame;

[0087] Constructing a prediction frame preceding the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server;

[0088] Before receiving the N+1th image frame, displaying the predicted frame;

[0089] Wherein, N is an integer greater than 1.

[0090] In an optional embodiment, after receiving the Nth image frame sent by the cloud desktop server, the processor 41 may further be configured to:

[0091] Predict whether the desktop object in the Nth image frame will only move linearly;

[0092] If so, the operation of constructing a predicted frame located before the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server is performed.

[0093] In an optional embodiment, the processor 41 may also be configured to:

[0094] If the Nth image frame satisfies a specified evaluation condition, then predicting that the desktop object in the Nth image frame will only undergo linear motion;

[0095] Among them, the evaluation conditions include a first condition and a second condition, the first condition includes that the image positions of the desktop objects that have changed in the Nth image frame and the N-1th image frame are consistent; the second condition includes that no other operation events other than scrolling and dragging operations occur within a specified time period before receiving the Nth image frame.

[0096] In an optional embodiment, the evaluation condition further includes that a plurality of consecutive image frames before the Nth image frame all meet the first condition and the second condition.

[0097] In an optional embodiment, when the processor 41 constructs a predicted frame located before the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server, it can be specifically configured to:

[0098] performing motion estimation based on the N-1th image frame and the Nth image frame to determine a motion vector corresponding to a target desktop object that has changed in the Nth image frame;

[0099] Calculating display data corresponding to the target desktop object after linear motion for a specified period of time according to the motion vector to construct the prediction frame;

[0100] The specified duration is smaller than the interval between the time instant at which the Nth image frame is received and the time instant at which the N+1th image frame is received.

[0101] In an optional embodiment, when the processor 41 performs motion estimation based on the N-1th image frame and the Nth image frame to determine the motion vector corresponding to the target desktop object that has changed in the Nth image frame, it can be specifically configured to:

[0102] Dividing the Nth image frame into a plurality of image blocks;

[0103] performing motion estimation on each of the plurality of image blocks;

[0104] If there are multiple motion vectors generated by the motion estimation, a target motion vector that meets the preset requirements is selected from the multiple motion vectors as the motion vector corresponding to the target desktop object.

[0105] In an optional embodiment, if the Nth image frame contains only one target desktop object, the processor 41 may be configured to select a target motion vector that meets preset requirements from the multiple motion vectors as the motion vector corresponding to the target desktop object by:

[0106] Deduplication of motion vectors corresponding to respective image blocks contained in the Nth image frame;

[0107] Starting from any candidate motion vector remaining after deduplication, detecting the number of image blocks applicable to the candidate motion vector around the image block corresponding to the candidate motion vector, so as to determine the number of image blocks corresponding to each candidate motion vector;

[0108] From each candidate motion vector, a target candidate motion vector corresponding to the largest number of image blocks is selected as the motion vector corresponding to the target desktop object.

[0109] In an optional embodiment, if the linear motion corresponding to the target desktop object is page scrolling, the processor 41 may be specifically configured to calculate the display data corresponding to the target desktop object after the target desktop object has moved for a specified period of time according to the motion vector:

[0110] determining a motion area from an image area corresponding to the target desktop object in the Nth image frame;

[0111] Calculating the motion position of each image block in the motion area after the specified time period according to the motion vector;

[0112] According to the calculated motion position, determine the image block that is not completely blocked;

[0113] The image data in the image block that is not completely blocked is used as the display data.

[0114] In an optional embodiment, the desktop object is a window or an icon, and the linear motion includes page scrolling or unidirectional dragging.

[0115] In an optional embodiment, the cloud desktop server determines the image position of the desktop object with changes in each image frame and sends it to the cloud desktop terminal.

[0116] Further, if Figure 4 As shown, the cloud desktop terminal may also include: a display 43, a power supply component 44, an audio component 45 and other components. Figure 4 Only some components are shown schematically, which does not mean that the cloud desktop terminal only includes Figure 4 Components shown.

[0117] It is worth noting that the technical details in the above-mentioned embodiments of the cloud desktop terminal can be referred to the relevant description in the aforementioned method embodiment. In order to save space, they will not be repeated here, but this should not cause any loss of the protection scope of this application.

[0118] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which can implement the steps in the above method embodiment when the computer program is executed.

[0119] above Figure 4 The memory in the computer is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0120] above Figure 4The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0121] above Figure 4 The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0122] above Figure 4 The power supply component in a device provides power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0123] above Figure 4 The audio component in the device may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0124] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0128] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0130] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.

Claims

1. A cloud desktop display method, applicable to a cloud desktop terminal, comprising: After receiving the Nth image frame sent by the cloud desktop server, display the Nth image frame, where N is an integer greater than 1; Dividing the Nth image frame into a plurality of image blocks, and performing motion estimation on each of the plurality of image blocks; If there are multiple motion vectors generated by motion estimation and there is only one target desktop object included in the Nth image frame, deduplication is performed on the motion vectors corresponding to the respective image blocks included in the Nth image frame; wherein the target desktop object is a desktop object that has changed between the N-1th image frame and the Nth image frame; Starting from any candidate motion vector remaining after deduplication, detecting the number of image blocks applicable to the candidate motion vector around the image block corresponding to the candidate motion vector, so as to determine the number of image blocks corresponding to each candidate motion vector; Selecting, from each candidate motion vector, a target candidate motion vector corresponding to the largest number of image blocks as the motion vector corresponding to the target desktop object; Calculating, according to the motion vector, display data corresponding to the target desktop object after linear motion for a specified duration to construct a predicted frame; wherein the specified duration is less than an interval between a time instant at which the Nth image frame is received and a time instant at which the N+1th image frame is received; Before receiving the N+1th image frame, the predicted frame is displayed.

2. The method according to claim 1, after receiving the Nth image frame sent by the cloud desktop server, further comprising: Predicting whether the desktop object in the Nth image frame only undergoes linear motion; If so, the operation of constructing a predicted frame located before the N+1th image frame based on the Nth image frame and the N-1th image frame received from the cloud desktop server is performed.

3. The method according to claim 2, further comprising: If the Nth image frame satisfies a specified evaluation condition, then predicting that the desktop object in the Nth image frame will only undergo linear motion; Among them, the evaluation conditions include a first condition and a second condition, the first condition includes that the image positions of the desktop objects that have changed in the Nth image frame and the N-1th image frame are consistent; the second condition includes that no other operation events other than scrolling and dragging operations occur within a specified time period before receiving the Nth image frame. 4 . The method according to claim 3 , wherein the evaluation condition further comprises that a plurality of consecutive image frames before the Nth image frame all meet the first condition and the second condition.

5. The method according to claim 1 , wherein if the linear motion corresponding to the target desktop object is page scrolling, calculating display data corresponding to the target desktop object after the motion for a specified duration according to the motion vector comprises: determining a motion area from an image area corresponding to the target desktop object in the Nth image frame; Calculating the motion position of each image block in the motion area after the specified time period according to the motion vector; According to the calculated motion position, determine the image block that is not completely blocked; The image data in the image block that is not completely blocked is used as the display data. 6 . The method according to claim 2 , wherein the desktop object is a window or an icon, and the linear motion comprises page scrolling or unidirectional dragging.

7. The method according to claim 3, wherein the cloud desktop server determines the image position of the desktop object with changes in each image frame and sends the position to the cloud desktop terminal.

8. A cloud desktop terminal comprising a memory, a processor, and a communication component; The memory is used to store one or more computer instructions; The processor is coupled to the memory and the communication component, and is configured to execute the one or more computer instructions to execute the cloud desktop display method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes the one or more processors to execute the cloud desktop display method according to any one of claims 1 to 7.

Images