Data processing method and device for cloud desktop or cloud application

Abstract

The present disclosure relates to a data processing method and apparatus for a cloud desktop or cloud application. In response to receiving a first input event from a client, the method predicts subsequent input events from the client to obtain an input sequence, the input sequence including at least one input event; obtains an output sequence corresponding to the input sequence, the output sequence including output data corresponding to at least some of the input events in the input sequence; and sends the output sequence to the client. This reduces the round-trip time (RTT) delay of multiple round-trip interactions and improves the smoothness of user operations.

Description

Technical Field

[0001] The present disclosure relates to the technical field of cloud computing, and in particular to a data processing method and device for cloud desktops or cloud applications. Background Art

[0002] As cloud computing becomes more and more mature, the current mainstream development trend is to configure program modules originally located on terminal devices (i.e., front-end devices) on the server side to achieve slimming down of terminal devices.

[0003] Based on this, products such as cloud desktops and cloud applications have been derived.

[0004] Cloud desktop, also known as desktop virtualization or cloud computer, is a new model to replace traditional computers.

[0005] Cloud desktop products mainly consist of two parts: front-end equipment and back-end servers.

[0006] The front-end device mainly uses a thin client (such as a device similar to a TV set-top box) to connect to a monitor, keyboard and / or mouse. After the user installs the client, he accesses the virtual machine host on the back-end server through a unique communication protocol to realize interactive operations, achieving the same experience as a computer.

[0007] Cloud desktops not only replace traditional computers but also support internet access from other smart devices like mobile phones and tablets, making them an effective solution for mobile office work. With cloud desktops, users no longer need to purchase a mainframe computer. The mainframe's CPU, memory, hard drive, and other components are all virtualized on the backend server. A single high-performance server can host anywhere from 1 to 50 virtual hosts.

[0008] Applications configured to run on the server, such as programs that require high device performance, are called cloud applications.

[0009] The characteristic of cloud applications is that although corresponding applications can be installed on terminal devices, the main data processing of these applications is implemented through cloud applications rather than applications on terminal devices, so as to achieve local application implementation through cloud applications without the terminal device user's perception.

[0010] In cloud desktop or cloud application scenarios, how to reduce the processing delay of continuous user input operations is a technical problem that urgently needs to be solved. Summary of the Invention

[0011] A technical problem to be solved by the present disclosure is to provide a data processing solution capable of reducing the processing delay of continuous input operations.

[0012] According to a first aspect of the present disclosure, a data processing method for a cloud desktop or cloud application is provided, which is suitable for execution on a server that provides a cloud desktop service or a cloud application service to a client. The method comprises: in response to receiving a first input event from a client, predicting an input event of the client after the first input event to obtain an input sequence, wherein the input sequence includes at least one input event; obtaining an output sequence corresponding to the input sequence, wherein the output sequence includes output data corresponding to at least part of the input events in the input sequence; and sending the output sequence to the client.

[0013] According to a second aspect of the present disclosure, a data processing method is provided, suitable for execution on a server side, the method comprising: in response to receiving a first input event from a client, predicting input events of the client subsequent to the first input event to obtain an input sequence, the input sequence including at least one input event; obtaining an output sequence corresponding to the input sequence, the output sequence including output data corresponding to at least some of the input events in the input sequence; and sending the output sequence to the client.

[0014] Optionally, the method also includes: in response to receiving a second input event from the client after receiving the first event, determining whether the second input event matches the current target input event in the input sequence; if the second input event matches the current target input event in the input sequence, taking the first input event in the input sequence that follows the current target input event as the new current target input event.

[0015] Optionally, before determining whether the second input event matches the prediction result of the first input sequence, the method further includes: determining whether the second input event satisfies an input tracking condition, where the input tracking condition is used to characterize the conditions that the current input event needs to meet when predicting an input event after the current input event; if the second input event satisfies the input tracking condition, determining whether the second input event complies with a prediction rule, where the prediction rule is used to characterize the rules that the predicted input event complies with; if the second input event complies with the prediction rule, executing the step of determining whether the second input event matches the current target input event in the input sequence.

[0016] Optionally, the method further includes: if the second input event does not meet the input tracking condition, obtaining output data corresponding to the second input event and sending the output data to the client; and / or if the second input event does not meet the prediction rule, updating the no-hit time and / or no-hit count of the input sequence.

[0017] Optionally, the method further includes: if the second input event does not comply with the prediction rule, determining whether the input sequence complies with the release rule, where the release rule is used to characterize the rule that needs to be satisfied to invalidate the input sequence; if the input sequence complies with the release rule, deleting the input sequence and / or the output sequence corresponding to the input sequence.

[0018] Optionally, the method also includes: if the second input event does not match the current target input event in the input sequence, updating the number of hits and / or hit duration, the number of hits is used to characterize the number of input events that meet the prediction rules but do not match the input sequence, and the hit duration is used to characterize the duration of the input events that meet the prediction rules but do not match the input sequence; if the number of hits is greater than or equal to the first threshold, or the hit duration is greater than or equal to the second threshold, predicting the input event of the client after the second input event to obtain a new input sequence, and obtaining the output sequence of the new input sequence, and sending the new output sequence to the client.

[0019] Optionally, the first input event is an operation event for a cloud application or cloud desktop, the input sequence is a set of operation events obtained by predicting the operation events performed by the client on the cloud application or cloud desktop after the first input event, and the output sequence includes an operation result screen corresponding to each operation event in the predicted set of operation events.

[0020] According to a third aspect of the present disclosure, a data processing method is provided, suitable for execution by a client, the method comprising: sending a first input event to a server; receiving an output sequence sent by the server, the output sequence comprising output data corresponding to at least some of the input events in the input sequence, the input sequence being a prediction by the server of an input event of the client subsequent to the first input event, the input sequence comprising at least one input event.

[0021] Optionally, the method further comprises: in response to receiving a second input event after the first input event, outputting the output data in the output sequence in sequence.

[0022] Optionally, the method further includes: sending the second input event to the server; if a new output sequence sent by the server is received, deleting the existing output sequence and retaining the new output sequence.

[0023] According to a fourth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in any one of the first to third aspects above.

[0024] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising an executable code. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described in any one of the first to third aspects above.

[0025] According to a sixth aspect of the present disclosure, a non-temporary machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in any one of the first to third aspects above.

[0026] Therefore, the present disclosure predicts the input events of the client after the first input event, obtains the output sequence corresponding to the predicted input sequence, and sends the output sequence to the client, so as to reduce the RTT delay of multiple round-trip interactions and improve the fluency of user operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0028] Figure 1 A schematic diagram comparing the effects of this solution and the existing solution is shown;

[0029] Figure 2 A schematic flowchart of a data processing method according to an embodiment of the present disclosure is shown.

[0030] Figure 3 The figure shows a schematic diagram of the process of the server processing a new input event received.

[0031] Figure 4 A schematic diagram showing the principle of the data processing mechanism of existing cloud applications / cloud desktops is shown.

[0032] Figure 5 A schematic diagram illustrating the principles of a data processing mechanism for cloud applications / cloud desktops based on the present disclosure is shown.

[0033] Figure 6 A schematic structural diagram of a data processing device according to an embodiment of the present disclosure is shown.

[0034] Figure 7 A structural schematic diagram of a data processing device according to another embodiment of the present disclosure is shown.

[0035] Figure 8 A schematic structural diagram of a computing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0037] The common feature of cloud desktops and cloud applications is that after the client detects an operation event input by the user, it needs to send the operation event to the server and obtain the processing result of the operation event from the server.

[0038] According to the existing operation event processing mechanism, the delay from when a user inputs an operation event to when the operation event processing result is displayed on the client is RTT (Round-Trip Time). If a user inputs N operation events continuously, the total delay is N × RTT. The total delay when inputting continuous operations is high, and the user experience is not smooth.

[0039] In response to this, the present disclosure proposes that when the server detects a user input event (i.e., an operation event corresponding to an input behavior), it can predict the user's subsequent input events, obtain a predicted sequence of input events, and pre-generate an output sequence corresponding to the predicted sequence (i.e., batch output data, such as batch updating the screen), and send the batch output data to the client to reduce the RTT delay of multiple back-and-forth interactions and improve the smoothness of user operations.

[0040] Figure 1 The figure shows a schematic diagram comparing the effects of this solution and the existing solution.

[0041] like Figure 1 As shown, for N consecutive user inputs, the total delay according to the existing operation event processing mechanism is N × RTT. However, with the data processing solution disclosed in this disclosure, the total delay for N operation events can be reduced to 1 × RTT. Compared with the existing solution, the total delay is greatly reduced, thereby improving the user's operation experience.

[0042] Figure 2 A schematic flowchart of a data processing method according to an embodiment of the present disclosure is shown. Figure 2 The method shown can be executed by a server that can provide cloud desktop services or cloud application services to a client.

[0043] See also Figure 2 In step S210 , in response to receiving a first input event from a client, input events of the client subsequent to the first input event are predicted to obtain an input sequence.

[0044] A first input event refers to an operation event input by a user through a client, such as an operation performed by the user on the client using a tool such as a keyboard, mouse, or touch screen. After receiving the first input event, the client may send the first input event to the server. After receiving the first input event, the server may predict operation events that the user may input through the client after the first input event, thereby obtaining an input sequence.

[0045] An input sequence includes at least one input event. It reflects a set of possible input events that a user may enter after entering a first input event through the client. The input events in an input sequence can have a specific order, which reflects the predicted order in which the input events will occur.

[0046] In step S220, an output sequence corresponding to the input sequence is obtained.

[0047] The output sequence includes output data corresponding to at least some of the input events in the input sequence. For example, the output sequence may include output data corresponding to each input event in the input sequence.

[0048] The first input event may or may not be included in the input sequence.

[0049] When the first input event is not included in the input sequence, the server can process the first input event and each input event in the input sequence separately (such as parallel processing) to obtain the processing results of the first input event and each input event in the input sequence.

[0050] The processing result refers to the output data of the input event. In cloud desktops or cloud applications, the output data of the input event usually refers to the updated interface in response to the input event, that is, the updated screen of the interface.

[0051] Taking the first input event as an operation event for a cloud application or cloud desktop (such as a picture viewing event) as an example, the input sequence can be a set of operation events (such as a backward scrolling picture event) obtained by predicting the operation events performed by the client on the cloud application or cloud desktop after the first input event, and the output sequence can include an operation result screen corresponding to each operation event in the predicted set of operation events (such as a batch update screen).

[0052] In step S230, the output sequence is sent to the client.

[0053] The output data of the first input event and the output sequence may be sent to the client together.

[0054] Considering that predicting the input sequence may take some time, after receiving the first input event, the prediction of subsequent input events and the acquisition of the output data of the first input event can be performed simultaneously. That is, the output data of the first input event can be sent to the client first. After the input sequence is predicted and the output sequence corresponding to the input sequence is acquired, the output sequence can be sent to the client. This reduces both the user's initial operation latency and the overall latency of subsequent operations.

[0055] For a single operation event input by the user through the client, by pre-evaluating the user's subsequent continuous actions on the server, obtaining the output data of batch estimated actions on the server in advance, and sending the output data to the client in batches at one time, the RTT delay of multiple back-and-forth interactions of the user can be reduced, thereby improving the smoothness of the operation.

[0056] The input event predicted in step S210 and the first input event may be for a cloud application or cloud desktop, or may be for a functional module within the same cloud application or cloud desktop. That is, based on the cloud application or cloud desktop (or a functional module within the cloud application or cloud desktop) involved in the first input event, the user's next or multiple steps of operation with respect to the cloud application or cloud desktop (or a functional module within the cloud application or cloud desktop) may be predicted.

[0057] In cloud applications, the placement of application icons, such as menu bars, toolbars, and navigation bars, is relatively fixed, and user operations are more streamlined and patterned. Therefore, the user behavior characteristics collected for a specific cloud application are more distinct, and the model is more convergent, resulting in higher accuracy of the input sequence used to predict user behavior.

[0058] When executing step S210, the user's operation intention can be analyzed with reference to the behavioral characteristics of the first input event and the working mechanism of the cloud application or cloud desktop involved in the first input event, and then a set of input events (i.e., input sequence) corresponding to the analyzed operation intention can be generated. The behavioral characteristics of the first input event may include but are not limited to the coordinate position of the first input event, the operation type of the first input event (such as click, double-click, slide, long press, etc.); the working mechanism of the cloud application or cloud desktop may include but is not limited to the operation type supported by the cloud application or cloud desktop, the operation range, the operation position and the correspondence between the type and the event type, etc.

[0059] Before executing step S210, it may also be determined whether the first input event satisfies an input tracking condition. The input tracking condition is used to represent a condition that the current input event needs to satisfy when predicting an input event after the current input event.

[0060] As an example, the input tracking conditions may include but are not limited to any one or more of the following combinations: sub-condition a1, the object targeted by the current input event is of a predetermined type (such as only a specific cloud application or cloud desktop); sub-condition a2, the type of the current input event matches the set of events of interest (such as the set of events of interest is an operation event that will cause the interface to update); sub-condition a3, the coordinates of the current input event match the coordinate range of interest.

[0061] For example, if the cloud application targeted by the current input event is an image browser (i.e., satisfies sub-condition a1), the current input event belongs to the image switching event of interest (i.e., satisfies sub-condition a2), and the operation of the current input event (such as a mouse click) is located within the image area (i.e., satisfies sub-condition a3), then it indicates that the current input event meets the input tracking condition.

[0062] If the first input event meets the input tracking condition, step S210 may be executed.

[0063] If the first input event does not meet the input tracking condition, it can be processed according to the existing event processing flow, that is, no prediction is performed and only the processing result of the first input event (ie, output data) is sent to the client.

[0064] The first input event that meets the input tracking condition can be called a tracking event. The input sequence can be regarded as the prediction result of the tracking event, that is, the prediction result of the user operation intention determined based on the first input event.

[0065] It should be noted that, after receiving one or more first input events from the client, the server may predict subsequent input events of the client to obtain an input sequence.

[0066] In other words, the generation of the input sequence may depend on one or more first input events received previously.

[0067] That is, it may be necessary to track multiple consecutive input events to obtain the input sequence.

[0068] For example, after the user clicks "Next" once in the application help wizard, he clicks "Next" again. The server can infer that the user will click Next three times based on these two clicks (assuming that the user help wizard has a total of five steps). At this time, the input sequence is predicted by the server based on the two click events received from the user.

[0069] The server's processing mechanism for new input events

[0070] When the client receives a new input event after the first input event, the new input event can be sent to the server, and the server determines whether the new input event matches the input sequence. If it matches, it indicates that the input sequence meets the user's true intention, and the output sequence sent can meet the client's data update requirements in response to the new input event. Conversely, if it does not match, it indicates that the input sequence does not meet the user's true intention, and the output sequence sent cannot meet the client's data update requirements in response to the new input event. When the new input event does not match the input sequence, the new input event can be used as the first input event, and the above steps S210 to S230 are executed to re-predict the input sequence corresponding to the input event, obtain the output sequence corresponding to the new input sequence, and send the new output sequence to the client so that the client replaces the old output sequence with the new output sequence for output.

[0071] The following combination Figure 3 The following further explains the processing mechanism when the server receives a new input event.

[0072] See also Figure 3 , in step S310, the next input event is received.

[0073] The next input event refers to an input event received from the client after the first event is received. For ease of description, the next input event may be referred to as a second input event.

[0074] In response to receiving a second input event from the client after receiving the first event, step S350 may be directly executed to determine whether the second input event matches the input sequence. Alternatively, the determination process of steps S315, S320, S330, and S335 may be executed before executing step S350.

[0075] In step S315 , it is determined whether the second input event meets the input tracking condition.

[0076] For input tracking conditions, please refer to the relevant description above. If the second input event does not meet the input tracking conditions, it indicates that the second input event is not an event that needs to be tracked. At this time, the process returns to step S310 and waits for the next input event. As described above, the output data corresponding to the second input event can be obtained according to the existing processing mechanism and sent to the client.

[0077] If the second input event meets the input tracking condition, step S320 can be executed to determine whether the tracked input event list is empty. The tracked input event list is used to record the tracked input events (i.e., the first input events). Each tracked event recorded in the input tracking event list has a corresponding tracking result (i.e., input sequence and / or output sequence). One or more tracked events can be recorded in the tracked input event list. Different tracked events can correspond to different cloud applications. The same cloud application can also correspond to one or more tracked events.

[0078] If the tracked input event list is empty, step S325 can be executed to record the tracked event. Thus, the second input event can be used as the event to be tracked, predict the input events of the client after the second input event, obtain a new input sequence, and obtain the output sequence of the new input sequence, and send the new output sequence to the client.

[0079] If the tracked input event list is not empty, step S330 can be executed to compare the second input event with the existing tracked events.

[0080] Each tracked event corresponds to an input sequence. The input sequence reflects the tracking result of the user operation intention determined based on the first input event. Comparing the second input event with the existing tracked events is to compare the prediction rules of the input sequences of the second input event and the existing tracked events to determine whether the second input event hits (i.e., conforms to) the prediction rule (i.e., step S335). Among them, when comparing the input event with the existing tracked events, the input event can be compared with the existing tracked events for the same application as the input event.

[0081] The prediction rule is used to characterize the rule that the predicted input event conforms to, that is, the rule that the input events in the input sequence conform to. As an example, the prediction rule can include, but is not limited to, any one or more of the following combinations: sub-rule b1, the type of the input event matches the type of the expected input event; sub-rule b2, the coordinates of the input event match the expected coordinates; sub-rule b3, the time interval between the input event and the previous input event is less than or equal to the expected time interval.

[0082] Taking the cloud application as a picture browser as an example, when the user switches pictures, when the second input event is a mouse click (conforming to sub-rule b1), the mouse click is within the screen right page-turning area position (conforming to sub-rule b2), and the interval between the mouse click and the previous event < N milliseconds (conforming to sub-rule b3). When all the above sub-rules are satisfied, it indicates that the second input event conforms to the prediction rule, and the second input event matches the tracking intention or tracked event represented by the predicted input sequence.

[0083] If the second input event meets the input tracking conditions and conforms to the prediction rules, it indicates that the second input event preliminarily meets the tracking intent determined when tracking the tracking event. At this point, step S350 can be executed to further determine whether the second input event matches the input sequence. If the second input event matches the input sequence, the event can be internally consumed, eliminating the need to obtain and send the output data of the second input event to the client.

[0084] When determining whether the second input event matches the input sequence, it can be determined whether there is an input event corresponding to (i.e., the same as) the second input event in the input sequence. If there is an input event corresponding to the second input event in the input sequence, it can be considered that the second input event matches the input sequence.

[0085] In one embodiment of the present disclosure, after receiving an output sequence corresponding to an input sequence, the client can output the output data in the output sequence in sequence in response to a new input event. That is, one output data is output each time an input event is received. In this case, considering that the input event and the output data in the output sequence have a strict correspondence in sequence, if the input event made by the user is inconsistent with the input event in the input sequence in sequence, the user experience may be confusing. Therefore, when judging whether the second input event matches the input sequence, it can be judged whether the second input event matches the current target input event in the input sequence. If the second input event matches the current target input event in the input sequence, step S360 is executed to use the first input event in the input sequence after the current target input event as the new current target input event to calibrate the input sequence. The current target input event may refer to an input event in the input sequence that corresponds to the output data that the client currently wants to output.

[0086] For example, if the input sequence includes four input events, for the first input event received after the first input event, it can be determined whether the input event matches (i.e., is the same as) the first input event in the input sequence. If so, it indicates that the predicted first input event in the input sequence is consistent with the actual situation. Thereafter, when a new input event is received, the new input event can be compared with the predicted second input event in the input sequence.

[0087] If the second input event does not match the input sequence, step S355 may be executed to update the hit count and / or hit duration. The hit count is used to indicate the number of input events that meet the prediction rules but do not match the input sequence, and the hit duration is used to indicate the duration of the input event that meets the prediction rules but does not match the input sequence.

[0088] In step S365, it is determined whether the cumulative hits have reached an upper limit.

[0089] It can be determined whether the number of hits is greater than or equal to a first threshold, and / or whether the hit duration is greater than or equal to a second threshold. If the number of hits is greater than or equal to the first threshold, or the hit duration is greater than or equal to the second threshold, it indicates that the cumulative hits have reached the upper limit.

[0090] If the cumulative hits reach the upper limit, it indicates that the accuracy of the previously predicted input event is low. At this time, step S370 can be executed to predict a new input sequence to update the input sequence. The second input event can be executed as the new first input event. Figure 2 Specifically, the input event after the second input event of the client can be predicted to obtain a new input sequence, and the output sequence of the new input sequence is obtained, and the new output sequence is sent to the client.

[0091] If the judgment results of both step S335 and step S365 are negative, the process proceeds to step S340 to determine whether there is a next tracking event that has not yet been compared. If so, the process returns to step S330. If not, the process proceeds to step S345 to update the tracking event no-hit time. The tracking event no-hit time is used to represent the duration of time that a tracking event (i.e., the input sequence corresponding to the tracking event) has not hit an input event.

[0092] In step S380, it is determined whether the input sequence complies with the release rule.

[0093] The release rule is used to represent the rule that needs to be met to invalidate the input sequence. When the input sequence meets the release rule, step S390 can be executed to release the tracking event.

[0094] Releasing a tracking event also means releasing the tracking result of the tracking event (i.e., the input sequence). Releasing an input sequence also invalidates the input sequence and / or the output sequence corresponding to the input sequence, such as by deleting the input sequence and / or the output sequence corresponding to the input sequence. When an input sequence is invalidated, the output sequence corresponding to the input sequence stored on the client can also be invalidated. For example, when the server determines that the input sequence is invalid, it can send a command to the client to delete the currently stored output sequence corresponding to the input sequence.

[0095] As an example, the release rules may include but are not limited to any one or more of the following combinations: sub-rule c1, the type of the current input event does not match the type of the expected input event; and / or sub-rule c2, the coordinates of the current input event exceed the expected coordinate range; and / or sub-rule c3, the time interval between the current input event and the previous input event exceeds the expected time interval.

[0096] For example, let's assume the cloud application is an image browser and the input sequence is the tracking sequence for the "image page turning" input event. If the user changes the mouse click action to something else while switching between consecutive images (sub-rule c1), or the mouse click is in a non-page turning area (sub-rule c2), or the interval between the mouse click and the previous event is greater than N seconds (sub-rule c3), the input sequence is released if all of these sub-rules are met.

[0097] Client processing mechanism

[0098] The client can send a first input event to the server and receive an output sequence sent by the server.

[0099] The output sequence includes output data corresponding to at least some (eg, individual) input events in the input sequence. The input sequence is a set of input events predicted by the server for the input events after the first input event of the client.

[0100] The input sequence includes at least one input event. The input sequence may include the first input event or may not include the first input event. If the input sequence does not include the first input event, the client may also receive output data of the first input event sent by the server.

[0101] The client can receive the output data and output sequence of the first input event sent by the server at the same time. The client can also receive the output data of the first input event sent by the server first, and then receive the output sequence sent by the server.

[0102] For the output data and output sequence of the received first input event, the client can first output the output data of the first input event. Then, in response to detecting an operation event input by the user through the keyboard, mouse, touch screen, etc., the client can output the output data in the output sequence in sequence, without having to wait for a new round of RTT delay to obtain the output data from the server. In other words, in response to receiving a new input event (the second input event) after the first input event, the output data in the output sequence can be output in sequence.

[0103] For the output data and output sequence received for the first input event, the client may first output the output data of the first input event, and then automatically output the output data in the output sequence in order at predetermined intervals. Thus, if the output series of output data can match the user's actual operation intention, the user no longer needs to perform manual operations using tools such as a keyboard, mouse, or touch screen.

[0104] In response to receiving (detecting) a second input event after the first input event, the client can also send the second input event to the server, which will determine whether the output sequence previously sent to the client matches the second input event. If a new output sequence is received from the server, it indicates that the previously received output sequence does not match the second input event. In this case, the client can delete the existing output sequence and retain the new output sequence.

[0105] Figure 4 A schematic diagram showing the principle of the data processing mechanism of existing cloud applications / cloud desktops is shown.

[0106] like Figure 4 As shown in the figure, from a user input (keyboard, mouse, touchpad, etc.) on a cloud application or cloud desktop to seeing the screen updated, this complete RTT needs to go through the following steps:

[0107] 1. The protocol end (ASP Client) program sends a user input event (Input Event) through the network;

[0108] 2. The protocol service (ASP Server) program receives user input events through the network;

[0109] 3. The protocol service program sends the input event to the Guest Agent;

[0110] 4. The Guest Agent sends the input event to the system;

[0111] 5. The response time of the application, the rendering engine updates the screen;

[0112] 6. Receive system callback and capture screen updates;

[0113] 7. Send the updated content to the protocol service program (ASP Server);

[0114] 8. The protocol service (ASP Server) program sends the updated screen through the network;

[0115] 9. The protocol end (ASP Client) program receives the updated screen through the network;

[0116] The final updated image is decoded and rendered and presented to the user.

[0117] The Guest Agent is a helper daemon installed in the Guest OS. For Shadowless Cloud Desktop / Cloud Applications, it exchanges information between the ASP Server and the Guest OS. The ASP (Adaptive Streaming Protocol) is a desktop transmission protocol. The Guest OS is the operating system for the cloud desktop.

[0118] Traditional solutions require each user action to go through the entire chain, with the total latency of each link equal to 1 RTT. For multiple user events, when there are N user actions and N screen updates, the total latency is N*RTT.

[0119] Figure 5 A schematic diagram illustrating the principles of a data processing mechanism for cloud applications / cloud desktops based on the present disclosure is shown.

[0120] like Figure 5 As shown, the present disclosure improves upon the existing process by adding a user behavior detection module and a user behavior prediction module to the ASP server. The user behavior prediction module can predict a sequence of events based on a single input event and output it to the guest agent. When the ASP server receives an updated screen, it caches it. During the process of sending it to the client, the detection module can be notified one by one to correct the send sequence count.

[0121] In high-bandwidth dedicated cloud network environments and cloud application scenarios, this disclosure detects user input behavior (keyboard / mouse events), infers the next user input based on historical input behavior, and pre-evaluates the continuity of user actions. It sends the estimated input behavior to the Guest OS in advance, caches the updated screens requested by the Guest OS in the protocol, and sends the updated screens to the user in batches, reducing the RTT delay of multiple back-and-forth user interactions, improving the smoothness of operations and screens, and enhancing the user experience.

[0122] The data processing method of the present disclosure can also be implemented as a data processing device. The functional units of the data processing device can be implemented by hardware, software, or a combination of hardware and software that implements the principles of the present disclosure. It can be understood by those skilled in the art that the present disclosure is combined with Figure 6 、 Figure 7 The functional units described can be combined or divided into sub-units to implement the principles of the above invention. Therefore, the description herein can support any possible combination, division, or further limitation of the functional units described herein.

[0123] The following is a brief description of the functional units that a data processing device may have and the operations that each functional unit may perform. For the details involved, please refer to the relevant description above and will not be repeated here.

[0124] Figure 6 A schematic structural diagram of a data processing device according to an embodiment of the present disclosure is shown.

[0125] See also Figure 6 The data processing device 600 may include a prediction module 610, an acquisition module 620, and a sending module 630. The data processing device 600 may be deployed in a server for providing cloud desktop or cloud application services to a client.

[0126] Prediction module 610 is configured to, in response to receiving a first input event from a client, predict subsequent input events from the client to obtain an input sequence, the input sequence comprising at least one input event. Acquisition module 620 is configured to obtain an output sequence corresponding to the input sequence, the output sequence comprising output data corresponding to at least some of the input events in the input sequence. Transmission module 630 is configured to transmit the output sequence to the client.

[0127] The data processing device 600 may further include a first judgment module and a correction module. The first judgment module is configured to, in response to receiving a second input event from the client after receiving the first event, determine whether the second input event matches the current target input event in the input sequence; and the correction module is configured to, if the second input event matches the current target input event in the input sequence, set the first input event in the input sequence that follows the current target input event as the new current target input event.

[0128] The data processing device 600 may further include a second judgment module and a third judgment module. The second judgment module is configured to determine whether the second input event satisfies an input tracking condition, which represents a condition that the current input event must meet when predicting an input event subsequent to the current input event. The third judgment module is configured to determine whether the second input event complies with a prediction rule, which represents a rule that the predicted input event complies with, if the second input event satisfies the input tracking condition. The first judgment module is configured to determine whether the second input event matches the current target input event in the input sequence, if the second input event complies with the prediction rule.

[0129] The acquisition module 620 may further acquire output data corresponding to the second input event if the second input event does not satisfy the input tracking condition, and send the output data to the client.

[0130] The data processing device 600 may further include a first updating module configured to update the no-hit time and / or the number of no-hit times of the input sequence if the second input event does not conform to the prediction rule.

[0131] The data processing device 600 may further include a fourth judgment module and a deletion module. The fourth judgment module is configured to determine whether the input sequence complies with a release rule if the second input event does not comply with the prediction rule, wherein the release rule is used to represent the rule that must be met to invalidate the input sequence; and the deletion module is configured to delete the input sequence and / or the output sequence corresponding to the input sequence if the input sequence complies with the release rule.

[0132] The data processing device 600 may further include a second updating module. The second updating module is used to update the hit count and / or hit duration if the second input event does not match the current target input event in the input sequence. The hit count is used to represent the number of input events that meet the prediction rules but do not match the input sequence, and the hit duration is used to represent the duration of the input event that meets the prediction rules but does not match the input sequence.

[0133] If the number of hits is greater than or equal to the first threshold, or the hit duration is greater than or equal to the second threshold, the prediction module 610 can predict the input event of the client after the second input event to obtain a new input sequence, the acquisition module 620 can obtain the output sequence of the new input sequence, and the sending module 630 can send the new output sequence to the client.

[0134] Figure 7 A structural schematic diagram of a data processing device according to another embodiment of the present disclosure is shown.

[0135] See also Figure 7 The data processing device 700 may include a sending module 710 and a receiving module 720. The data processing device 700 may be deployed in a front-end device for providing cloud desktop or cloud application services to users.

[0136] The sending module 710 sends the first input event to the server; the receiving module 720 receives the output sequence sent by the server, the output sequence includes output data corresponding to at least part of the input events in the input sequence, the input sequence is obtained by the server predicting the input events of the client after the first input event, and the input sequence includes at least one input event.

[0137] The data processing apparatus 700 may further include an output module. In response to receiving a second input event after the first input event, the output module may output output data in an output sequence in sequence.

[0138] The data processing device 700 may further include a maintenance module. The sending module 710 may send the second input event to the server. If the receiving module 720 receives a new output sequence sent by the server, the maintenance module may delete the existing output sequence and retain the new output sequence.

[0139] Figure 8 A schematic structural diagram of a computing device that can be used to implement the above-mentioned data processing method according to an embodiment of the present invention is shown.

[0140] See also Figure 8 , the computing device 800 includes a memory 810 and a processor 820 .

[0141] The processor 820 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 820 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 820 may be implemented using customized circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0142] The memory 810 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 820 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 810 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 810 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0143] The memory 810 stores executable codes. When the executable codes are processed by the processor 820 , the processor 820 can execute the data processing method described above.

[0144] The data processing method, apparatus, and device according to the present invention have been described above in detail with reference to the accompanying drawings.

[0145] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present invention.

[0146] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present invention.

[0147] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0148] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems and methods according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A data processing method for a cloud desktop or cloud application, suitable for execution by a server providing a cloud desktop service or a cloud application service to a client, the method comprising: In response to receiving a first input event from a client, predicting an input event of the client subsequent to the first input event to obtain an input sequence, where the input sequence includes at least one input event; Acquire an output sequence corresponding to the input sequence, wherein the output sequence includes output data corresponding to at least part of the input events in the input sequence; as well as Sending the output sequence to the client; The method further comprises: In response to receiving a second input event from the client after receiving the first input event, determining whether the second input event matches a current target input event in the input sequence; If the second input event does not match the current target input event in the input sequence, the number of hits and / or hit duration are updated, and if the number of hits and / or hit duration meet the preset conditions, the input event of the client after the second input event is predicted to obtain a new input sequence, and the output sequence of the new input sequence is obtained, and the new output sequence is sent to the client. The number of hits and / or hit duration are used to represent the number and / or duration of the unmatched input events.

2. A data processing method, suitable for execution by a server, comprising: In response to receiving a first input event from a client, predicting an input event of the client subsequent to the first input event to obtain an input sequence, where the input sequence includes at least one input event; Acquire an output sequence corresponding to the input sequence, wherein the output sequence includes output data corresponding to at least part of the input events in the input sequence; as well as Sending the output sequence to the client; The method further comprises: In response to receiving a second input event from the client after receiving the first input event, determining whether the second input event matches a current target input event in the input sequence; If the second input event does not match the current target input event in the input sequence, the number of hits and / or hit duration are updated, and if the number of hits and / or hit duration meet the preset conditions, the input event of the client after the second input event is predicted to obtain a new input sequence, and the output sequence of the new input sequence is obtained, and the new output sequence is sent to the client. The number of hits and / or hit duration are used to represent the number and / or duration of the unmatched input events.

3. The method according to claim 2, further comprising: If the second input event matches the current target input event in the input sequence, the first input event in the input sequence that follows the current target input event is used as a new current target input event.

4. The method according to claim 3, wherein: Before determining whether the second input event matches the first input sequence prediction result, the method further includes: determining whether the second input event satisfies an input tracking condition, where the input tracking condition is used to represent a condition that the current input event needs to satisfy when predicting an input event subsequent to the current input event; If the second input event satisfies the input tracking condition, determining whether the second input event complies with the prediction rule; If the second input event meets the prediction rule, a step of determining whether the second input event matches a current target input event in the input sequence is performed.

5. The method according to claim 4, further comprising: If the second input event does not satisfy the input tracking condition, obtaining output data corresponding to the second input event and sending the output data to the client; and / or If the second input event does not comply with the prediction rule, the no-hit time and / or the no-hit count of the input sequence are updated.

6. The method according to claim 4, further comprising: If the second input event does not comply with the prediction rule, determining whether the input sequence complies with a release rule, the release rule being used to represent a rule that needs to be satisfied to invalidate the input sequence; If the input sequence meets the release rule, the input sequence and / or the output sequence corresponding to the input sequence is deleted.

7. The method according to claim 4, further comprising: The hit count is used to represent the number of input events that meet the prediction rule but do not match the input sequence, and the hit duration is used to represent the duration of the input event that meets the prediction rule but does not match the input sequence; Correspondingly, the preset condition is that the number of hits is greater than or equal to a first threshold, or the hit duration is greater than or equal to a second threshold.

8. The method according to any one of claims 2 to 7, wherein: The first input event is an operation event for a cloud application or a cloud desktop. The input sequence is a set of operation events obtained by predicting the operation events performed by the client on the cloud application or cloud desktop after the first input event. The output sequence includes an operation result screen corresponding to each operation event in a set of predicted operation events.

9. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 8.

10. A computer program product comprising executable codes, which, when executed by a processor of an electronic device, causes the processor to perform the method according to any one of claims 1 to 8.

11. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method according to any one of claims 1 to 8.

Images