Cloud application interaction method, device, storage medium and system

Abstract

The application provides a cloud application interaction method, device, storage medium and system. The method comprises the following steps: a mobile terminal receives video stream data of a cloud application sent by a cloud server, decodes the video stream data to display a decoded video picture; based on a plurality of touch interaction operation behaviors corresponding to the cloud application registered in the mobile terminal, a target touch interaction operation behavior triggered by a user on the video picture is identified, and the target touch interaction operation behavior is processed in response. The interaction components corresponding to the plurality of touch interaction behaviors are adapted to the system architecture of the mobile terminal, and are not adapted to the system architecture of the cloud server. Through the scheme, the user can operate the cloud application in the cloud server according to the interaction operation habit of the mobile terminal.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a cloud application interaction method, device, storage medium, and system. Background Technology

[0002] Cloud applications are the application layer manifestation of cloud computing technology. Simply put, cloud applications transform the traditional "local installation" of software into an "on-demand" service. They deploy corresponding cloud services based on server clusters provided by cloud vendors, allowing users to remotely access and operate cloud servers through terminal devices to complete data access and computation. Common cloud applications include, but are not limited to, cloud desktops, cloud gaming, and cloud phones.

[0003] Currently, cloud applications are deployed on x86 architecture servers. However, with the development of mobile internet technology, more and more users want to access and operate remote cloud applications through mobile devices (such as smartphones, laptops, and digital assistants). To support user interaction on mobile devices, cloud applications currently need to provide interactive controls adapted to common x86 architecture interaction methods for user interaction on mobile devices. This results in a high barrier to entry for users to understand and use the applications, leading to poor ease of operation. Summary of the Invention

[0004] This invention provides a cloud application interaction method, device, storage medium, and system for use in...

[0005] In a first aspect, embodiments of the present invention provide a cloud application interaction method applied to a mobile terminal, the method comprising:

[0006] The system receives video stream data from a cloud application sent by a cloud server. The video stream data is obtained by encoding the interface of the cloud application. The system architecture of the cloud server is different from that of the mobile terminal.

[0007] Decode the video stream data to display the decoded video image;

[0008] Based on the various touch interaction behaviors registered in the mobile terminal and corresponding to cloud applications, the target touch interaction behavior triggered by the user on the video screen is identified; wherein, the interaction components corresponding to the various touch interaction behaviors are adapted to the system architecture of the mobile terminal but not to the system architecture of the cloud server; the target touch interaction behavior is included in the various touch interaction behaviors.

[0009] The target touch interaction behavior is responded to and processed.

[0010] Secondly, embodiments of the present invention provide a cloud application interaction device applied to a mobile terminal, the device comprising:

[0011] The receiving module is used to receive video stream data of a cloud application sent by a cloud server. The video stream data is obtained by encoding the interface of the cloud application. The system architecture of the cloud server is different from that of the mobile terminal.

[0012] The display module is used to decode the video stream data to display the decoded video image;

[0013] The interaction module is used to identify the target touch interaction behavior triggered by the user on the video screen based on multiple touch interaction behaviors registered in the mobile terminal and corresponding to cloud applications, and to respond to the target touch interaction behavior; wherein, the interaction components corresponding to the multiple touch interaction behaviors are adapted to the system architecture of the mobile terminal but not to the system architecture of the cloud server; the target touch interaction behavior is included in the multiple touch interaction behaviors.

[0014] Thirdly, embodiments of the present invention provide a mobile terminal, including: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the cloud application interaction method as described in the first aspect is executed.

[0015] Fourthly, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor in a mobile terminal, can at least implement the cloud application interaction method described in the first aspect.

[0016] Fifthly, embodiments of the present invention provide a cloud application system, including:

[0017] A cloud server running cloud applications, and a mobile terminal accessing the cloud applications, wherein the system architecture of the cloud server is different from that of the mobile terminal.

[0018] The cloud server is used to encode the interface of the cloud application into video stream data for transmission to the mobile terminal.

[0019] The mobile terminal is configured to receive the video stream data sent by the cloud server, decode the video stream data to display the decoded video image; and, based on multiple touch interaction behaviors registered in the mobile terminal corresponding to cloud applications, identify a target touch interaction behavior triggered by the user on the video image, and respond to the target touch interaction behavior; wherein the interaction components corresponding to the multiple touch interaction behaviors are adapted to the system architecture of the mobile terminal but not to the system architecture of the cloud server, and the target touch interaction behavior is included in the multiple touch interaction behaviors.

[0020] The solution provided in this invention enables users to perform interactive operations such as video browsing and information input by following the user's operating habits when operating cloud applications on a cloud server via a mobile terminal. This lowers the barrier to entry for users operating remote cloud applications on mobile terminals and improves the user experience. First, the system architecture used by the server running the cloud application (e.g., x86 architecture) differs from the system architecture used by the mobile terminal (e.g., ARM (Advanced RISC Machines) architecture). This difference in system architecture often means different operating systems and instruction sets (e.g., cloud servers use complex instruction sets, while mobile terminals use simplified instruction sets). Furthermore, it also means different user interaction habits. Therefore, for cloud applications, multiple touch interaction behaviors corresponding to the cloud application are pre-registered on the mobile terminal. The interactive components corresponding to these touch interaction behaviors are adapted to the mobile terminal's system architecture but not to the cloud server's system architecture. Simply put, these multiple interactive behaviors for operating cloud applications are set on the mobile terminal according to the user's interaction habits on the mobile terminal. Thus, when a mobile terminal receives video stream data from a cloud application sent by a cloud server, decodes it, and displays the decoded video on the screen, the user can trigger a target touch interaction action among various touch interaction actions on the displayed video. The mobile terminal recognizes the target touch interaction action and responds according to the corresponding response logic. This allows mobile users to operate cloud applications in accordance with their existing interaction habits on the mobile terminal. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of an interactive operation interface of a cloud application on a mobile terminal side, provided as an embodiment of the present invention.

[0023] Figure 2 A schematic diagram of a cloud application system provided in an embodiment of the present invention;

[0024] Figure 3 A flowchart illustrating a cloud application interaction method provided in an embodiment of the present invention;

[0025] Figure 4 A flowchart illustrating a cloud application interaction method provided in an embodiment of the present invention;

[0026] Figure 5 A flowchart illustrating a cloud application interaction method provided in an embodiment of the present invention;

[0027] Figure 6 A flowchart illustrating a cloud application interaction method provided in an embodiment of the present invention;

[0028] Figure 7 A flowchart illustrating a cloud application interaction method provided in an embodiment of the present invention;

[0029] Figure 8 This is a schematic diagram of the structure of a cloud application interaction device provided in an embodiment of the present invention;

[0030] Figure 9 This is a schematic diagram of the structure of an electronic device provided in this embodiment. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the timing of the steps in the following method embodiments is only an example and not a strict limitation.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0033] In increasingly diverse scenarios, users' demand for operating cloud applications via mobile devices is growing stronger. Common interactive operations for cloud applications mainly include browsing screens, scrolling screens, inputting content, and deleting content. However, in practice, cloud applications are generally deployed on cloud servers using a primary system architecture, while mobile devices often use a different secondary system architecture. The user's interactive operating system differs significantly under these different system architectures. For example, typically, the primary system architecture is x86, while the secondary system architecture is ARM.

[0034] For example, PCs and servers often use the x86 system architecture, while smartphones, digital assistants, and portable laptops (pads) often use the ARM system architecture. Generally speaking, different system architectures correspond to different operating systems and instruction sets. For instance, the x86 system architecture often uses Complex Instruction Set Computing (CISC), while the ARM system architecture often uses Reduced Instruction Set Computing (RISC).

[0035] There are significant differences between user operating habits under the x86 system architecture and those on mobile terminals. If the interaction methods under the x86 system architecture are used to operate cloud applications on mobile terminals, the user experience will be poor.

[0036] The following examples illustrate the differences in user operating habits when mobile terminals and cloud servers adopt the aforementioned ARM system architecture and x86 system architecture, respectively:

[0037] 1. The touch operation logic on the mobile terminal side is different from the mouse operation logic on the cloud server side;

[0038] 2. The input habits of the input boxes on the mobile terminal side are different from those on the cloud server side;

[0039] 3. The copy-paste logic on the mobile terminal side is different from that on the cloud server side.

[0040] Even though solutions exist for operating cloud applications via touch on mobile devices, these solutions merely activate the interactive components supported by the x86 system architecture. The operational logic still follows the x86 system architecture, meaning users on mobile devices still need to adhere to the operating habits of the x86 system architecture to operate cloud applications, which presents a certain understanding cost and usage threshold. The following section will combine... Figure 1 Provide an example.

[0041] like Figure 1 As shown, to meet the information input requirements of cloud application interfaces, when an input box is triggered on the mobile terminal screen, an input method interface adapted to the x86 system architecture can be displayed. This input method interface corresponds to the common input method used in x86 architecture PCs. Under the x86 system architecture, mouse "right-click" operations are also common. To meet this operation requirement, such as... Figure 1 As shown, a "right-click" button can be set on the screen of the mobile terminal. When the user touches and clicks this button, the function window corresponding to the right-click operation will be brought up. For example, in the x86 system architecture, to meet the needs of zooming in and out, a... Figure 1 The image shows a zoom slider, which is also available on the mobile device screen. Users can drag the slider using touch to zoom in and out.

[0042] As can be seen from the examples above, although users can operate the mobile terminal by touch, the interactive components and operation logic corresponding to the touch operation are still adapted to the x86 system architecture, which is different from the native operation logic of the mobile terminal.

[0043] Based on this, in scenarios where cloud applications on remote cloud servers are operated via mobile terminals, embodiments of the present invention provide end-to-end cloud input capabilities, allowing users to use their mobile terminal operating habits to control the screen and input content for cloud applications.

[0044] First, it should be noted that, as mentioned above, in this embodiment of the invention, the system architecture of the mobile terminal is different from that of the cloud server. This difference in system architecture will lead to different user interaction habits. For example, to achieve the same interaction purpose, the two use different interaction components and interaction logic.

[0045] Figure 2 This is a schematic diagram of a cloud application system provided in an embodiment of the present invention, such as... Figure 2 As shown in the diagram, assuming a cloud application is deployed on a cloud server in the cloud, the mobile terminal can access the remote cloud application through a client or a webpage.

[0046] In practical applications, this cloud server can be a virtual machine or a container. Cloud applications can be cloud desktops, cloud gaming, etc. Mobile terminals include smartphones.

[0047] Based on user access to cloud applications triggered by mobile devices, the cloud server encodes the application's interface into video stream data and transmits the video stream data to the mobile device using a pre-defined data transmission protocol. The mobile device decodes the received video stream data and displays the decoded video image on its screen. Users can then view the content contained in the cloud application's interface through their mobile devices.

[0048] like Figure 2 As shown, the cloud application interface actually includes interactive elements such as input boxes. Based on the video screen corresponding to the cloud application interface displayed on the mobile terminal, users can perform touch-based interactive operations on the video screen (touch interaction behavior), that is, trigger interactive operations through gestures. Subsequently, the user-triggered touch interaction behavior can be processed by the interaction processing modules set up in the mobile terminal and the cloud server respectively. It is understandable that when data transmission between the mobile terminal and the cloud server is required during processing, it must conform to the data transmission protocol used by both.

[0049] Figure 3 This is a flowchart illustrating a cloud application interaction method provided in an embodiment of the present invention. This method can be executed by a mobile terminal, such as... Figure 3 As shown, the method may include the following steps:

[0050] 301. Receive video stream data of cloud applications sent by the cloud server. The video stream data is obtained by encoding the interface of the cloud application.

[0051] 302. Decode the video stream data to display the decoded video image.

[0052] For ease of description, in this embodiment of the invention, it is assumed that the video frame is any frame from a multi-frame video frame obtained from decoded video stream data.

[0053] 303. Based on the various touch interaction behaviors registered in the mobile terminal and corresponding to cloud applications, identify the target touch interaction behaviors triggered by the user on the video screen; wherein, the interaction components corresponding to the various touch interaction behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server, and the system architecture of the cloud server is different from that of the mobile terminal.

[0054] 304. Respond to the target touch interaction behavior, which is included in a variety of touch interaction behaviors.

[0055] In this embodiment of the invention, in order to enable users to perform interactive operations such as video browsing and information input by following the user's operating habits when operating cloud applications on the cloud server through the mobile terminal, thereby reducing the threshold for users to operate remote cloud applications on the mobile terminal and improving the user experience, a variety of touch interactive operation behaviors corresponding to the cloud applications are pre-registered in the mobile terminal. The interactive components corresponding to these various touch interactive operation behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server. Simply put, these various interactive operation behaviors for operating cloud applications are set in the mobile terminal according to the user's interactive operation habits on the mobile terminal.

[0056] In summary, assuming a certain interactive function (or interactive intent, interactive event) is to be implemented, to adapt to the first system architecture used in the cloud server, an interactive component A corresponding to this interactive function, along with the operation method and response logic for interactive component A, can be registered on the mobile terminal side. To adapt to the second system architecture used by the mobile terminal, an interactive component B corresponding to this interactive function, along with the operation method and response logic for interactive component B, can be registered on the mobile terminal side. Interactive components A and B, and their respective operation methods, are determined based on the user's operating habits on the devices corresponding to the first and second system architectures, respectively, when implementing this interactive function.

[0057] For example, assuming it's adapted to the first system architecture, the interactive function for scaling the video frame needs to be implemented through... Figure 1 The zoom function is implemented using the slider shown in the image, but it is adapted to the second system architecture. By analyzing the common operating habits of users zooming in and out on the corresponding mobile terminal, it was determined that multi-finger pinch and stretch gestures are used to implement the screen zoom function.

[0058] As can be seen from this example, in this embodiment of the invention, it is not limited to registering the corresponding interactive component B for every interactive function. It is possible that for some interactive functions, only the corresponding touch interactive operation behavior is registered.

[0059] Therefore, in summary, multiple touch interaction behaviors are pre-registered on the mobile terminal side for cloud applications. The registration information for each touch interaction behavior includes, but is not limited to, interactive components, touch interaction operations, and corresponding response logic. These multiple touch interaction behaviors are adapted to the system architecture of the mobile terminal but not to the system architecture of the cloud server.

[0060] Therefore, when a user triggers a certain touch interaction behavior (referred to as the target touch interaction behavior) on the video screen currently displayed on the mobile terminal, the mobile terminal recognizes the target touch interaction behavior and responds to it according to the corresponding response logic.

[0061] In practice, during the traditional interaction process adapted to the first system architecture, the mobile terminal simply transmits the user's interactive actions triggered on the mobile terminal to the cloud server, and the response logic for these interactive actions is implemented by the cloud server. However, in this embodiment of the invention, the response logic for different touch interactive actions may not all be implemented on the cloud server side.

[0062] Optionally, if the target touch interaction behavior is a first touch interaction behavior that adjusts the visual effect of the video screen, the mobile terminal responds to the target touch interaction behavior locally; if the target touch interaction behavior is a second touch interaction behavior that adjusts the content of the video screen, the mobile terminal and the cloud server respond to the target touch interaction behavior in collaboration.

[0063] Optionally, the first touch interaction behavior includes any one of the following: screen zooming behavior, screen panning behavior.

[0064] Optionally, the second touch interaction behavior includes any one of the following: information input behavior and function window activation behavior. The information input behavior includes, for example, activating an input method to enter information in an input box, or pasting data from the clipboard into an input box. The function window activation behavior corresponds to the operation of activating a function window using the right mouse button in a traditional x86 system architecture.

[0065] In fact, touch interaction behaviors that do not affect the content displayed in the video can be processed locally on the mobile terminal. However, touch interaction behaviors that update the content displayed in the video can be processed collaboratively by the mobile terminal and the cloud server, which can reduce response latency.

[0066] For ease of understanding, in Figure 4 The diagram illustrates several common interaction event types, along with the corresponding triggering preconditions, triggering actions (i.e., touch interaction behaviors), and triggering results for each event. The relationship between the triggering preconditions and triggering actions will be explained in subsequent specific embodiments. It should be noted that... Figure 4 The triggering actions mentioned are merely examples and are not limited to these.

[0067] The following examples illustrate the interaction processing of some of the touch interaction behaviors mentioned above.

[0068] Figure 5 This is a flowchart illustrating a cloud application interaction method provided in an embodiment of the present invention. This method can be executed by a mobile terminal, such as... Figure 5 As shown, the method may include the following steps:

[0069] 501. Receive video stream data from the cloud server for the cloud application. The video stream data is obtained by encoding the interface of the cloud application.

[0070] 502. Decode the video stream data to display the decoded video image.

[0071] 503. Based on the various touch interaction behaviors registered in the mobile terminal corresponding to cloud applications, identify the target touch interaction behavior triggered by the user on the video screen, wherein the target touch interaction behavior is screen zooming operation behavior and / or screen panning operation behavior.

[0072] 504. Based on the screen scaling operation and / or screen panning operation, perform corresponding scaling and / or panning processing on the locally displayed video screen, wherein the video screen in the cloud server remains unchanged.

[0073] This embodiment corresponds to Figure 4 The video playback involves two interactive events: zooming and panning. These events aim to allow users to view all or part of the video displayed on the mobile device screen more clearly and in focus. They do not update the content of the video frame; therefore, the zooming and panning operations are handled locally on the mobile device.

[0074] based on Figure 4 From the triggering preconditions and triggering actions corresponding to the two interactive events exemplified in the example, it can be seen that when the mobile terminal recognizes that the user has triggered a three-finger simultaneous touch operation on the screen displaying the video image, it means that the user may be about to trigger a screen zoom operation or a screen pan operation. Further recognition is performed, and when the user's three fingers are further recognized as pinching (i.e.,...) Figure 4 The sum of the distances between the three fingers and the center point of the three fingers tends to decrease, or the fingers are stretched out (i.e., Figure 4 When the gesture feature (the sum of the distances between the three fingers and their center points tends to increase) is detected, it is determined that the user has triggered a screen zooming operation, and the local system can perform corresponding zooming processing on the video displayed on the screen; when the gesture feature (i.e., the user's three fingers move in a certain direction) is further detected, it is determined that the user has triggered a screen zooming operation, and the local system can perform corresponding zooming processing on the video displayed on the screen. Figure 4 When a gesture indicates a change in the center point of three fingers on the screen (a change of more than 10 pixels in distance), it is determined that the user has triggered a screen panning operation. The local system then moves the video displayed on the screen according to the direction of the finger movement. Here, 10 pixels represents a distance of 10 pixels.

[0075] It should be noted that because the mobile terminal responds locally to the aforementioned screen zooming and panning operations, the video image only exhibits these effects on the mobile terminal's side; the cloud application interface on the cloud server does not display these changes. In other words, the mobile terminal does not upload these screen zooming and panning operations to the cloud server.

[0076] Figure 6 This is a flowchart illustrating a cloud application interaction method provided in an embodiment of the present invention. This method can be executed by a mobile terminal, such as... Figure 6 As shown, the method may include the following steps:

[0077] 601. Receive video stream data of cloud applications sent by the cloud server. The video stream data is obtained by encoding the interface of the cloud application.

[0078] 602. Decode the video stream data to display the decoded video image.

[0079] 603. Based on the various touch interaction behaviors registered in the mobile terminal corresponding to cloud applications, identify the target touch interaction behavior triggered by the user on the video screen. The target touch interaction behavior is an information input behavior, which includes the set touch operation behavior triggered in the input box area of ​​the video screen.

[0080] 604. Set the input box to information input mode, call the local input method of the mobile terminal to display the information input interface, which includes a data paste button.

[0081] 605. In response to the user's sending operation of the information content entered through the information input interface, the information content and the indication information of the input box are sent to the cloud server so that the cloud server displays the information content in the input box of the corresponding cloud application interface.

[0082] 606. In response to the user's selection of the data paste button, the target copied data in the local clipboard and the instruction information in the input box are sent to the cloud server, so that the cloud server displays the target copied data in the input box in the corresponding cloud application interface.

[0083] In this embodiment, it is assumed that the video screen currently displayed on the mobile terminal corresponds to a cloud application interface on the cloud server.

[0084] This embodiment corresponds to Figure 4 The two interactive events in the text are activating the local input method and pasting.

[0085] Assuming the video displayed on the mobile terminal screen includes an input box, such as Figure 4As described, when a user is detected to have double-tapped an input box area (e.g., within a 5px radius) within a 1-second timeframe, it is determined that the user has triggered an information input action. The input box is then set to an information input state (e.g., by blinking the cursor within the input box, or adding a colored border to indicate selection), and the mobile terminal's local input method is invoked to display the information input interface. The common local input method used on mobile terminals is the 9-key grid input method, which differs from the input method used in the x86 architecture. The information input interface is the input interface corresponding to this input method.

[0086] In addition, since users can enter information into the input box by typing it directly using the input method or by pasting copied data from the clipboard, the above information input interface also includes a data paste button.

[0087] Optionally, when the user triggers the above click operation, if it is further determined that there is copied data in the clipboard of the mobile terminal, the above data paste button can be displayed; otherwise, it can be hidden or made unselectable.

[0088] When there is more than one copied data in the clipboard, the last copied data is determined as the target copied data.

[0089] When a user finishes entering information into the input box using the input method and triggers the send operation, or when a user selects the data paste button to paste the target copied data into the input box and triggers the send operation, the mobile terminal sends the information entered by the user or the target copied data, along with the indication information of the input box, to the cloud server. This allows the cloud server to locate the input box in the corresponding cloud application interface based on the indication information of the input box and display the information or target copied data in the input box.

[0090] It is understandable that when a mobile terminal sends the above information to a cloud server, it must also follow the data transmission protocol between the mobile terminal and the cloud server.

[0091] Figure 7 This is a flowchart illustrating a cloud application interaction method provided in an embodiment of the present invention. This method can be executed by a mobile terminal, such as... Figure 7 As shown, the method may include the following steps:

[0092] 701. Receive video stream data from the cloud server for cloud applications. The video stream data is obtained by encoding the interface of the cloud application.

[0093] 702. Decode the video stream data to display the decoded video image.

[0094] 703. Based on the various touch interaction behaviors registered in the mobile terminal corresponding to cloud applications, identify the target touch interaction behavior triggered by the user on the video screen. The target touch interaction behavior is a function window wake-up behavior, which refers to the set touch operation behavior triggered in the video screen to wake up the target function window.

[0095] 704. Send a target notification message to the cloud server. The target notification message includes the location where the user triggered the function window activation operation in the video frame. The target notification message is used to indicate that the user triggered the function window activation operation at the location, so that the cloud server displays the target function window at the location in the corresponding cloud application interface, and transmits the video frame containing the target function window to the mobile terminal for display.

[0096] This embodiment corresponds to Figure 4 The right-click interaction event in the browser.

[0097] like Figure 4 As shown, when a user's touch operation of simultaneously tapping the screen with two fingers on a mobile terminal displaying the aforementioned video is detected, it is determined that the user has triggered a function window activation operation. At this time, the mobile terminal can notify the cloud server of the corresponding trigger location in the video frame, informing the cloud server where the user triggered the function window activation operation within the video frame. The cloud server can then invoke the target function window corresponding to that video frame and display it at that location, thereby forming a new interface. This new interface is then encoded to obtain a new video frame, which is transmitted to the mobile terminal for decoding and display.

[0098] above Figure 6 , Figure 7 The illustrated embodiments demonstrate the process by which a mobile terminal and a cloud server collaboratively respond to touch-based interactive operations that update video content. These examples show that, in these embodiments of the invention, by completing the recognition and partial processing of user touch-based interactive operations on the mobile terminal side as much as possible, the processing logic of the cloud server is simplified, resulting in better response timeliness.

[0099] The cloud application interaction device according to one or more embodiments of the present invention will be described in detail below. Those skilled in the art will understand that these devices can all be configured using commercially available hardware components through the steps taught in this solution.

[0100] Figure 8 This is a schematic diagram of the structure of a cloud application interaction device provided in an embodiment of the present invention. This device is applied in the aforementioned mobile terminal, such as... Figure 8As shown, the device includes: a receiving module 11, a display module 12, and an interaction module 13.

[0101] The receiving module 11 is used to receive video stream data of a cloud application sent by a cloud server. The video stream data is obtained by encoding the interface of the cloud application. The system architecture of the cloud server is different from that of the mobile terminal.

[0102] Display module 12 is used to decode the video stream data to display the decoded video image.

[0103] The interaction module 13 is used to identify the target touch interaction behavior triggered by the user on the video screen based on the various touch interaction behaviors registered in the mobile terminal corresponding to the cloud application, and to respond to the target touch interaction behavior; wherein, the interaction components corresponding to the various touch interaction behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server; the target touch interaction behavior is included in the various touch interaction behaviors.

[0104] Optionally, the interaction module 13 is specifically used to: if the target touch interaction operation behavior belongs to a first touch interaction operation behavior that adjusts the visual effect of the video screen, then respond to the target touch interaction operation behavior locally; if the target touch interaction operation behavior belongs to a second touch interaction operation behavior that adjusts the screen content of the video screen, then coordinate with the cloud server to respond to the target touch interaction operation behavior.

[0105] Optionally, the first touch interaction behavior includes any one of the following: screen zooming behavior, screen panning behavior. Optionally, the second touch interaction behavior includes any one of the following: information input behavior, function window activating behavior.

[0106] Optionally, the target touch interaction operation behavior is a screen zoom operation behavior and / or a screen pan operation behavior, and the interaction module 13 is specifically used to: perform corresponding zoom and / or pan processing on the locally displayed video screen according to the screen zoom operation behavior and / or screen pan operation behavior, wherein the video screen in the cloud server remains unchanged.

[0107] Optionally, the target touch interaction behavior is an information input behavior, which includes a set touch operation behavior triggered by the input box area in the video frame. The interaction module 13 is specifically used to: set the input box to an information input state; invoke the local input method of the mobile terminal to display an information input interface; and, in response to the user's sending operation of the information content entered through the information input interface, send the information content and the indication information of the input box to the cloud server, so that the cloud server displays the information content in the input box of the corresponding cloud application interface.

[0108] Optionally, the information input interface includes a data paste button, and the interaction module 13 is further configured to: in response to the user's selection operation via the data paste button, send the target copied data in the local clipboard and the indication information of the input box to the cloud server, so that the cloud server displays the target copied data in the input box of the corresponding cloud application interface.

[0109] Optionally, the target touch interaction behavior is a function window activation behavior, which refers to a set touch operation behavior triggered in the video frame to activate a target function window. The interaction module 13 is specifically used to: send a target notification message to the cloud server, the target notification message including the location in the video frame where the user triggered the function window activation behavior, the target notification message indicating that the user triggered the function window activation behavior at that location, so that the cloud server displays the target function window at that location in the corresponding cloud application interface, and transmits the video frame containing the target function window to the mobile terminal for display.

[0110] Figure 8 The device shown can perform the steps provided in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0111] This invention also provides a mobile terminal, such as... Figure 9 As shown, it may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code, which, when executed by the processor 21, implements the cloud application interaction method as described in the preceding embodiments.

[0112] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium on which executable code is stored. When the executable code is executed by a processor in a mobile terminal, the cloud application interaction method provided in the foregoing embodiments can be implemented.

[0113] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cloud application interaction method, characterized in that, Applied to a mobile terminal, the method includes: Access to the cloud application triggered by the mobile terminal allows the user to receive video stream data from the cloud application sent by the cloud server. The cloud application is deployed on the cloud server, and the video stream data is obtained by encoding the interface of the cloud application. The system architecture of the cloud server is different from that of the mobile terminal. Decode the video stream data to display the decoded video image; Based on the various touch interaction behaviors registered in the mobile terminal corresponding to cloud applications, the system identifies the target touch interaction behavior triggered by the user on the video screen. The interaction components corresponding to these various touch interaction behaviors are adapted to the system architecture of the mobile terminal but not to the system architecture of the cloud server. These various touch interaction behaviors are interactive behaviors set according to the user's interaction habits with the mobile terminal for operating cloud applications. The target touch interaction behavior is included within these various touch interaction behaviors. If the target touch interaction operation is a first touch interaction operation that adjusts the visual effect of the video screen, the mobile terminal responds to the target touch interaction operation locally, and the target touch interaction operation is not uploaded to the cloud server. If the target touch interaction operation is a second touch interaction operation that adjusts the content of the video screen, then the cloud server will respond in coordination with the target touch interaction operation.

2. The method according to claim 1, characterized in that, The first touch interaction operation behavior includes any one of the following: screen zoom operation behavior, screen pan operation behavior.

3. The method according to claim 2, characterized in that, The target touch interaction operation behavior is screen zoom operation behavior and / or screen pan operation behavior; The mobile terminal locally responds to the target touch interaction behavior, including: Based on the screen scaling operation and / or screen panning operation, the video screen displayed locally on the mobile terminal is subjected to corresponding scaling and / or panning processing, wherein the video screen in the cloud server remains unchanged.

4. The method according to claim 1, characterized in that, The second touch interaction behavior includes any one of the following: information input behavior, function window activation behavior.

5. The method according to claim 4, characterized in that, The target touch interaction behavior is an information input behavior, which includes the set touch behavior triggered in the input box area of ​​the video screen; The coordinated response to the target touch interaction behavior with the cloud server includes: Set the input box to information input mode; The local input method of the mobile terminal is invoked to display the information input interface; In response to the user's sending operation of the information content entered through the information input interface, the information content and the indication information of the input box are sent to the cloud server, so that the cloud server displays the information content in the input box in the corresponding cloud application interface.

6. The method according to claim 5, characterized in that, The information input interface includes a data paste button; The coordinated response to the target touch interaction behavior with the cloud server includes: In response to the user's selection of the data paste button, the target copied data in the local clipboard of the mobile terminal and the indication information of the input box are sent to the cloud server, so that the cloud server displays the target copied data in the input box in the corresponding cloud application interface.

7. The method according to claim 4, characterized in that, The target touch interaction behavior is a function window wake-up behavior, which refers to the set touch operation behavior triggered in the video screen to wake up the target function window. The coordinated response to the target touch interaction behavior with the cloud server includes: A target notification message is sent to the cloud server. The target notification message includes the location in the video frame where the user triggered the function window activation action. The target notification message is used to indicate that the user triggered the function window activation action at the location, so that the cloud server displays the target function window at the location in the corresponding cloud application interface, and transmits the video frame containing the target function window to the mobile terminal for display.

8. A mobile terminal, characterized in that, include: The system includes a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, it performs the cloud application interaction method as described in any one of claims 1 to 7.

9. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code, which, when executed by a processor in a mobile terminal, performs the cloud application interaction method as described in any one of claims 1 to 7.

10. A cloud application system, characterized in that, include: A cloud server running cloud applications, and a mobile terminal accessing the cloud applications, wherein the system architecture of the cloud server is different from that of the mobile terminal. The cloud server is used to encode the interface of the cloud application into video stream data for transmission to the mobile terminal. The mobile terminal is used to receive the video stream data sent by the cloud server through access to the cloud application triggered by the mobile terminal, and to decode the video stream data to display the decoded video image. Furthermore, based on the various touch interaction behaviors registered in the mobile terminal corresponding to cloud applications, the target touch interaction behavior triggered by the user on the video screen is identified. If the target touch interaction behavior belongs to the first touch interaction behavior that adjusts the visual effect of the video screen, the mobile terminal responds locally to the target touch interaction behavior, and the target touch interaction behavior is not uploaded to the cloud server; if the target touch interaction behavior belongs to the second touch interaction behavior that adjusts the screen content of the video screen, the mobile terminal responds in coordination with the cloud server to the target touch interaction behavior; wherein, the interaction components corresponding to the various touch interaction behaviors are adapted to the system architecture of the mobile terminal but not to the system architecture of the cloud server, the various touch interaction behaviors are interaction behaviors set according to the user's interaction habits on the mobile terminal for operating cloud applications, and the target touch interaction behavior is included in the various touch interaction behaviors.

Images