Cloud application interaction method, device, storage medium and system

Abstract

The invention provides a cloud application interaction method, device, storage medium and system, and the method comprises the steps that a mobile terminal receives video stream data of a cloud application sent by a cloud server, and decodes the video stream data to display a decoded video image; and based on various touch interaction operation behaviors corresponding to the cloud application and registered in the mobile terminal, identifying a target touch interaction operation behavior triggered by the user on the video picture, and performing response processing on the target touch interaction operation behavior. Wherein the interaction components corresponding to the multiple touch interaction behaviors are adaptive to the system architecture of the mobile terminal and are not adaptive to the system architecture of the cloud server. According to the scheme, the user can operate the cloud application in the cloud server according to the interactive operation habit of the mobile terminal.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a cloud application interaction method, device, storage medium and system. Background Art

[0002] Cloud applications are the embodiment of cloud computing technology at the application layer. The working principle of cloud applications is simply to change the traditional software "local installation" usage mode into a "ready-to-use" service. The corresponding cloud services are deployed based on the server cluster provided by the cloud vendor. Users remotely access and operate the cloud server through terminal devices to complete the corresponding data access and calculation. Common cloud applications include but are not limited to cloud desktops, cloud games, cloud phones, etc.

[0003] Since cloud applications are currently deployed in x86 architecture servers, and with the development of mobile Internet technology, more and more users want to access and operate remote cloud applications through mobile terminals (such as smart phones, portable laptops, digital assistants, etc.). At present, in order to support user operations on mobile terminals, cloud applications need to provide interactive controls that are adapted to the common interaction methods of the x86 architecture for users to perform interactive operations on mobile terminals. Users have high understanding and usage thresholds, and the operation convenience is poor. Summary of the invention

[0004] Embodiments of the present invention provide a cloud application interaction method, device, storage medium and system for.

[0005] In a first aspect, an embodiment of the present invention provides a cloud application interaction method, which is applied to a mobile terminal, and the method includes:

[0006] Receiving video stream data of a cloud application sent by a cloud server, wherein the video stream data is obtained by encoding an interface of the cloud application, and a system architecture of the cloud server is different from a system architecture of the mobile terminal;

[0007] Decoding the video stream data to display the decoded video picture;

[0008] Based on the multiple touch interaction operation behaviors corresponding to the cloud application registered in the mobile terminal, identifying the target touch interaction operation behavior triggered by the user on the video screen; wherein the interaction components corresponding to the multiple touch interaction behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server; the target touch interaction operation behavior is included in the multiple touch interaction operation behaviors;

[0009] The target touch interaction operation behavior is responded to and processed.

[0010] In a second aspect, an embodiment of the present invention provides a cloud application interaction device, which is applied to a mobile terminal, and the device includes:

[0011] A receiving module, used for receiving video stream data of a cloud application sent by a cloud server, wherein the video stream data is obtained by encoding an interface of the cloud application, and the system architecture of the cloud server is different from that of the mobile terminal;

[0012] A display module, used for decoding the video stream data to display the decoded video picture;

[0013] An interaction module is used to identify a target touch interaction operation behavior triggered by a user on the video screen based on a plurality of touch interaction operation behaviors corresponding to the cloud application registered in the mobile terminal, and to respond to and process the target touch interaction operation behavior; wherein the interaction components corresponding to the plurality of touch interaction operations are adapted to the system architecture of the mobile terminal but not to the system architecture of the cloud server; and the target touch interaction operation behavior is included in the plurality of touch interaction operation behaviors.

[0014] In a third aspect, an embodiment of the present invention provides a mobile terminal, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the cloud application interaction method described in the first aspect is executed.

[0015] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor in a mobile terminal, at least the cloud application interaction method described in the first aspect can be implemented.

[0016] In a fifth aspect, an embodiment of the present invention provides a cloud application system, including:

[0017] A cloud server running a cloud application, and a mobile terminal accessing the cloud application, wherein the system architecture of the cloud server is different from the system architecture of the mobile terminal;

[0018] The cloud server is used to encode the interface of the cloud application into video stream data to send to the mobile terminal;

[0019] The mobile terminal is used to receive the video stream data sent by the cloud server, decode the video stream data to display the decoded video screen; and, based on the multiple touch interaction operation behaviors corresponding to the cloud application registered in the mobile terminal, identify the target touch interaction operation behavior triggered by the user on the video screen, and respond to the target touch interaction operation behavior; wherein the interaction components corresponding to the multiple touch interaction operation behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server, and the target touch interaction operation behavior is included in the multiple touch interaction operation behaviors.

[0020] The solution provided by the embodiment of the present invention is used to enable users to browse videos, input information and other interactive operations in accordance with the user operation habits of the mobile terminal when operating cloud applications in the cloud server through a mobile terminal, thereby reducing the threshold for users to operate remote cloud applications in the mobile terminal and improving the user operation experience. First, the system architecture (such as x86 architecture) used by the server running the cloud application is different from the system architecture (such as ARM (short for Advanced RISC Machines) architecture) used by the mobile terminal. The different system architectures often mean that the operating systems and instruction sets used by the two are different (for example, the cloud server uses a complex instruction set and the mobile terminal uses a reduced instruction set). In addition, it also means that the user interaction operation habits of the two are different. Based on this, for cloud applications, multiple touch interaction operation behaviors corresponding to the cloud application are pre-registered in the mobile terminal. The interaction components corresponding to these multiple touch interaction operation behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server. In short, these multiple interaction operation behaviors for operating cloud applications are set in the mobile terminal according to the user's interaction operation habits with the mobile terminal. In this way, when the mobile terminal receives the video stream data of a cloud application sent by the cloud server, decodes it, and displays the decoded video picture on the screen, the terminal user can trigger a target touch interaction operation behavior among multiple touch interaction operations on the screen for the displayed video picture. After the mobile terminal recognizes the target touch interaction operation behavior, it responds to the target touch interaction operation behavior according to the corresponding response logic. In this way, the user of the mobile terminal can operate the cloud application in the cloud according to the interactive operation habits of the user on the mobile terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram of an interactive operation interface of a cloud application on a mobile terminal side provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a cloud application system provided by an embodiment of the present invention;

[0024] Figure 3 A flow chart of a cloud application interaction method provided by an embodiment of the present invention;

[0025] Figure 4 A flow chart of a cloud application interaction method provided by an embodiment of the present invention;

[0026] Figure 5 A flow chart of a cloud application interaction method provided by an embodiment of the present invention;

[0027] Figure 6 A flow chart of a cloud application interaction method provided by an embodiment of the present invention;

[0028] Figure 7 A flow chart of a cloud application interaction method provided by an embodiment of the present invention;

[0029] Figure 8 A schematic diagram of the structure of a cloud application interaction device provided by an embodiment of the present invention;

[0030] Fig. 9 A schematic diagram of the structure of an electronic device provided in this embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the step timing in the following method embodiments is only an example, not a strict limitation.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] In the increasingly diverse scenarios, users have a growing demand to operate cloud applications through mobile terminals. Common interactive operations on cloud applications mainly include browsing screens, moving screens, inputting content, deleting content, etc. However, in actual applications, cloud applications are generally deployed in cloud servers that adopt the first system architecture, while mobile terminals often adopt a different second system architecture. The user's interactive operating system under different system architectures is quite different. For example, generally, the first system architecture is the x86 system architecture, and the second system architecture is the ARM system architecture.

[0034] For example, PCs and servers often use the x86 system architecture, while smartphones, digital assistants, and portable notebooks (pads) often use the ARM system architecture. Generally speaking, different system architectures often correspond to different operating systems and instruction sets. For example, the x86 system architecture often uses the Complex Instruction Set Computing (CISC), while the ARM system architecture often uses the Reduced Instruction Set Computing (RISC).

[0035] There are significant differences between user operation habits under the x86 system architecture and those in mobile terminals. If the interactive operation method under the x86 system architecture is used to operate cloud applications on mobile terminals, the user operation experience will be poor.

[0036] The following example illustrates the differences in user operating habits when the mobile terminal and cloud server use the ARM system architecture and x86 system architecture respectively:

[0037] 1. The touch operation on the mobile terminal side and the mouse operation logic on the cloud server side are different;

[0038] 2. The input box input habits on the mobile terminal side are different from those on the cloud server side;

[0039] 3. The copy and paste logic on the mobile terminal side is different from the copy and paste logic on the cloud server side.

[0040] Even though there are currently solutions for operating cloud applications through touch on mobile terminals, in these solutions, in fact, the touch gesture operation only evokes the interactive components supported by the x86 system architecture, and the operation logic still follows the operation logic of the x86 system architecture. Users still need to follow the operation habits of the x86 system architecture to operate cloud applications on the mobile terminal side, which will have a certain understanding cost and usage threshold. Figure 1 Provide an example.

[0041] like Figure 1 As shown in , in order to meet the information input requirements in the cloud application interface, when the input operation of the input box is triggered on the mobile terminal side screen, an input method interface adapted to the x86 system architecture can be displayed. The input method corresponding to this input method interface is the input method commonly used in PCs under the x86 architecture. Under the x86 system architecture, there is also a common mouse "right button" operation. To meet this operation requirement, Figure 1 As shown in , a "right key" button can be set in the screen on the mobile terminal side. When the user touches and clicks the button, the function window corresponding to the right key operation of the mouse can be called out. For another example, in the x86 system architecture, in order to meet the needs of zooming the screen, Figure 1 A zoom function slider is shown in the figure. The zoom function slider is still provided in the screen on the mobile terminal side, and the user can drag the slider in a touch-control manner to zoom the screen.

[0042] From the above examples, it can be seen that on the mobile terminal side, although the user can operate by touch, the interactive components and operation logic corresponding to the touch operation are still adapted to the x86 system architecture, which is different from the native operation logic on the mobile terminal side.

[0043] Based on this, in the scenario of operating a cloud application in a remote cloud server through a mobile terminal, an embodiment of the present invention provides a terminal-cloud integrated input capability, allowing users to use the operating habits of the mobile terminal to control the screen and input content of the cloud application.

[0044] First of all, it should be noted that in the embodiment of the present invention, as described above, the system architecture of the mobile terminal is different from the system architecture of the cloud server. The difference in the system architecture will lead to different user interaction operation habits between the two. For example, to achieve the same interaction purpose, the interaction components and interaction operation logic used by the two are different.

[0045] Figure 2 A schematic diagram of a cloud application system provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown in , assuming that a cloud application is deployed in a cloud server in the cloud, the remote cloud application can be accessed in a mobile terminal through a client or a web page.

[0046] In actual applications, the cloud server can be a virtual machine or a container. Cloud applications can be cloud desktops, cloud games, etc. Mobile terminals are, for example, smart phones.

[0047] Based on the access of the cloud application triggered by the user through the mobile terminal, the cloud server will encode the interface of the cloud application into video stream data and transmit the video stream data to the mobile terminal through the set data transmission protocol. The mobile terminal decodes the received video stream data to display the decoded video screen on the screen. The user can then see the content contained in the interface of the cloud application through the mobile terminal.

[0048] like Figure 2 As shown in , in fact, the cloud application interface will include interactive elements such as input boxes. Based on the video screen corresponding to the interface of the cloud application displayed on the mobile terminal, the user can perform touch-based interactive operations (touch interactive operations) on the video screen, that is, trigger interactive operations through some gestures. Afterwards, the touch interactive operations triggered by the user can be processed based on the interactive processing modules respectively provided in the mobile terminal and the cloud server. It is understandable that when data transmission between the mobile terminal and the cloud server is required during the processing, it is necessary to comply with the data transmission protocol adopted between the two.

[0049] Figure 3 A flowchart of a cloud application interaction method provided by an embodiment of the present invention, which can be executed by a mobile terminal, such as Figure 3 As shown, the method may include the following steps:

[0050] 301. Receive video stream data of a cloud application sent by a cloud server, where the video stream data is obtained by encoding an interface of the cloud application.

[0051] 302. Decode the video stream data to display the decoded video picture.

[0052] For ease of description, it is assumed in the embodiment of the present invention that the video picture is any frame of multiple frames of video pictures obtained from decoding video stream data.

[0053] 303. Based on the various touch interaction operation behaviors corresponding to the cloud application registered in the mobile terminal, identify the target touch interaction operation behavior triggered by the user on the video screen; wherein, the interaction components corresponding to the various touch interaction behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server, and the system architecture of the cloud server is different from the system architecture of the mobile terminal.

[0054] 304. Perform response processing on a target touch interaction operation behavior, where the target touch interaction operation behavior is included in a plurality of touch interaction operation behaviors.

[0055] In an embodiment of the present invention, in order to enable users to follow the user operation habits of the mobile terminal to perform interactive operations such as video browsing and information input when operating cloud applications in a cloud server through a mobile terminal, so as to lower the threshold for users to operate remote cloud applications in mobile terminals and improve the user operation experience, a variety of touch interactive operation behaviors corresponding to cloud applications will be pre-registered in the mobile terminal. The interactive components corresponding to these multiple touch interactive operation behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server. In simple terms, these multiple interactive operation behaviors for operating cloud applications are set in the mobile terminal according to the user's interactive operation habits of the mobile terminal.

[0056] In summary, assuming that in order to realize a certain interactive function (or interactive intention, interactive event), in order to adapt to the first system architecture adopted in the cloud server, the interactive component A corresponding to the interactive function and the operation mode and response logic of the interactive component A can be registered on the mobile terminal side. In order to adapt to the second system architecture adopted by the mobile terminal, the interactive component B corresponding to the interactive function and the operation mode and response logic of the interactive component B can be registered on the mobile terminal side. The interactive component A and the interactive component B and their corresponding operation modes are determined based on the user's operating habits in the devices corresponding to the first system architecture and the second system architecture respectively when realizing such an interactive function.

[0057] For example, assuming that the first system architecture is adapted, the interactive function of zooming in and out of the video screen needs to be implemented through Figure 1 The zoom function slider shown in the figure is used to implement the zoom function, but it is adapted to the second system architecture. By analyzing the common operating habits of users for zooming the screen on the corresponding mobile terminal side, it is determined that a multi-finger pinch and stretch gesture is set to implement the screen zoom function.

[0058] It can be seen from this example that in the embodiment of the present invention, it is not limited that each interactive function must register a corresponding interactive component B, and it is possible that only the corresponding touch interactive operation behavior is registered for some interactive functions.

[0059] Therefore, in summary, multiple touch interaction operation behaviors are pre-registered for cloud applications on the mobile terminal side, and the registration information of each touch interaction operation behavior includes but is not limited to interaction components, touch interaction operations, and corresponding response logic. These multiple touch interaction operation behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server.

[0060] Therefore, when the user triggers a certain touch interaction operation behavior (called target touch interaction operation behavior) on the video screen currently displayed on the mobile terminal screen, the mobile terminal recognizes the target touch interaction operation behavior and responds to the target touch interaction operation behavior according to the corresponding response logic.

[0061] In fact, in the traditional interaction process adapted to the first system architecture, the mobile terminal side only transparently transmits the interactive operation behavior triggered by the user on the mobile terminal side to the cloud server, and the response logic to the interactive operation behavior is implemented by the cloud server. However, in the embodiment of the present invention, the response logic to different touch interactive operation behaviors may not be implemented on the cloud server side.

[0062] Optionally, if the target touch interaction operation behavior belongs to the first touch interaction operation behavior of adjusting the visual effect of the video screen, the mobile terminal locally responds to the target touch interaction operation behavior; if the target touch interaction operation behavior belongs to the second touch interaction operation behavior of adjusting the picture content of the video screen, the mobile terminal and the cloud server collaboratively respond to the target touch interaction operation behavior.

[0063] Optionally, the first touch interaction operation behavior includes any one of the following: a screen zooming operation behavior, a screen panning operation behavior.

[0064] Optionally, the second touch interaction operation behavior includes any of the following: information input operation behavior, function window awakening operation behavior. Among them, the information input operation behavior is, for example, the behavior of awakening the input method to input information in the input box, and the behavior of pasting data in the clipboard in the input box. Among them, the function window awakening operation behavior is to respond to the operation of awakening the function window by the right-click operation of the mouse in the traditional x86 system architecture.

[0065] In fact, target touch interaction operations that will not affect the content displayed in the video screen can be responded to and processed locally on the mobile terminal, while target touch interaction operations that will update the content displayed in the video screen can be responded to and processed collaboratively by the mobile terminal and the cloud server, which can reduce response delays.

[0066] For ease of understanding, Figure 4 Several common types of interactive events are illustrated in FIG. 1 , as well as the triggering preconditions, triggering actions (i.e., touch interactive operation behaviors), and triggering results corresponding to each interactive event. The relationship between the triggering preconditions and the triggering actions will be described in the subsequent specific embodiments. It should be noted that Figure 4 The triggering actions are only examples and are not limited to this.

[0067] The following is an illustrative description of the interaction processing process of some of the above-mentioned touch interaction operation behaviors.

[0068] Figure 5 A flowchart of a cloud application interaction method provided by an embodiment of the present invention, which can be executed by a mobile terminal, such as Figure 5 As shown, the method may include the following steps:

[0069] 501. Receive video stream data of a cloud application sent by a cloud server, where the video stream data is obtained by encoding an interface of the cloud application.

[0070] 502. Decode the video stream data to display the decoded video picture.

[0071] 503. Based on the multiple touch interaction operation behaviors corresponding to the cloud application registered in the mobile terminal, identify the target touch interaction operation behavior triggered by the user on the video screen, where the target touch interaction operation behavior is a screen zoom operation behavior and / or a screen pan operation behavior.

[0072] 504. Perform corresponding zooming and / or panning processing on the locally displayed video image according to the image zooming operation behavior and / or the image panning operation behavior, wherein the video image in the cloud server remains unchanged.

[0073] This embodiment corresponds to Figure 4 The purpose of these two interactive events is to allow users to watch all or part of the video displayed on the mobile terminal screen more clearly and focusedly. The content of the video screen will not be updated, so the mobile terminal responds to the zoom operation and the pan operation locally.

[0074] based on Figure 4 From the trigger preconditions and trigger actions corresponding to the two interactive events exemplified in the example, it can be seen that when the mobile terminal recognizes that the user triggers the operation of touching the screen with three fingers at the same time on the screen displaying the video screen, it means that the user may want to trigger the screen zoom operation behavior or the screen translation operation behavior. Further recognition is performed. When it is further recognized that the user's three fingers are pinching (i.e. Figure 4 The sum of the distances between the three fingers and the center of the three fingers tends to decrease) or stretch (i.e. Figure 4 When the gesture feature of the three-finger position and the sum of the distances between the three-finger center points has a tendency to increase, it is determined that the user has triggered the screen zoom operation behavior, and the local video screen displayed on the screen can be zoomed in and out accordingly; when it is further recognized that the user's three fingers are moving in a certain direction (i.e. Figure 4 When the position of the three-finger center point in the screen moves more than 10px on the screen, it is determined that the user has triggered the screen panning operation, and the local device moves the video screen displayed on the screen in the direction of the finger movement. Among them, 10px represents a distance of 10 pixels.

[0075] It should be noted that since the mobile terminal responds to the above-mentioned screen zooming and panning operations locally, the video screen only has the corresponding zooming and panning effects on the mobile terminal side, and the interface of the cloud application in the cloud server does not have such change effects. In other words, the mobile terminal will not upload the screen zooming and panning operations to the cloud server.

[0076] Figure 6 A flowchart of a cloud application interaction method provided by an embodiment of the present invention, which can be executed by a mobile terminal, such as Figure 6 As shown, the method may include the following steps:

[0077] 601. Receive video stream data of a cloud application sent by a cloud server, where the video stream data is obtained by encoding an interface of the cloud application.

[0078] 602. Decode the video stream data to display the decoded video picture.

[0079] 603. Based on the multiple touch interaction operation behaviors corresponding to the cloud application registered in the mobile terminal, identify the target touch interaction operation behavior triggered by the user on the video screen, the target touch interaction operation behavior is the information input operation behavior, and the information input operation behavior includes the set touch operation behavior triggered by the input box area in the video screen.

[0080] 604 . Set the input box to an information input state, call the local input method of the mobile terminal to display an information input interface, and the information input interface includes a data paste button.

[0081] 605. In response to the user sending operation of the information content input through the information input interface, the information content and the indication information of the input box are sent to the cloud server, so that the cloud server displays the information content in the input box in the corresponding cloud application interface.

[0082] 606. In response to the user's selection operation of the data paste button, the target copy data and the indication information of the input box in the local clipboard are sent to the cloud server, so that the cloud server displays the target copy data in the input box in the corresponding cloud application interface.

[0083] In this embodiment, it is assumed that the video screen currently displayed by the mobile terminal corresponds to a cloud application interface in the cloud server.

[0084] This embodiment corresponds to Figure 4 There are two interactive events: invoking the local input method and pasting.

[0085] Assume that the video image displayed on the mobile terminal screen includes an input box, such as Figure 4As described in, when it is identified that the user has triggered two consecutive clicks with one finger within 1 second in the input box area of ​​the video screen (for example, within the vicinity of 5px), it is determined that the user has triggered the information input operation, and the input box is set to the information input state (for example, a flashing cursor in the input box, or a set colored border representing the selected state is added to the input box, etc.), and the local input method of the mobile terminal is called to display the information input interface. Among them, the common input method used locally in mobile terminals is the 9-grid input method, which is different from the input method used in the x86 architecture. The information input interface is the input interface corresponding to the input method.

[0086] In addition, in actual applications, the user may input information in the input box by directly typing the information content through an input method or by pasting the copied data in the clipboard. Therefore, the above information input interface also includes a data paste button.

[0087] Optionally, when it is identified that the user triggers the above-mentioned click operation, if it is further determined that there is copy data in the clipboard of the mobile terminal, the above-mentioned data paste button can be displayed, otherwise it can be not displayed or placed in an unselectable state.

[0088] When there is more than one copy data in the clipboard, the last copy data is determined as the target copy data.

[0089] When the user completes entering information content in the input box through the input method and triggers a sending operation, or when the user selects the data paste button and pastes the target copied data into the input box and triggers a sending operation, the mobile terminal sends the information content or target copied data typed by the user and the indication information of the input box to the cloud server, so that the cloud server locates the input box in the corresponding cloud application interface according to the indication information of the input box, and displays the information content or target copied data in the input box.

[0090] It is understandable that when the mobile terminal sends the above information to the cloud server, it is also necessary to comply with the data transmission protocol between the mobile terminal and the cloud server.

[0091] Figure 7 A flowchart of a cloud application interaction method provided by an embodiment of the present invention, which can be executed by a mobile terminal, such as Figure 7 As shown, the method may include the following steps:

[0092] 701. Receive video stream data of a cloud application sent by a cloud server, where the video stream data is obtained by encoding an interface of the cloud application.

[0093] 702. Decode the video stream data to display the decoded video picture.

[0094] 703. Based on the various touch interaction operation behaviors corresponding to the cloud application registered in the mobile terminal, identify the target touch interaction operation behavior triggered by the user on the video screen, the target touch interaction operation behavior is the function window awakening operation behavior, and the function window awakening operation behavior refers to the set touch operation behavior triggered in the video screen for awakening the target function window.

[0095] 704. Send a target notification message to the cloud server, where the target notification message includes the location where the user triggers the function window to invoke the operation in the video screen. The target notification message is used to indicate that the user has triggered the function window to invoke the operation at the location, so that the cloud server displays the target function window at the location in the corresponding cloud application interface, and transmits the video screen containing the target function window to the mobile terminal for display.

[0096] This embodiment corresponds to Figure 4 The right-click interaction event in .

[0097] like Figure 4 As shown in , when it is identified that the user triggers a touch operation behavior of clicking the screen with two fingers at the same time on the screen of the mobile terminal displaying the above video screen, it is determined that the user has triggered the function window awakening operation behavior. At this time, the mobile terminal can notify the cloud server of the trigger position corresponding to the operation behavior in the video screen, and inform the cloud server where the user triggered the function window awakening operation behavior in the video screen. The cloud server can then call the target function window corresponding to the video screen, display the target function window at this position, thereby forming a new interface, encode the new interface to obtain a new video screen, and transmit it to the mobile terminal for decoding and display.

[0098] above Figure 6 , Figure 7 The illustrated embodiment illustrates the process of the mobile terminal and the cloud server cooperating to respond to and process some touch interaction operations to update the video screen content. Through these examples, it can be seen that in the embodiments of the present invention, the recognition and partial processing of the user's touch interaction operation behavior are completed on the mobile terminal side as much as possible, and the processing logic of the cloud server is simplified, so that better response timeliness can be obtained.

[0099] The cloud application interaction device of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these devices can be configured using commercially available hardware components through the steps taught in this solution.

[0100] Figure 8 A schematic diagram of the structure of a cloud application interaction device provided by an embodiment of the present invention, the device is applied to the above-mentioned mobile terminal, such as Figure 8As shown, the device includes: a receiving module 11, a display module 12, and an interaction module 13.

[0101] The receiving module 11 is used to receive video stream data of a cloud application sent by a cloud server, wherein the video stream data is obtained by encoding the interface of the cloud application, and the system architecture of the cloud server is different from that of the mobile terminal.

[0102] The display module 12 is used to decode the video stream data to display the decoded video pictures.

[0103] The interaction module 13 is used to identify the target touch interaction operation behavior triggered by the user on the video screen based on the multiple touch interaction operation behaviors corresponding to the cloud application registered in the mobile terminal, and respond to the target touch interaction operation behavior; wherein the interaction components corresponding to the multiple touch interaction behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server; the target touch interaction operation behavior is included in the multiple touch interaction operation behaviors.

[0104] Optionally, the interaction module 13 is specifically used for: if the target touch interaction operation behavior belongs to the first touch interaction operation behavior of adjusting the visual effect of the video screen, then locally responding to the target touch interaction operation behavior; if the target touch interaction operation behavior belongs to the second touch interaction operation behavior of adjusting the picture content of the video screen, then collaboratively responding to the target touch interaction operation behavior with the cloud server.

[0105] Optionally, the first touch interaction operation behavior includes any one of the following: screen zooming operation behavior, screen translation operation behavior. Optionally, the second touch interaction operation behavior includes any one of the following: information input operation behavior, function window waving operation behavior.

[0106] Optionally, the target touch interaction operation behavior is a screen zoom operation behavior and / or a screen translation operation behavior, and the interaction module 13 is specifically used to: perform corresponding zooming and / or translation processing on the locally displayed video screen according to the screen zoom operation behavior and / or screen translation operation behavior, wherein the video screen in the cloud server remains unchanged.

[0107] Optionally, the target touch interaction operation behavior is an information input operation behavior, and the information input operation behavior includes a set touch operation behavior triggered by an input box area in the video screen. The interaction module 13 is specifically used to: set the input box to an information input state; call the local input method of the mobile terminal to display an information input interface; in response to the user's sending operation of the information content input through the information input interface, send the information content and the indication information of the input box to the cloud server, so that the cloud server displays the information content in the input box in the corresponding cloud application interface.

[0108] Optionally, the information input interface includes a data paste button, and the interactive module 13 is also used to: in response to the user's selection operation through the data paste button, send the target copied data in the local clipboard and the indication information of the input box to the cloud server, so that the cloud server displays the target copied data in the input box in the corresponding cloud application interface.

[0109] Optionally, the target touch interaction operation behavior is a function window awakening operation behavior, and the function window awakening operation behavior refers to a set touch operation behavior for awakening the target function window triggered in the video screen. The interaction module 13 is specifically used to: send a target notification message to the cloud server, the target notification message includes the position where the user triggers the function window awakening operation behavior in the video screen, and the target notification message is used to indicate that the user has triggered the function window awakening operation behavior at the position, so that the cloud server displays the target function window at the position in the corresponding cloud application interface, and transmits the video screen containing the target function window to the mobile terminal for display.

[0110] Figure 8 The device shown can execute the steps provided in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments, which will not be repeated here.

[0111] The embodiment of the present invention also provides a mobile terminal, such as Fig. 9 As shown, it may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable codes, and when the executable codes are executed by the processor 21, the cloud application interaction method in the above-mentioned embodiment is implemented.

[0112] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor in a mobile terminal, the cloud application interaction method provided in the aforementioned embodiment can be implemented.

[0113] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0114] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cloud application interaction method, characterized in that: Applied to a mobile terminal, the method comprises: Receiving video stream data of a cloud application sent by a cloud server, wherein the video stream data is obtained by encoding an interface of the cloud application, and a system architecture of the cloud server is different from a system architecture of the mobile terminal; Decoding the video stream data to display the decoded video picture; Based on the multiple touch interaction operation behaviors corresponding to the cloud application registered in the mobile terminal, identifying the target touch interaction operation behavior triggered by the user on the video screen; wherein the interaction components corresponding to the multiple touch interaction behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server; the target touch interaction operation behavior is included in the multiple touch interaction operation behaviors; The target touch interaction operation behavior is responded to and processed.

2. The method according to claim 1, characterized in that The responding process to the touch interaction operation behavior includes: If the target touch interaction operation behavior belongs to the first touch interaction operation behavior for adjusting the visual effect of the video screen, locally responding to the target touch interaction operation behavior; If the target touch interaction operation behavior belongs to the second touch interaction operation behavior of adjusting the screen content of the video screen, then collaboratively responding to the target touch interaction operation behavior with the cloud server.

3. The method according to claim 2, characterized in that The first touch interaction operation behavior includes any one of the following: a screen zooming operation behavior, a screen panning operation behavior.

4. The method according to claim 3, characterized in that The target touch interaction operation behavior is a screen zooming operation behavior and / or a screen translation operation behavior; The local response to the target touch interaction operation behavior includes: According to the picture zooming operation behavior and / or picture translation operation behavior, the locally displayed video picture is correspondingly zoomed and / or translated, wherein the video picture in the cloud server remains unchanged.

5. The method according to claim 2, characterized in that: The second touch interaction operation behavior includes any one of the following: an information input operation behavior, a function window awakening operation behavior.

6. The method according to claim 5, characterized in that The target touch interaction operation behavior is an information input operation behavior, and the information input operation behavior includes a setting touch operation behavior triggered by an input box area in the video screen; The collaboratively responding to the target touch interaction operation behavior with the cloud server includes: Setting the input box to an information input state; Invoking a local input method of the mobile terminal to display an information input interface; In response to a user sending operation of information content input through the information input interface, the information content and indication information of the input box are sent to the cloud server, so that the cloud server displays the information content in the input box in the corresponding cloud application interface.

7. The method according to claim 6, characterized in that The information input interface includes a data paste button; The collaboratively responding to the target touch interaction operation behavior with the cloud server includes: In response to the user's selection operation through the data paste button, the target copy data in the local clipboard and the indication information of the input box are sent to the cloud server, so that the cloud server displays the target copy data in the input box in the corresponding cloud application interface.

8. The method according to claim 5, characterized in that The target touch interaction operation behavior is a function window calling operation behavior, and the function window calling operation behavior refers to a set touch operation behavior triggered in the video screen for calling up a target function window; The collaboratively responding to the target touch interaction operation behavior with the cloud server includes: A target notification message is sent to the cloud server, wherein the target notification message includes the position in the video screen where the user triggers the function window to invoke the operation behavior, and the target notification message is used to indicate that the user has triggered the function window to invoke the operation behavior at the position, so that the cloud server displays the target function window at the position in the corresponding cloud application interface, and transmits the video screen containing the target function window to the mobile terminal for display.

9. A mobile terminal, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the cloud application interaction method as described in any one of claims 1 to 8 is executed.

10. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor in the mobile terminal, the cloud application interaction method according to any one of claims 1 to 8 is executed.

11. A cloud application system, characterized in that: include: A cloud server running a cloud application, and a mobile terminal accessing the cloud application, wherein the system architecture of the cloud server is different from the system architecture of the mobile terminal; The cloud server is used to encode the interface of the cloud application into video stream data to send to the mobile terminal; The mobile terminal is used to receive the video stream data sent by the cloud server, and decode the video stream data to display the decoded video picture; And, based on the multiple touch interaction operation behaviors corresponding to the cloud application registered in the mobile terminal, the target touch interaction operation behavior triggered by the user on the video screen is identified, and the target touch interaction operation behavior is responded to and processed; wherein, the interaction components corresponding to the multiple touch interaction operation behaviors are adapted to the system architecture of the mobile terminal, but not to the system architecture of the cloud server, and the target touch interaction operation behavior is included in the multiple touch interaction operation behaviors.

Images