Cloud application picture transmission method, device and storage medium

Abstract

The application provides a cloud application picture transmission method, device and storage medium. The method comprises the following steps: loading encoded multimedia data which needs to be displayed at a current time in a cloud application interface; generating first cloud application picture data at the current time according to the multimedia data and the cloud application interface; determining playing area picture data and non-playing area picture data in the first cloud application picture data and picture layout information corresponding to the playing area picture data; and sending the playing area picture data and the picture layout information corresponding to the playing area picture data to a corresponding client of the cloud application respectively, so that the client decodes the playing area picture data and renders the decoded playing area picture data according to the picture layout information. Based on the cloud application picture transmission method, the computing power consumption of the cloud in the process of transmitting cloud application pictures to the client can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, and in particular to a cloud application picture transmission method, device and storage medium. BACKGROUND

[0002] A cloud application runs on a cloud server, and a user end does not need to install an application. The user can use a remote cloud application through a client or a webpage. The cloud server encodes a cloud application interface in the form of a video stream through a video stream push protocol, and then sends a frame of cloud application picture data encoded to a client, so that the user can use the cloud application of the cloud end. In actual application, the cloud end can run various cloud applications such as cloud games, metaverse, etc. Of course, in a general way, cloud desktops, cloud phones, etc. can also be regarded as a kind of cloud application.

[0003] In the running process of the cloud application, multimedia data (such as video, animation, picture, etc.) may need to be loaded in a certain interface of the cloud application. The multimedia data may come from the outside, or may be originally set in the cloud application interface. These multimedia data have often been encoded in a certain encoding manner. In this case, the transmission process of the cloud application interface to the client is: the cloud server loads the multimedia data, decodes the multimedia data, then plays the decoded multimedia data in the cloud application interface, then samples the cloud application picture of the played multimedia data frame by frame, samples a frame of cloud application picture, encodes the sampled cloud application picture into a video stream based on a certain video stream push protocol, and sends the encoded frame of cloud application picture to the client. The client decodes the frame of cloud application picture by using a corresponding decoding manner and renders and displays it.

[0004] Because the cloud server needs to perform decoding and rendering display of the multimedia data, and processing of encoding of the cloud application interface containing the multimedia data when transmitting the cloud application picture to the client, repeated encoding, decoding and rendering processing brings large computing power overhead, and also affects the picture quality. SUMMARY

[0005] The embodiments of the present application provide a cloud application picture transmission method, device and storage medium, to reduce the computing power overhead of the cloud end in the process of transmitting the cloud application picture to the client.

[0006] In a first aspect, the embodiments of the present application provide a cloud application picture transmission method applied to a cloud server, and the method comprises:

[0007] loading encoded multimedia data that needs to be displayed at a current time in a cloud application interface;

[0008] generate first cloud application picture data of the current moment according to the multimedia data and the cloud application interface;

[0009] determine the play region picture data and the non-play region picture data in the first cloud application picture data and picture layout information corresponding to the play region picture data;

[0010] send the play region picture data and the picture layout information corresponding to the play region picture data to a client corresponding to the cloud application respectively, so that the client decodes the play region picture data and renders the decoded play region picture data according to the picture layout information.

[0011] In a second aspect, an embodiment of the present application provides a cloud application picture transmission device applied to a cloud server, and the device comprises:

[0012] The obtaining module is configured to load encoded multimedia data to be displayed at a current moment in a cloud application interface, and generate first cloud application picture data of the current moment according to the multimedia data and the cloud application interface.

[0013] The determining module is configured to determine the play region picture data and the non-play region picture data in the first cloud application picture data and picture layout information corresponding to the play region picture data; wherein the multimedia data is contained in the play region picture data.

[0014] The sending module is configured to send the play region picture data and the picture layout information corresponding to the play region picture data to a client corresponding to the cloud application respectively, so that the client decodes the play region picture data and renders the decoded play region picture data according to the picture layout information.

[0015] In a third aspect, an embodiment of the present application provides a cloud server, comprising a memory, a processor and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the cloud application picture transmission method in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a non-transitory machine readable storage medium, and the non-transitory machine readable storage medium stores executable code, and when the executable code is executed by a processor of a cloud server, the processor executes the cloud application picture transmission method in the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides a cloud application picture transmission method applied to a client corresponding to a cloud application, and the method comprises:

[0018] send a data playing request to a cloud server corresponding to the cloud application, so that the cloud server loads encoded multimedia data to be displayed at a current time in a cloud application interface based on the data playing request;

[0019] receive playing region picture data in first cloud application picture data sent by the cloud server and picture layout information corresponding to the playing region picture data, wherein the first cloud application picture data is generated according to the multimedia data and the cloud application interface, and the multimedia data is contained in the playing region picture data;

[0020] decode the playing region picture data and render the decoded playing region picture data according to the picture layout information.

[0021] In a sixth aspect, an embodiment of the present application provides a cloud application picture transmission apparatus applied to a client corresponding to a cloud application, and the apparatus comprises:

[0022] a sending module configured to send a data playing request to a cloud server corresponding to the cloud application, so that the cloud server loads encoded multimedia data to be displayed at a current time in a cloud application interface based on the data playing request;

[0023] a receiving module configured to receive playing region picture data in first cloud application picture data sent by the cloud server and picture layout information corresponding to the playing region picture data, wherein the first cloud application picture data is generated according to the multimedia data and the cloud application interface, and the multimedia data is contained in the playing region picture data;

[0024] a rendering module configured to decode the playing region picture data and render the decoded playing region picture data according to the picture layout information.

[0025] In a seventh aspect, an embodiment of the present application provides a terminal device, which comprises a memory, a processor and a communication interface, wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the cloud application picture transmission method in the fifth aspect.

[0026] In an eighth aspect, an embodiment of the present application provides a non-transitory machine readable storage medium, which stores executable code, and when the executable code is executed by a processor of a terminal device, the processor executes the cloud application picture transmission method in the fifth aspect.

[0027] The embodiment of the present application provides a cloud application picture transmission method which can be used for transmitting a cloud application picture to a client. Based on the method, a cloud server acquires originally coded multimedia data which needs to be displayed in a cloud application interface, generates first cloud application picture data which needs to be sent to the client according to other content which needs to be displayed in the cloud application interface, and the cloud server does not render and display the cloud application interface locally. In the process of transmitting the first cloud application picture data to the client, first, the first cloud application picture data is divided into two parts based on the dynamic change characteristics of the picture data, one is the play area picture data, and the remaining is the non-play area picture data. The dynamically changed multimedia data is generally included in the play area picture data. Then, the play area picture data and the picture layout information corresponding to the play area picture data are sent to the client respectively, so that the client decodes the play area picture data and renders and displays according to the picture layout information. In this way, the complete cloud application interface is synthesized at the client.

[0028] Through the above process, it can be known that the cloud server no longer needs to decode the multimedia data and render and display, and no longer needs to capture a frame of complete cloud application picture containing the multimedia data and then encode, but directly redirects the play area picture data in the cloud application picture data to the client to decode and display based on the data variability after loading the complete cloud application picture data into the memory. The cloud end does not need additional decoding, rendering and encoding processing, and only needs to send the updated non-play area picture data and the picture layout information of the play area picture data to the client, thereby saving the cloud end computing power consumption and ensuring the picture quality. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0030] Figure 1 A schematic diagram of a cloud application picture transmission system provided by the embodiment of the present application;

[0031] Figure 2 A principle schematic diagram of a traditional cloud application picture transmission method;

[0032] Figure 3 A flowchart of a cloud application picture transmission method provided by the embodiment of the present application;

[0033] Figure 4a A principle schematic diagram of a cloud application picture transmission method provided by the embodiment of the present application;

[0034] Figure 4b An execution process schematic diagram of a cloud application picture transmission method provided by an embodiment of the present application is provided.

[0035] Figure 5 A flow chart of a cloud application picture transmission method provided by an embodiment of the present application is provided.

[0036] Figure 6 A flow chart of a cloud application picture transmission method provided by an embodiment of the present application is provided.

[0037] Figure 7 An application schematic diagram of a cloud application picture transmission method provided by an embodiment of the present application is provided.

[0038] Figure 8 An application schematic diagram of a cloud application picture transmission method provided by an embodiment of the present application is provided.

[0039] Figure 9 A structure schematic diagram of a cloud application picture transmission device provided by an embodiment of the present application is provided.

[0040] Figure 10 A structure schematic diagram of a cloud application picture transmission device provided by an embodiment of the present application is provided.

[0041] Figure 11 A structure schematic diagram of a cloud server provided by an embodiment of the present application is provided.

[0042] Figure 12 A structure schematic diagram of a terminal device provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0044] In addition, the step sequence in each of the following method embodiments is only an example, not a strict limitation.

[0045] Figure 1 A schematic diagram of a cloud application picture transmission system provided by an embodiment of the present application is provided, as shown in the figure, the picture transmission system includes a client, a cloud server, and a plurality of data sources (assuming N in the figure). Figure 1

[0046] ​The cloud server can be built based on an operating system (such as an Android operating system) to construct some virtual machines for running a certain cloud application. A plurality of cloud applications can be deployed in one cloud server.

[0047] During the running of the cloud application, based on the interactive behavior triggered by the user through the client or some links set in the cloud application, multimedia data provided by an external data source may need to be loaded, such as Figure 1 Some external data sources as shown in the figure can be some video websites and the like. The multimedia data includes audio and video, animation, pictures and the like.

[0048] In some actual application scenarios, multimedia data such as video data is already encoded by using a certain encoding mode when the multimedia data is generated, so that when the cloud server running the cloud application downloads the video data from the data source, the downloaded video data is already encoded.

[0049] Based on the above system composition, in combination with Figure 2 A traditional cloud application picture transmission scheme is exemplarily illustrated.

[0050] As shown in Figure 2 It is assumed that multimedia data needs to be displayed in a certain cloud application interface, at which time the transmission process is: the cloud server loads the encoded multimedia data from the corresponding data source to the local cloud server, then the cloud server decodes the multimedia data by using an internally set decoder (such as a Media coder), then the decoded multimedia data is played in the cloud application interface, that is, a frame of multimedia data is decoded and rendered. Then, the cloud application interface playing the multimedia data is frame by frame captured, and each frame of the captured cloud application picture is encoded into a video stream based on an encoding mode provided in a certain video stream pushing protocol, and the encoded frame of the cloud application picture is sent to the client. The client decodes and renders the frame of the cloud application picture by using a corresponding decoding mode.

[0051] The above data stream needs to go through the process of decoding-rendering display-encoding-decoding, and repeated decoding, rendering and encoding consume a large amount of computing power of the cloud server.

[0052] In order to reduce the computing power consumption of the cloud server, in the embodiment of the application, the working process of each component in the above picture transmission system can be summarized as:

[0053] The client sends a data playing request to the cloud server corresponding to the cloud application, so that the cloud server loads the encoded multimedia data that needs to be displayed at the current time in a certain cloud application interface based on the data playing request.

[0054] The cloud server loads the encoded multimedia data, generates first cloud application picture data of a current time according to the loaded multimedia data and the cloud application interface, determines the playing area picture data and the non-playing area picture data in the first cloud application picture data and the picture layout information corresponding to the playing area picture data, and sends the playing area picture data and the picture layout information corresponding to the playing area picture data to the client corresponding to the cloud application respectively.

[0055] The client decodes the playing area picture data and renders the decoded playing area picture data according to the picture layout information.

[0056] In fact, if the non-playing area picture data has been updated compared with the last non-playing area picture data sent to the client (last non-playing area picture data), the current non-playing area picture data is also sent to the client to make the client render the non-playing area picture data.

[0057] It should be noted that the cloud application picture data in the embodiment of the application refers to picture data waiting to be rendered, rather than picture data that has been rendered. Moreover, if the interactive behavior triggered by a user in the cloud application interface and the dynamically changing multimedia data that need to be loaded in the interface are not considered, and it is assumed that the cloud application interface is fixed and unchangeable, the cloud application picture obtained by the cloud server at different times is actually the cloud application interface. However, if additional dynamically changing content is added to the cloud application interface, the cloud application picture at different times not only contains the cloud application interface, but also includes corresponding dynamic content, such as multimedia data and interactive behavior data.

[0058] The specific execution process of the cloud server in the cloud application picture transmission process will be described in detail below with reference to the following embodiments.

[0059] Figure 3 A flowchart of a cloud application picture transmission method provided by the embodiment of the application is shown in FIG. 1. Figure 3 The method can be applied to a cloud server and specifically includes the following steps.

[0060] 301. Load the encoded multimedia data that needs to be displayed at a current time in the cloud application interface.

[0061] 302. Generate first cloud application picture data of a current time according to the multimedia data and the cloud application interface.

[0062] 303. Determine the playing area picture data and the non-playing area picture data in the first cloud application picture data and the picture layout information corresponding to the playing area picture data.

[0063] 304. Send the playback area screen data and the corresponding screen layout information to the client of the cloud application respectively, so that the client can decode the playback area screen data and render the decoded playback area screen data according to the screen layout information.

[0064] Suppose a cloud application screen A is displayed on the client's screen. Screen A contains at least one video playback window with a play button. Multiple video options are displayed around the video playback window for the user to choose from. If the user selects a target video using a mouse or touch, the client sends a data playback request containing this selection information to the cloud server. Assume that the cloud application interface displayed after the user selects the target video is still screen A, but the data within the video playback window has changed. Based on the data playback request received at time T1, the cloud server loads the encoded target video data from the corresponding data source into its memory. Simultaneously, the cloud server also loads the interface data of cloud application screen A into its memory. Therefore, based on the interface data of cloud application screen A and the target video data, the cloud server can obtain the first cloud application screen data at the current moment. In general, interface data contains descriptive information about how and what data should be displayed at each pixel position in the cloud application interface A. If we compare the cloud application interface A to an image, the interface data is the information describing this image, including attribute information such as pixel value, color, and brightness at each pixel position.

[0065] In this embodiment of the invention, after obtaining the first cloud application screen data based on the encoded multimedia data and cloud application interface data loaded into memory, the cloud server does not render and display the corresponding cloud application screen on the cloud server. The cloud application screen data mentioned in this embodiment refers to the data required to render and display a complete frame of the cloud application interface.

[0066] After receiving the aforementioned first cloud application screen data, the cloud server, based on the dynamic nature of the cloud application screen data, divides it into two parts: the playback area screen data and the remaining non-playback area screen data. Dynamically changing multimedia data is generally included in the playback area screen data.

[0067] It is understood that the first cloud application screen data includes the screen data to be displayed within the aforementioned video playback window, as well as the interface data contained within the cloud application interface A outside the video playback window. Specifically, the screen data to be displayed within the video playback window is included in the playback area screen data, while the interface data outside the video playback window may be partially included in the playback area screen data, or it may all be included in the non-playback area screen data.

[0068] Specifically, the dynamic change of the cloud application picture data refers to comparing the cloud application picture data generated at different time points (different sampling time points or different loading time points) in a short time during the use of a certain cloud application by the same user to determine the dynamic change region in the different cloud application picture data, which corresponds to the playing region picture data.

[0069] For the first cloud application picture data generated at the current time point, the first cloud application picture data is compared with the second cloud application picture data generated at a historical time point to determine the playing region picture data in the first cloud application picture data as the picture data different from the second cloud application picture data, and the non-playing region picture data as the same picture data. The historical time point can be assumed to be (T1-Δt) if the cloud server continuously generates one frame of cloud application picture data to be sent to the client at a frequency of Δt, and the first cloud application picture data is obtained at T1.

[0070] After the playing region picture data and the non-playing region picture data in the first cloud application picture data are determined through the above comparison process, the playing region picture data and the non-playing region picture data, and the picture layout information of the playing region picture data can be sent to the client.

[0071] The non-playing region picture data has a lower update frequency than the playing region picture data. If the non-playing region picture data in the second cloud application picture data is different from the non-playing region picture data in the first cloud application picture data, the non-playing region picture data in the first cloud application picture data is sent to the client to render the non-playing region picture data in the first cloud application picture data. That is, if the non-playing region picture data determined at the current time point is the same as the non-playing region picture data determined at the last historical time point, the non-playing region picture data does not need to be repeatedly sent to the client.

[0072] The picture layout information simply includes the position of the playing region in the complete cloud application picture and the view level information. The picture layout information can be extracted from the related attribute information of the cloud application interface and the attribute information of the multimedia data.

[0073] In an optional embodiment, after obtaining the non-play area picture data in the first cloud application picture data, a comparison can be made to determine whether the non-play area picture data in the second cloud application picture data is consistent. If not, the non-play area picture data in the first cloud application picture data is sent to the client. If consistent, the non-play area picture data in the first cloud application picture data is not sent to the client.

[0074] Due to the definition of the play area and non-play area picture data, in actual application, the update frequency of the non-play area picture data is much lower than that of the play area picture data. Therefore, the same non-play area picture data does not need to be repeatedly sent to the client, so as to reduce the bandwidth cost of the cloud server.

[0075] In addition, when the cloud server sends the play area picture data and the non-play area picture data in the first cloud application picture data to the client, the play area picture data is directly sent to the client, while the non-play area picture data can be encoded by using the encoding mode provided in the video stream pushing protocol and then transmitted to the client.

[0076] For convenience, the encoding mode used for encoding the non-play area picture data is referred to as a second encoding mode, while the encoding mode used by the data source for encoding the multimedia data is referred to as a first encoding mode. The two encoding modes can be different. Since the dynamically changing multimedia data is generally included in the play area picture data after the determination of the play area and non-play area picture data, the client can use a first decoder (such as a Media Coder) to decode the play area picture data, wherein the decoding mode used by the first decoder corresponds to the first encoding mode used by the multimedia data. The client can use a second decoder to decode the non-play area picture data, wherein the decoding mode used by the second decoder corresponds to the second encoding mode used by the non-play area picture data. It should be noted that if the play area picture data includes other data in addition to the multimedia data, such as user interaction behavior data, the server also encodes the other data by using the first encoding mode, so that the play area picture data includes the multimedia data and the other data encoded by the first encoding mode. The client decodes the play area picture data by using the first decoder.

[0077] Compared with the traditional cloud application picture transmission scheme, actually, in the scheme provided by the embodiment of the application, the computing power of the client side does not increase, because in the traditional scheme, the client also needs to decode and then render and display the received complete and encoded cloud application picture, in the embodiment of the application, the client respectively decodes and renders the play area picture data and decodes and renders the non-play area picture data, the play area picture data and the non-play area picture data are part of the complete cloud application picture data, so the computing power does not increase. Moreover, as described above, it is assumed that the cloud server performs N times of different cloud application picture data transmission to the client, wherein N different play area picture data need to be transmitted in turn, but only M different non-play area picture data need to be transmitted in turn, M is less than N, so that the decoding computing power of the client for the non-play area picture data is lower.

[0078] In summary, in the embodiment of the application, by directly redirecting and pushing the streaming multimedia data and the picture layout information thereof to the client side for decoding and rendering display, the cloud server side multimedia data decoding, screenshot (grabbing the cloud application picture), and re-encoding processing process can be omitted, and the computing resources of the cloud server are saved. Moreover, the client directly processes the original multimedia data instead of the grabbed cloud application picture, and better picture quality and perception can be provided.

[0079] In an optional embodiment, the comparison of the first cloud application picture data and the second cloud application picture data can be in the form of comparison of data in pieces. One piece of data can be data corresponding to one pixel point, which includes pixel position, pixel value, view level, and various related attribute information.

[0080] In another optional embodiment, to improve the processing efficiency, the following comparison method can also be used:

[0081] Determine a target interface component contained in the cloud application interface, the target interface component being a preset type of interface component whose displayed data can dynamically change;

[0082] Compare whether the picture data required to be displayed on the target interface component in the first cloud application picture data and the second cloud application picture data is consistent;

[0083] If not consistent, it is determined that the picture data required to be displayed on the target interface component in the first cloud application picture data is contained in the play area picture data.

[0084] That is, some types of interface components on which the displayed data often dynamically changes can be preset as target interface components, such as a video playing window, a pop-up window, an input box, and the like. Based on the known interface layout description information of the cloud application interface, the various interface components contained in the application interface and the corresponding positions thereof can be known. Then, the target interface components contained in the several interface components in the cloud application interface are determined. Since the first cloud application picture data and the second cloud application picture data correspond to the same cloud application interface, whether the picture data required to be displayed on the same target interface components in the first cloud application picture data and the second cloud application picture data is consistent can be compared to determine the playing area picture data in the first cloud application picture data. Specifically, if the corresponding picture data of the same target interface components in the two cloud application picture data is different, it is considered that the picture data corresponding to the target interface components in the first cloud application picture data is the playing area picture data that should be contained in the first cloud application picture data.

[0085] Although the comparison manner described above can miss some data that should belong to the playing area picture data, generally, most of the picture data with dynamic change characteristics can be identified, and the playing area picture data obtained through the comparison manner is directly sent to the client for decoding, rendering and display, which can also reduce the computing overhead of the cloud server and ensure the picture quality of the client.

[0086] In order to more intuitively understand the difference between the cloud application picture transmission scheme provided by the embodiments of the present application and the traditional scheme, the following Figure 4a and Figure 4b are schematically described.

[0087] Figure 4a A principle schematic diagram of a cloud application picture transmission method provided by the embodiments of the present application.

[0088] As shown in Figure 4a , it is assumed that multimedia data needs to be displayed in a certain cloud application interface, and the transmission process at this time is: the cloud server loads the encoded multimedia data from the corresponding data source to the local cloud server, then generates the first cloud application picture data at the current time according to the loaded encoded multimedia data and the cloud application interface, and determines the playing area picture data and the non-playing area picture data in the first cloud application picture data and the picture layout information corresponding to the playing area picture data. Then, the playing area picture data, the encoded non-playing area picture data and the picture layout information are sent to the client. The client renders the non-playing area picture data and decodes the playing area picture data, and renders the decoded playing area picture data according to the picture layout information.

[0089] Figure 4bAn execution process diagram of a cloud application picture transmission method provided by an embodiment of the present application is shown in Figure 4b It is assumed that the client needs to play video data (i.e. multimedia data) 402 in the cloud application interface 401.

[0090] The cloud server loads the encoded video data 402 in the cloud application interface 401, and generates first cloud application picture data according to the cloud application interface 401 and the video data 402. Then, the cloud server determines the playing region picture data, the non-playing region picture data and the picture layout information corresponding to the playing region picture data in the first cloud application picture data. In this embodiment, the video data 402 is taken as the playing region picture data, and the cloud application interface 401 is taken as the non-playing region picture data. Accordingly, the picture layout information is the picture layout information corresponding to the video data 402.

[0091] The cloud server sends the playing region picture data, the non-playing region picture data and the picture layout information to the client respectively. The client renders the non-playing region picture data and decodes the playing region picture data, and renders the decoded playing region picture data according to the picture layout information to obtain the cloud application picture.

[0092] Figure 5 A flowchart of a cloud application picture transmission method provided by an embodiment of the present application is shown in Figure 5 The method can be applied to a cloud server, and specifically includes the following steps:

[0093] 501. Load the encoded multimedia data that needs to be displayed at the current time in the cloud application interface. The multimedia data has been encoded by using a first encoding mode.

[0094] 502. If it is detected that the user triggers the interaction behavior data in the cloud application interface at the current time, generate the first cloud application picture data at the current time according to the multimedia data, the interaction behavior data and the cloud application interface.

[0095] 503. Determine the playing region picture data and the non-playing region picture data in the first cloud application picture data, and the picture layout information corresponding to the playing region picture data.

[0096] 504. If it is determined that the playing region picture data contains the interaction behavior data, encode the interaction behavior data by using the first encoding mode.

[0097] 505、The play area picture data, the non-play area picture data encoded by the second encoding mode and the picture layout information corresponding to the play area picture data are respectively sent to the client corresponding to the cloud application, so that the client decodes and renders the non-play area picture data, decodes the play area picture data and renders the decoded play area picture data according to the picture layout information.

[0098] In the embodiment, it is assumed that the non-play area picture data determined at the current time has been updated compared with the non-play area picture data last sent to the client.

[0099] In the embodiment, it is assumed that the user triggers an interaction behavior on the screen of the client during the process of watching a multimedia data through a certain cloud application, so that the client collects corresponding interaction behavior data and sends the interaction behavior data to the cloud server. The occurrence of the interaction behavior data also causes the cloud application picture data to be updated, which can not only cause new play area picture data to be generated, but also cause the non-play area picture data to be updated. In the interaction behavior data, it is described that the user triggers what kind of interaction behavior at what position in the locally displayed cloud application picture, and what interface element corresponds to the interaction behavior. For example, the user drags the mouse, and the interaction behavior data includes different coordinate points and mouse graphics. For another example, the user clicks the mouse behavior, and the corresponding interaction behavior data includes the corresponding coordinate point and the click operation behavior information.

[0100] Specifically, in the embodiment, it is assumed that a frame of cloud application picture data corresponding to a certain cloud application interface needs to be generated at the current T1 time. At this time, on the one hand, the encoded multimedia data (encoded by the first encoding mode) that needs to be displayed on the cloud application interface is loaded, and on the other hand, it is assumed that the interaction behavior data sent by the client is also received at this time. Therefore, the encoded multimedia data, the interaction behavior data and the interface data of the cloud application interface are all loaded into the memory, so that the first cloud application picture data including the multimedia data, the interaction behavior data and the interface data of the cloud application interface can be obtained. Then, through the comparison process introduced in the foregoing embodiments, the play area picture data and the non-play area picture data in the first cloud application picture data are determined, and the picture layout information corresponding to the play area picture data is obtained.

[0101] Since the first cloud application picture data at T1 time is assumed to have added the interactive behavior data and the multimedia data content has also changed compared to the second cloud application picture data at T2 time, it can be understood that the playing area picture data in the first cloud application picture data at this time will contain the interactive behavior data and the multimedia data encoded by the first encoding mode. At this time, the cloud server encodes the interactive behavior data by using the first encoding mode, so that the playing area picture data contains the data encoded by the same encoding mode. In this way, after the playing area picture data is sent to the client, the client can decode the playing area picture data using the same decoder.

[0102] It can be understood that the interactive behavior data in the second cloud application picture data at T2 time is the same as the interactive behavior data contained in the first cloud application picture data at T1 time, and the playing area picture data in the first cloud application picture data will not contain the interactive behavior data. For example, the user places the mouse graphic at a certain position on the screen at T2 time and keeps it still, so that the interactive behavior data contained in the second cloud application picture data at T2 time will be the same as the interactive behavior data contained in the first cloud application picture data at T1 time.

[0103] From the above embodiments, it can be seen that a key point of the scheme provided by the embodiments of the application is that the playing area picture data in the cloud application picture data displayed on the client screen needs to be redirected. Based on the comparison of the cloud application picture data loaded at different times in the cloud, the area changed in the picture is defined as the playing area. The playing area picture data often includes multimedia data and interactive data whose contents need to be displayed in the picture and change constantly, and is redirected to the client for direct decoding, rendering and display, without the need for decoding, rendering and picture capturing in the cloud. The so-called redirection simply means that the operation of decoding multimedia data performed by the cloud server in the traditional scheme is replaced by the client.

[0104] In actual application, the decoder used by the client to decode the playing area picture data can be the same as the decoder used to decode the multimedia data in the cloud server. If the decoder is based on a certain operating system, it can be understood that the operating system used by the virtual machine running the cloud application in the cloud server is consistent with the operating system on the client side.

[0105] Figure 6 A flowchart of a cloud application picture transmission method provided by the embodiments of the application is shown in FIG. 1. Figure 6 As shown in FIG. 1, the method can be applied to a client corresponding to a certain cloud application, and specifically includes the following steps.

[0106] 601. Send a data playback request to the cloud server corresponding to the cloud application, so that the cloud server can load the encoded multimedia data that needs to be displayed in the cloud application interface at the current moment based on the data playback request.

[0107] 602. Receive the playback area screen data, non-playback area screen data, and screen layout information corresponding to the playback area screen data from the first cloud application screen data sent by the cloud server; wherein, the first cloud application screen data is generated based on the multimedia data and the cloud application interface.

[0108] 603. Render the non-playback area screen data, decode the playback area screen data, and render the decoded playback area screen data according to the screen layout information.

[0109] The cloud application screen transmission solution provided in this invention is cross-application and cross-data type. First, regarding cross-data type, simply put, the screen data in the playback area mentioned above can include different types of multimedia data, as well as interactive behavior data. These different types of multimedia data include, for example, videos, animations, images, tables, and so on.

[0110] For cross-application, the following will be combined Figure 7 and Figure 8 The examples illustrate two different application scenarios.

[0111] exist Figure 7 In the example, assuming the user is using a cloud desktop, such as... Figure 7 As shown, the cloud desktop 700 contains multiple cloud applications: Application 1, Application 2, Application 3, and Application 4. Assume a user is using Application 1 and Application 2. An application interface 701 of Application 1 is displayed in the middle area of ​​the cloud desktop, and an application interface 702 of Application 2 is displayed in the upper right corner of the cloud desktop. Assume Application 1 is a video application and Application 2 is a social networking application. The user is watching a video using Application 1 and chatting with a friend using Application 2.

[0112] In the application scenarios described above, the cloud server will generate complete cloud application screen data at different times based on the interface data of the cloud desktop, application 1, and application 2 loaded at different times. As can be seen from the comparison process above, at this time, the playback area screen data in a certain cloud application screen data corresponds to the area where application interface 701 and application interface 702 are shown in the figure, and the remaining area on the cloud desktop corresponds to the non-playback area screen data.

[0113] exist Figure 8In the embodiment, it is assumed that a user is using a cloud application A, and a link address corresponding to a video is contained in an interface of the cloud application A, the video is provided by another cloud application B and is encoded. If the user clicks the link address, an application interface 801 as shown in FIG. 8A is displayed on the interface of the cloud application A. Figure 8 The application interface 801 contains a video playing window for playing the video.

[0114] In the above-mentioned assumption, the cloud server forms complete cloud application picture data at different time based on the encoded video frames loaded at different time and the interface data of the application interface 801. As known from the comparison process above, at this time, the playing region picture data in the cloud application picture data corresponds to the video playing window shown in the diagram, and the remaining region corresponds to the non-playing region picture data. The cloud application picture transmission device of one or more embodiments of the present application will be described in detail below. Those skilled in the art can understand that these devices can be configured by using commercially available hardware components and the steps taught in the present solution.

[0115] Figure 9 A structure diagram of a cloud application picture transmission device provided by an embodiment of the present application is shown in FIG. 8B, which is applied to a cloud server and includes an acquisition module 11, a determination module 12 and a sending module 13. Figure 9

[0116] The acquisition module 11 is configured to load the encoded multimedia data to be displayed at the current time in the cloud application interface, and generate first cloud application picture data at the current time according to the multimedia data and the cloud application interface.

[0117] The determination module 12 is configured to determine the playing region picture data and the non-playing region picture data in the first cloud application picture data and the picture layout information corresponding to the playing region picture data, wherein the multimedia data is contained in the playing region picture data.

[0118] The sending module 13 is configured to send the playing region picture data and the picture layout information corresponding to the playing region picture data to the client corresponding to the cloud application, so that the client decodes the playing region picture data and renders the decoded playing region picture data according to the picture layout information.

[0119] Optionally, the determination module 12 is specifically configured to compare the first cloud application picture data and second cloud application picture data generated at a historical time, so as to determine the picture data different from the second cloud application picture data in the first cloud application picture data as the playing region picture data, and the same picture data as the non-playing region picture data.

[0120] ​Optionally, the determining module 12 is specifically configured to determine a target interface component contained in the cloud application interface, the target interface component being a preset type of interface component whose displayed data can dynamically change; compare whether the data to be displayed on the target interface component in the first cloud application picture data and the second cloud application picture data are consistent; and if not, determine that the data to be displayed on the target interface component in the first cloud application picture data is contained in the playing area picture data.

[0121] Optionally, the sending module 13 is specifically configured to, if the non-playing area picture data in the second cloud application picture data is different from the non-playing area picture data in the first cloud application picture data, send the non-playing area picture data in the first cloud application picture data to the client, so that the client renders the non-playing area picture data in the first cloud application picture data.

[0122] Optionally, the encoded multimedia data is obtained by encoding the multimedia data in the first encoding manner by a corresponding data source.

[0123] Optionally, the obtaining module 11 is specifically configured to, if it is detected that the user triggers the interaction behavior data in the cloud application interface at the current time, generate the first cloud application picture data at the current time according to the multimedia data, the interaction behavior data and the cloud application interface. Correspondingly, before the sending module 13 sends the playing area picture data to the client corresponding to the cloud application, the sending module 13 is further configured to, if it is determined that the interaction behavior data is contained in the playing area picture data, encode the interaction behavior data in the first encoding manner.

[0124] Optionally, the sending module 13 is specifically configured to encode the non-playing area picture data in the first cloud application picture data in the second encoding manner, and send the encoded non-playing area picture data to the client.

[0125] Figure 10 A structure diagram of a cloud application picture transmission device provided by an embodiment of the application is shown in FIG. 1. Figure 10 As shown in FIG. 1, the device is applied to a client corresponding to a cloud application, and includes a sending module 21, a receiving module 22 and a rendering module 23.

[0126] The sending module 21 is configured to send a data playing request to a cloud server corresponding to the cloud application, so that the cloud server loads the encoded multimedia data to be displayed at the current time in the cloud application interface based on the data playing request.

[0127] The receiving module 22 is used to receive the playback area screen data and the screen layout information corresponding to the playback area screen data in the first cloud application screen data sent by the cloud server; wherein, the first cloud application screen data is generated based on multimedia data and cloud application interface, and the multimedia data is included in the playback area screen data.

[0128] The rendering module 23 is used to decode the playback area screen data and render the decoded playback area screen data according to the screen layout information.

[0129] Optionally, the playback area screen data and non-playback area screen data in the first cloud application screen data are determined by the cloud server by comparing the first cloud application screen data with the second cloud application screen data generated at historical times. The screen data that is different from the screen data in the first cloud application screen data and the screen data in the second cloud application screen data generated at historical times is the playback area screen data, and the screen data that is the same is the non-playback area screen data.

[0130] Figure 9 and Figure 10 The device shown can perform the steps in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0131] In one possible design, the above Figure 9 The structure of the cloud application screen transmission device shown can be implemented as a cloud server. For example... Figure 11 As shown, the cloud server may include: a processor 31, a memory 32, and a communication interface 33. The memory 32 stores executable code, which, when executed by the processor 31, enables the processor 31 to at least implement the cloud application screen transmission method provided in the foregoing embodiments.

[0132] In one possible design, the above Figure 10 The structure of the cloud application screen transmission device shown can be implemented as a terminal device. For example... Figure 12 As shown, the terminal device may include: a processor 41, a memory 42, and a communication interface 43. The memory 42 stores executable code, which, when executed by the processor 41, enables the processor 41 to at least implement the cloud application screen transmission method provided in the foregoing embodiments.

[0133] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the cloud application screen transmission method provided in the foregoing embodiments.

[0134] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate. Part or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0135] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of the necessary general hardware platform, and of course can also be realized by means of the combination of hardware and software. Based on such understanding, the above technical solutions can be embodied in the form of computer products, and the present application can be embodied in the form of computer program products implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0136] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for transmitting images in a cloud application, characterized in that, Applied to cloud servers, the method includes: Load the encoded multimedia data that needs to be displayed in the cloud application interface at the current moment; Based on the multimedia data and the cloud application interface, the first cloud application screen data at the current moment is generated; Compare the first cloud application screen data with the second cloud application screen data generated at a historical moment to determine the screen data that is different from the first cloud application screen data and the second cloud application screen data as the playback area screen data, and the screen data that is the same as the non-playback area screen data. The playback area screen data and the screen layout information corresponding to the playback area screen data in the first cloud application screen data are respectively sent to the client corresponding to the cloud application, so that the client decodes the playback area screen data and renders the decoded playback area screen data according to the screen layout information.

2. The method according to claim 1, characterized in that, The step of comparing the first cloud application screen data with the second cloud application screen data generated at a historical time to determine the screen data in the first cloud application screen data that differs from the screen data in the second cloud application screen data as the playback area screen data, and the screen data that is the same as the non-playback area screen data, includes: Identify the target interface components contained in the cloud application interface, wherein the target interface components are interface components of a preset type whose displayed data can change dynamically; Compare whether the screen data to be displayed on the target interface component in the first cloud application screen data and the second cloud application screen data are consistent; If there is a discrepancy, then it is determined that the screen data to be displayed on the target interface component in the first cloud application screen data is included in the playback area screen data.

3. The method according to claim 1, characterized in that, The method further includes: If the non-playback area screen data in the second cloud application screen data is different from the non-playback area screen data in the first cloud application screen data, then the non-playback area screen data in the first cloud application screen data is sent to the client so that the client can render the non-playback area screen data in the first cloud application screen data.

4. The method according to claim 1, characterized in that, The encoded multimedia data is obtained by encoding the multimedia data using a first encoding method from the corresponding data source.

5. The method according to claim 4, characterized in that, The step of generating the first cloud application screen data at the current moment based on the multimedia data and the cloud application interface includes: If it is detected that the user has triggered interactive behavior data in the cloud application interface at the current moment, then the first cloud application screen data at the current moment is generated based on the multimedia data, the interactive behavior data and the cloud application interface. Before sending the playback area screen data to the client corresponding to the cloud application, the method further includes: If it is determined that the interactive behavior data is contained in the playback area screen data, then the interactive behavior data is encoded using the first encoding method.

6. The method according to claim 3, characterized in that, Sending the non-playback area screen data from the first cloud application screen data to the client includes: The non-playing area screen data in the first cloud application screen data is encoded using a second encoding method, and the encoded non-playing area screen data is sent to the client.

7. A method for transmitting images in a cloud application, characterized in that, Applied to clients corresponding to cloud applications, the method includes: Send a data playback request to the cloud server corresponding to the cloud application, so that the cloud server loads the encoded multimedia data that needs to be displayed in the cloud application interface at the current moment based on the data playback request. The system receives playback area screen data and corresponding screen layout information from the first cloud application screen data sent by the cloud server; wherein, the first cloud application screen data is generated based on the multimedia data and the cloud application interface, and the playback area screen data and non-playback area screen data in the first cloud application screen data are determined by the cloud server by comparing the first cloud application screen data with the second cloud application screen data generated at a historical time. The screen data that is different from the second cloud application screen data is the playback area screen data, and the screen data that is the same is the non-playback area screen data. Decode the playback area image data and render the decoded playback area image data according to the image layout information.

8. A cloud server, characterized in that, include: The system includes a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the cloud application screen transmission method as described in any one of claims 1 to 6.

9. A terminal device, characterized in that, include: The system includes a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the cloud application screen transmission method as described in claim 7.

10. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code, which, when executed by a processor of an electronic device, causes the processor to perform the cloud application screen transmission method as described in any one of claims 1 to 6, or claim 7.

Images