A cloud game data decoding method, device, electronic device and storage medium

Abstract

This invention provides a cloud gaming data decoding method, apparatus, electronic device, and storage medium, relating to the field of computer technology. The method includes: upon startup of a cloud game, acquiring historical decoding capability data of a terminal device, and then determining whether the terminal device meets hardware decoding requirements based on the historical decoding capability data. If the terminal device meets the hardware decoding requirements, hardware decoding is used to decode the video data of the cloud game. Subsequently, during the hardware decoding process, a determination is made based on the real-time decoding capability data of the terminal device to determine whether to switch to software decoding. If so, software decoding is used to decode the video data of the cloud game. This avoids game stuttering.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a cloud gaming data decoding method, apparatus, electronic device, and storage medium. Background Art

[0002] Cloud gaming is an online gaming technology based on cloud technology. Its advantages include low latency, smooth operation, and no need for downloading or installation. The server transmits the game footage as video data to the player's terminal device, which then decodes the received video data to display the game.

[0003] Currently, there is a variety of terminal devices that run cloud gaming. Each terminal device has different decoding capabilities for video data. As the duration of the game increases, the decoding speed of some terminal devices becomes slower and slower, which may result in game screen stuttering. Summary of the Invention

[0004] The purpose of this invention is to provide a cloud gaming data decoding method, apparatus, electronic device, and storage medium to solve the problem of game screen stuttering when terminal devices run cloud games. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a cloud gaming data decoding method, including:

[0006] When cloud gaming starts, acquire historical decoding capability data of the terminal device;

[0007] Based on the historical decoding capability data, determine whether the terminal device meets the hardware decoding requirements;

[0008] If the terminal device meets the hardware decoding requirements, the video data of the cloud game is decoded using hardware decoding.

[0009] During the process of decoding the video data of the cloud game using hardware decoding, it is determined whether to switch to software decoding based on the real-time decoding capability data of the terminal device;

[0010] If so, then switch to software decoding to decode the video data of the cloud game.

[0011] Secondly, embodiments of this application provide a cloud gaming data decoding device, comprising:

[0012] The acquisition module is used to acquire historical decoding capability data of the terminal device when cloud gaming starts.

[0013] The judgment module is used to determine whether the terminal device meets the hardware decoding requirements based on the historical decoding capability data obtained by the acquisition module.

[0014] The decoding module is used to decode the video data of the cloud game using hardware decoding when the judgment module determines that the terminal device meets the hardware decoding requirements;

[0015] The judgment module is further configured to determine whether to switch to software decoding mode based on the real-time decoding capability data of the terminal device during the process of the decoding module decoding the video data of the cloud game using hardware decoding mode.

[0016] The switching module is used to switch to software decoding mode to decode the video data of the cloud game when the judgment module determines that the software decoding mode has been switched.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0018] Memory, used to store computer programs;

[0019] When a processor executes a program stored in memory, it implements any of the cloud gaming data decoding method steps described above.

[0020] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the cloud gaming data decoding method described in the first aspect.

[0021] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to execute the cloud gaming data decoding method described in the first aspect above.

[0022] Beneficial effects of the embodiments of the present invention:

[0023] The cloud gaming data decoding method, apparatus, electronic device, and storage medium provided in this invention can acquire historical decoding capability data of the terminal device when the cloud game starts. Since hardware decoding speed is usually faster than software decoding speed, if the terminal device meets the hardware decoding requirements based on the historical decoding capability data, hardware decoding is used to decode the video data of the cloud game to ensure smooth playback of the game screen. During the operation of the cloud game, the terminal device continuously performs hardware decoding operations, which may cause the hardware decoding speed to decrease. Therefore, during the hardware decoding process, it can be determined whether to switch to software decoding mode based on the real-time decoding capability data of the terminal device, and the decoding mode can be switched in a timely manner according to the determination result to avoid stuttering due to insufficient real-time decoding capability during hardware decoding, thereby avoiding stuttering of the cloud game screen. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0025] Figure 1 A schematic diagram of the structure of a cloud system provided in an embodiment of the present invention;

[0026] Figure 2 A flowchart of a cloud gaming data decoding method provided in an embodiment of the present invention;

[0027] Figure 3 A flowchart of another cloud gaming data decoding method provided in an embodiment of the present invention;

[0028] Figure 4 A flowchart of another cloud gaming data decoding method provided in an embodiment of the present invention;

[0029] Figure 5 A flowchart illustrating a cloud gaming data decoding method provided in an embodiment of the present invention;

[0030] Figure 6 This is a schematic diagram of the structure of a cloud gaming data decoding device provided in an embodiment of the present invention;

[0031] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed Implementation

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0033] In cloud gaming scenarios, such as Figure 1 As shown, the cloud system used to run cloud gaming may include: terminal device 101 and cloud device 102.

[0034] The cloud device 102 can be a virtual device or a physical device. For example, the cloud device 102 can be a logical device virtualized on a cloud server using virtualization technology. Specifically, multiple logical devices can be virtualized on any single cloud server. When the cloud device is a physical device, it includes, but is not limited to, smartphones, tablets, laptops, desktop computers, servers, etc.

[0035] In practice, cloud device 102 can provide corresponding network services by installing cloud applications (also known as cloud applications, cloud apps, cloud-based applications, etc.). For example, cloud device 102 can provide cloud gaming services by installing cloud gaming applications; that is, cloud applications run on cloud device 102. One or more cloud applications can run on any cloud device 102. When multiple cloud applications are installed, their application identifiers are generally different. In one possible scenario, a single cloud device can have multiple cloud applications with different application identifiers installed.

[0036] Terminal device 101 is a physical device, which can be various electronic devices with a display screen, including but not limited to smartphones, tablets, laptops, desktop computers, etc. Terminal device 101 can provide corresponding network services by installing client applications, such as providing video or image playback services by installing a video player.

[0037] Cloud device 102 and terminal device 101 establish a communication connection through a cloud communication module. Optionally, the cloud communication module can be built into cloud device 102 or independent of cloud device 102, and can be used to realize the communication connection between one or more cloud applications on cloud device 102 and their corresponding terminal devices 101, and / or to realize the communication connection between one or more cloud devices 102 and their corresponding terminal devices 101. The module mentioned above is broad and can be a class, function, thread, process, etc. For example, the cloud communication module can be a cloud application service program.

[0038] exist Figure 1In the illustrated cloud system, on one hand, cloud device 102 runs cloud applications and sends data streams of the running cloud applications to terminal device 101, which then displays the received data streams. On the other hand, terminal device 101 collects user operation data and sends this data (collected or processed) to cloud device 102. Cloud device 102 responds to user operations based on the received operation data and continues to send response data streams to terminal device 101, which then displays the received response data streams. Thus, cloud device 102 and terminal device 101 form a closed loop, allowing users to access cloud applications installed on cloud device 102 through terminal device 101. Since the cloud applications do not actually run on terminal device 101, this reduces the operating resources required by terminal device 101.

[0039] Taking cloud gaming applications as an example, cloud device 102 sends video data streams of the cloud gaming application to terminal device 101. Terminal device 101 can then display the received video stream through an installed application, such as a video player. Furthermore, each time terminal device 101 collects user operation data, it sends operation data to cloud device 102. Cloud device 102 then responds based on the received operation data, refreshes the video stream, and sends the refreshed video data back to terminal device 101. In this way, users can use the cloud gaming application installed on cloud device 102 through terminal device 101.

[0040] It is understood that the number of devices shown in the diagram is for illustrative purposes only. Depending on actual needs, a cloud system can include any number of networks and devices, and this embodiment of the invention does not impose any limitations on this.

[0041] To ensure a good user experience during cloud gaming, the smoothness of cloud gaming operations needs to be roughly the same as that of the game client installed on the terminal device. However, in practical applications, due to the diversity of terminal device models running cloud games, and the varying decoding capabilities of different devices for video data, some terminal devices experience slower decoding speeds as the game runs longer, potentially leading to game stuttering and a degraded user experience. This application provides a cloud gaming data decoding method to reduce stuttering in cloud games, improve their smoothness, and ultimately enhance user satisfaction.

[0042] based on Figure 1 The cloud gaming system shown in this application provides a cloud gaming data decoding method, which is applied to a terminal device, such as... Figure 2 As shown, the method includes:

[0043] S201. When cloud gaming starts, obtain the historical decoding capability data of the terminal device.

[0044] S202. Determine whether the terminal device meets the hardware decoding requirements based on historical decoding capability data. If the terminal device meets the hardware decoding requirements, proceed to S203.

[0045] S203. Hardware decoding is used to decode the video data of cloud gaming.

[0046] S204. During the decoding of video data for cloud gaming using hardware decoding, determine whether to switch to software decoding based on the real-time decoding capability data of the terminal device. If yes, proceed to S205.

[0047] S205, Switch to software decoding to decode video data for cloud gaming.

[0048] The cloud gaming data decoding method provided in this invention can acquire historical decoding capability data of the terminal device when the cloud game starts. Since hardware decoding speed is usually faster than software decoding speed, if the terminal device meets the hardware decoding requirements based on the historical decoding capability data, hardware decoding is used to decode the video data of the cloud game to ensure smooth playback of the game screen. During the operation of the cloud game, the terminal device continuously performs hardware decoding operations, which may cause the hardware decoding speed to decrease. Therefore, during the hardware decoding process, it can be determined whether to switch to software decoding mode based on the real-time decoding capability data of the terminal device, and the decoding mode can be switched in a timely manner according to the determination result to avoid stuttering due to insufficient real-time decoding capability during hardware decoding, thus avoiding stuttering of the cloud game screen.

[0049] In S201 above, the historical decoding capability data of the terminal device is data used to reflect the decoding capability of the terminal device to perform hardware decoding of video data.

[0050] The historical decoding capability data may include, but is not limited to, at least one of the following: historical decoding capability data corresponding to the model of the terminal device, historical decoding capability data corresponding to the device manufacturer to which the terminal device belongs, and historical decoding capability data of the terminal devices used by the account that logged into the cloud gaming.

[0051] In S202 above, if the historical decoding capability data of the terminal device meets the hardware decoding requirements, it means that the historical decoding capability of the terminal device can meet the requirements for decoding the video data of cloud games. Therefore, the hardware decoding method can be used to decode the video data of cloud games.

[0052] Conversely, if the historical decoding capability data of the terminal device does not meet the hardware decoding requirements, it means that the historical decoding capability of the terminal device cannot meet the requirements for decoding video data of cloud games. The current hardware decoding method is prone to causing game lag. Therefore, in this case, software decoding can be used to decode the video data of cloud games.

[0053] In S203 above, the terminal device can transmit cloud game video data to the hardware decoder based on the real-time decoding speed of the terminal device, so that the hardware decoder can decode the cloud game video data.

[0054] In S204 above, during the process of running cloud games on the terminal device, the real-time decoding capability data of the terminal device may change. For example, if the terminal device heats up due to running cloud games for too long, the real-time decoding capability of the terminal device will also decrease accordingly. If the real-time decoding capability data is too poor, it will cause playback stuttering and affect the user experience. Therefore, in this embodiment of the application, the terminal device can determine whether to switch to software decoding mode based on the real-time decoding capability data to avoid long-term stuttering during cloud game operation.

[0055] In one embodiment of this application, for the above-described S202, the following two implementation methods are provided:

[0056] In the first possible implementation, the historical decoding capability data may include the historical decoding speed. Correspondingly, the hardware decoding requirement in S202 above could be that the historical decoding speed reaches a second decoding speed threshold, which can be set according to actual needs. S202 can be implemented as follows: determining whether the historical decoding speed is greater than or equal to the second decoding speed threshold. If the historical decoding speed is greater than or equal to the second decoding speed threshold, then the terminal device is determined to meet the hardware decoding requirement; if the historical decoding speed is less than the second decoding speed threshold, then the terminal device is determined not to meet the hardware decoding requirement.

[0057] The historical decoding speed can be the maximum, minimum, or average hardware decoding speed during the historical running of cloud games, and can be set based on actual needs.

[0058] In the second possible implementation, the historical decoding capability data may include historical decoding speed and decoding stability index. Correspondingly, the hardware decoding requirement in S202 above could be that the historical decoding speed reaches a second decoding speed threshold, and the decoding stability index reaches a decoding stability threshold. S202 can be implemented as follows: determining whether the historical decoding speed is greater than or equal to the second decoding speed threshold, and whether the decoding stability index is greater than or equal to the decoding stability threshold. If the historical decoding speed is greater than or equal to the second decoding speed threshold, and the decoding stability index is greater than or equal to the decoding stability threshold, then the terminal device is determined to meet the hardware decoding requirements; otherwise, the terminal device is determined not to meet the hardware decoding requirements.

[0059] The decoding stability index can be obtained based on the average historical decoding speed, or it can be a fluctuation parameter of the historical decoding speed curve. The fluctuation parameter characterizes the fluctuation of the curve; the smaller the fluctuation parameter, the better the decoding stability index. For example, the fluctuation parameter includes, but is not limited to, at least one of the following: amplitude, slope of the rising process, slope of the falling process, and variance. This decoding stability index can be obtained through statistical analysis of historical decoding data.

[0060] In another embodiment of this application, during the decoding of video data for cloud gaming using hardware decoding, the hardware decoding capability of the terminal device may fluctuate, resulting in insufficient hardware decoding capability and causing the cloud gaming to lag. Therefore, it is necessary to determine whether to switch to software decoding based on the real-time decoding capability data of the terminal device.

[0061] Real-time decoding capability data is used to reflect the real-time decoding status of the terminal device, including but not limited to at least one of the following: the real-time decoding speed of the terminal device, the amount of data to be decoded cached by the terminal device, etc., which will not be listed exhaustively here.

[0062] In one possible implementation, where the real-time decoding capability data includes, but is not limited to, the real-time decoding speed of the terminal device, the above S204 can be implemented as follows: determining whether the real-time decoding speed is less than a first decoding speed threshold. If the real-time decoding speed is less than the first decoding speed threshold, it is determined that it is necessary to switch to software decoding mode; otherwise, it is determined that it is not necessary to switch to software decoding mode.

[0063] The real-time decoding speed may include, but is not limited to, the average decoding speed over a specified period of time. For example, the specified period of time may be 5 seconds, and the terminal device may calculate the average decoding speed every 5 seconds.

[0064] For example, in the first second, 21 frames are decoded, with 500 milliseconds used for decoding within that first second, leaving 500 milliseconds undecoded. In the second second, 20 frames are decoded, with 510 milliseconds used for decoding within that second, leaving 490 milliseconds undecoded. In the third second, 22 frames are decoded, with all 1000 milliseconds of that third second used for decoding. In the fourth second, 21 frames are decoded, with 900 milliseconds used for decoding within that fourth second, leaving 100 milliseconds undecoded. In the fifth second, 23 frames are decoded, with 590 milliseconds used for decoding within that fifth second, leaving 410 milliseconds undecoded. In total, 107 frames are decoded over five seconds, with a total of 3500 milliseconds used for decoding. The average speed is 1 frame / 32.7 milliseconds, resulting in an average decoding speed of 1 frame / 32.7 milliseconds over these five seconds.

[0065] When the real-time decoding speed falls below the first decoding speed threshold, it indicates that the real-time decoding speed is insufficient for smooth cloud gaming operation. Continuing to use a hardware decoder to decode the cloud gaming video data will cause stuttering and negatively impact the user experience. Therefore, it is advisable to switch to software decoding for cloud gaming video data at this point.

[0066] In another possible implementation, where the real-time decoding capability data includes, but is not limited to, the amount of data to be decoded cached by the terminal device, the above S204 can be implemented as follows: determine whether the amount of data to be decoded cached is greater than or equal to a preset quantity threshold. If the amount of data to be decoded cached is greater than or equal to the preset quantity threshold, then it is determined that it is necessary to switch to software decoding mode; otherwise, it is determined that it is not necessary to switch to software decoding mode.

[0067] Understandably, if the real-time decoding speed slows down, the amount of data to be decoded cached in the terminal device will increase accordingly. If the amount of data to be decoded cached is too large, playback may be interrupted. Therefore, the embodiments of this application can switch to software decoding in a timely manner to avoid long-term lag in cloud gaming.

[0068] In another possible implementation, where the real-time decoding capability data includes, but is not limited to, the real-time decoding speed of the terminal device and the amount of data to be decoded in the buffer, the above S204 can be implemented as follows: determining whether the real-time decoding speed is less than a first decoding speed threshold and whether the amount of data to be decoded in the buffer is greater than or equal to a preset quantity threshold. If the real-time decoding speed is less than the first preset decoding speed threshold and the amount of data to be decoded in the buffer is greater than or equal to the preset quantity threshold, then it is determined that it is necessary to switch to software decoding mode; otherwise, it is determined that it is not necessary to switch to software decoding mode.

[0069] Using a combination of real-time decoding speed and the amount of data to be decoded in the buffer to determine whether to switch to software decoding mode can make the judgment more accurate and avoid the problem of accidental switching.

[0070] In one embodiment of this application, since some terminal devices do not support hardware decoding, it can be determined in advance whether the terminal device supports hardware decoding before obtaining the historical decoding capability data of the terminal device, such as... Figure 3 As shown, the method may include the following steps:

[0071] S206. Determine whether the terminal device supports hardware decoding of video data for cloud gaming.

[0072] If yes, that is, if the terminal device supports hardware decoding of video data for cloud gaming, proceed to step S201. If no, that is, if the terminal device does not support hardware decoding of video data for cloud gaming, proceed to step S207.

[0073] Specifically, this step can be implemented as follows: When cloud gaming starts, the terminal device negotiates the encoding format of the video data with the cloud device. Then, based on the negotiated encoding format, the terminal device attempts to obtain hardware decoder information for hardware decoding of the video data in that format. If the hardware decoder information cannot be obtained, it is determined that the terminal device does not support hardware decoding of the cloud gaming video data; if the hardware decoder information is obtained, it is determined that the terminal device supports hardware decoding of the cloud gaming video data.

[0074] When cloud gaming starts, the terminal device sends a list of supported encoding formats to the cloud device. This list may include formats such as H.261, H.262, H.263, H.263+, and H.264. Upon receiving this list, the cloud device selects a decoding format and sends it back to the terminal device as the agreed-upon encoding format. The specific method for negotiating the encoding format of video data between the terminal device and the cloud device can be found in related technologies; this embodiment does not limit this approach.

[0075] For example, the agreed-upon encoding format could be H.264, meaning that the cloud device will subsequently send video data encoded in H.264 format to the terminal device. The terminal device can call a specified Application Programming Interface (API) to attempt to obtain the hardware decoder used to decode the H.264 format video data. If the information obtained from calling the specified API is empty, it can be determined that the terminal device does not support hardware decoding of cloud gaming data.

[0076] S207. Software decoding is used to decode the video data of cloud gaming.

[0077] Using the above method, when a cloud game starts, the hardware decoding resources of the terminal device are first adapted to determine whether the terminal device supports hardware decoding of the cloud game's video data. If hardware decoding is not supported, software decoding can be used directly to decode the cloud game's video data without further determining the terminal device's hardware decoding capabilities, thus speeding up the cloud game's startup.

[0078] In addition, if Figure 3 As shown, after S202 determines whether the terminal device meets the hardware decoding requirements based on historical decoding capability data, if the terminal device does not meet the hardware decoding requirements, it means that the hardware decoding capability of the terminal device is insufficient to support smooth decoding and playback of video data. Therefore, S207 can be executed to decode the video data of cloud gaming using software decoding.

[0079] In another embodiment of this application, such as Figure 4 As shown in S203 above, when the terminal device meets the hardware decoding requirements, the video data of cloud gaming is decoded using hardware decoding, including the following steps:

[0080] S2031. If the terminal device meets the hardware decoding requirements, start the hardware decoder and software decoder.

[0081] The software decoder can serve as a backup decoder. Optionally, the software decoder can be started simultaneously with the hardware decoder. Alternatively, in another embodiment of this application, the hardware decoder can be started first, and then the software decoder can be started after the hardware decoder is working normally, in order to avoid decoding delay caused by the delay in starting the software decoder, which is beneficial to improving decoding response speed.

[0082] S2032. Obtain video data from cloud gaming and decode the video data from cloud gaming using a hardware decoder.

[0083] The terminal device can receive audio and video data sent by the cloud device. By default, it decodes the audio data using hardware decoding, and if the terminal device meets the hardware decoding requirements, it decodes the video data using a hardware decoder. Since the amount of audio data is much smaller than that of video data, the hardware decoding capability of the terminal device is generally sufficient to meet the decoding requirements for audio data. Therefore, the audio data can be decoded directly using hardware decoding.

[0084] Based on the already activated hardware decoder and software decoder, such as Figure 4As shown, the above-mentioned S205, switching to software decoding to decode the video data of cloud gaming, can be achieved as follows:

[0085] S2051. Disable the hardware decoder and use the software decoder to decode the video data of cloud gaming.

[0086] After disabling the hardware decoder, the terminal device can re-request video data from the cloud device. The cloud device will then send video data to the terminal device starting from the first undecoded keyframe. The terminal device will then use software decoding to decode the subsequently received video data, completing the switching of decoding methods. In this embodiment, the decoding pause time caused by switching from hardware to software decoding is in the millisecond range, thus achieving a seamless switching experience for the user.

[0087] The terminal device can record the maximum, minimum, and average hardware decoding speeds before the switch and write them to a local configuration file. It also sends these speeds to the cloud device to provide reference data for updating the historical decoding capability data of the terminal device in the cloud configuration file.

[0088] Using this method, after determining that hardware decoding is to be used, both the hardware decoder and the software decoder are started simultaneously. In this way, when it is necessary to switch decoding methods, it is not necessary to start the software decoder and then call it. Instead, the software decoder can be directly called to decode the video data of cloud gaming, thus ensuring the switching speed and enabling cloud gaming to run smoothly, avoiding the stuttering caused by switching decoding methods.

[0089] In this embodiment, the local configuration file of the terminal device stores the historical decoding capability data of the terminal device, and the cloud configuration file can store the historical decoding capability data of various models of terminal devices, the historical decoding capability data corresponding to each device manufacturer, and / or the historical decoding capability data of each account logged into the cloud game. As an example, the cloud configuration file stores the historical decoding capability data of terminal devices of models A and B of device manufacturer 1, and the historical decoding capability data of terminal devices of models C and D of device manufacturer 2.

[0090] Accordingly, there are two ways in which the above-mentioned S201 obtains the historical decoding capability data of the terminal device:

[0091] Method 1: Obtain a local configuration file and retrieve the historical decoding capability data of the terminal device from the local configuration file. Alternatively, download a cloud configuration file and retrieve the historical decoding capability data of the terminal device from the cloud configuration file.

[0092] In this embodiment, a local configuration file can be obtained from the local Secure Digital (SD) card of the terminal device. The local configuration file includes historical decoding capability data recorded by the terminal device itself. For example, the local configuration file includes, but is not limited to, at least one of the following: the average hardware decoding speed, the maximum hardware decoding speed, and the minimum hardware decoding speed for each game run by the terminal. Specifically, each time the terminal device runs a cloud game, it saves the decoding capability data collected during the cloud game run in the local configuration file.

[0093] The historical decoding capability data included in the cloud configuration file is obtained by statistically analyzing the historical decoding capability data of multiple terminal devices of the same model, manufacturer, or series.

[0094] For example, historical decoding capability data includes historical decoding capability data for terminal devices from manufacturers A, B, C, D, and E. Taking manufacturer A as an example, the historical decoding capability data for manufacturer A can reflect the overall decoding capability level of various terminal devices from manufacturer A.

[0095] Terminal devices can comprehensively determine whether they meet hardware decoding requirements based on historical decoding capability data in local configuration files and cloud configuration files.

[0096] The hardware decoding requirement can be set according to actual needs. For example, the hardware decoding requirement can be that the average hardware decoding speed of the historical decoding capability data in the local configuration file is greater than the hardware decoding speed threshold, and the average hardware decoding speed of the terminal device in the cloud configuration file is also greater than the hardware decoding speed threshold.

[0097] As an example, the hardware decoding requirement could be that the average of 10 hardware decoding speeds in the local configuration file is greater than 1 frame / 33 milliseconds, and the hardware decoding speed saved in the cloud configuration file is greater than 1 frame / 33 milliseconds. Alternatively, the hardware decoding requirement could also be that the minimum of 10 hardware decoding speeds in the local configuration file is greater than 1 frame / 33 milliseconds, and the hardware decoding speed saved in the cloud configuration file is greater than 1 frame / 33 milliseconds. Or, the hardware decoding requirement could also be that the maximum of 10 hardware decoding speeds in the local configuration file is greater than 1 frame / 20 milliseconds, and the hardware decoding speed saved in the cloud configuration file is greater than 1 frame / 33 milliseconds.

[0098] By adopting the embodiments of this application, by comprehensively considering the historical decoding capability data recorded in the local configuration file and the historical decoding capability data of the terminal device obtained from the cloud configuration file, since the historical decoding capability data in the local configuration file represents the decoding capability of the terminal device itself, and the historical decoding capability data in the cloud configuration file represents the comprehensive decoding capability of terminal devices of the same type as the terminal device, the hardware decoding capability of the terminal device can be measured more accurately.

[0099] Method 2: Obtain the local configuration file and retrieve the historical decoding capability data of the terminal device from the local configuration file. If the local configuration file cannot be obtained, or the amount of historical decoding capability data of the terminal device obtained from the local configuration file does not reach the specified amount, obtain the cloud configuration file and retrieve the historical decoding capability data of the terminal device from the cloud configuration file.

[0100] In this embodiment of the application, if the terminal device has not run cloud games, the terminal device does not record historical decoding capability data, that is, it cannot obtain the local configuration file. Therefore, the cloud configuration file can be downloaded and the historical decoding capability data of the terminal device can be obtained from the cloud configuration file.

[0101] Meanwhile, if the local configuration file of the terminal device can be obtained, but the local configuration file of the terminal device contains too little historical decoding capability data, in order to more accurately measure the hardware decoding capability of the terminal device, the cloud configuration file can be downloaded to obtain the historical decoding capability data of the terminal device from the cloud configuration file.

[0102] This method first obtains the local configuration file of the terminal device. If the local configuration file is available, the historical decoding capability data included in it can be used to determine whether the terminal device meets the hardware decoding requirements. If the terminal device meets the hardware decoding requirements, hardware decoding is then used to decode the video data for cloud gaming. This process eliminates the need for the terminal device to download a cloud configuration file, avoiding the latency caused by cloud file retrieval and accelerating cloud gaming response times. Furthermore, if the local configuration file is unavailable, downloading the cloud configuration file ensures access to the terminal device's historical decoding capability data, providing a basis for determining the decoding method.

[0103] In one embodiment of this application, the terminal device may also record real-time decoding capability data and update historical decoding capability data based on the real-time decoding capability data.

[0104] During cloud gaming, terminal devices can save real-time decoding capability data in a local configuration file and upload it to the cloud device, so that maintenance personnel can flexibly adjust the above hardware decoding requirements based on the historical decoding capability data of the terminal devices.

[0105] The following describes the cloud gaming data decoding method provided in the embodiments of this application with reference to specific processes, such as... Figure 5 As shown, taking running a cloud game in a video app as an example, after the cloud game displayed in the video app is triggered by the user, the signaling module of the video app interacts with the cloud phone (which runs in a cloud device) through a Selective Forwarding Unit (SFU) to start the cloud game. After the cloud game starts, the video app obtains a cloud configuration file and a local configuration file, and adapts the obtained cloud configuration file with the terminal device (such as a mobile phone) where the video app is located, obtaining historical decoding capability data adapted to the terminal device where the video app is located, which is included in the cloud configuration file. Then, based on the historical decoding capability data obtained from the local configuration file and the cloud configuration file, it can determine whether to use hardware decoding to decode the video data.

[0106] If hardware decoding is selected to decode hardware data, the video app will activate both the hardware decoder and the software decoder. If hardware decoding is not selected, the software decoder will be activated.

[0107] The video app interacts with the signaling modules of the SFU and cloud phone to obtain audio and video data of cloud gaming from the cloud phone, transmits audio data to the audio module, and transmits video data to the video module.

[0108] In this embodiment, a hardware decoder is used by default to decode the audio data, and the decoded audio is played through a speaker.

[0109] In the case where hardware decoding is used as described above, the video module inputs the video data to be decoded into the hardware decoder for hardware decoding, and renders the decoded video data into game screens for display.

[0110] During the process of decoding video data using a hardware decoder, real-time decoding capability data (including the real-time decoding speed of the terminal device and the amount of data to be decoded in the buffer) is monitored in real time. If it is determined that the real-time decoding performance represented by the real-time decoding capability data has decreased, the software decoding method is switched to be used for decoding. The video module inputs the video data to be decoded into the software decoder for software decoding, and renders the decoded video data into game screens and displays them.

[0111] When software decoding is used as described above, the video module inputs the video data to be decoded into the software decoder for software decoding, and renders the decoded video data into game screens for display.

[0112] By using the cloud gaming data decoding method provided in this embodiment of the invention, after the cloud game starts, and provided the terminal device meets the hardware decoding requirements, both a hardware decoder and a software decoder are simultaneously activated. The hardware decoder is used to decode the video data of the cloud game, thereby playing the video data. During the playback of the video data, the system determines whether to switch to software decoding mode based on the real-time decoding speed of the terminal device and the amount of data to be decoded in the buffer. This achieves dynamic switching of the decoding mode based on the real-time decoding performance of the terminal device, reducing lag during cloud game operation and greatly enhancing the user's gaming experience.

[0113] By monitoring the real-time decoding speed of the mobile phone, the system switches from hardware decoding to software decoding when the real-time decoding speed drops, ensuring the smooth operation of cloud gaming.

[0114] Based on the same inventive concept, corresponding to the above method embodiments, this invention also provides a cloud gaming data decoding device, see [link to relevant documentation]. Figure 6 The device includes: an acquisition module 601, a judgment module 602, a decoding module 603, and a switching module 604;

[0115] The acquisition module 601 is used to acquire historical decoding capability data of the terminal device when the cloud game starts.

[0116] The judgment module 602 is used to determine whether the terminal device meets the hardware decoding requirements based on the historical decoding capability data obtained by the acquisition module 601.

[0117] The decoding module 603 is used to decode the video data of cloud gaming using hardware decoding when the judgment module 602 determines that the terminal device meets the hardware decoding requirements.

[0118] The judgment module 602 is also used to determine whether to switch to software decoding mode based on the real-time decoding capability data of the terminal device during the decoding process of the video data of cloud games using hardware decoding mode in the decoding module 603.

[0119] The switching module 604 is used to switch to software decoding mode to decode video data of cloud games when the judgment module 602 determines that the software decoding mode has been switched.

[0120] The cloud gaming data decoding device provided in this embodiment of the invention can acquire historical decoding capability data of the terminal device when the cloud game starts. Since hardware decoding speed is usually faster than software decoding speed, if the terminal device meets the hardware decoding requirements based on the historical decoding capability data, hardware decoding is used to decode the video data of the cloud game to ensure smooth playback of the game screen. During the operation of the cloud game, the terminal device continuously performs hardware decoding operations, which may cause the hardware decoding speed to decrease. Therefore, during the hardware decoding process, it can determine whether to switch to software decoding mode based on the real-time decoding capability data of the terminal device, and switch the decoding mode in a timely manner according to the determination result to avoid stuttering due to insufficient real-time decoding capability during hardware decoding, thus preventing stuttering of the cloud game screen.

[0121] Optionally, the real-time decoding capability data includes the real-time decoding speed of the terminal device; the judgment module 602 is specifically used for:

[0122] Determine if the real-time decoding speed is less than the first decoding speed threshold;

[0123] If the real-time decoding speed is less than the first decoding speed threshold, it is determined that it is necessary to switch to software decoding mode; otherwise, it is determined that it is not necessary to switch to software decoding mode.

[0124] Optionally, the real-time decoding capability data includes the amount of data to be decoded buffered by the terminal device; the judgment module 602 is specifically used for:

[0125] Determine whether the amount of buffered data to be decoded is greater than or equal to a preset threshold.

[0126] If the amount of data to be decoded in the buffer is greater than or equal to a preset threshold, then it is determined that it is necessary to switch to software decoding mode; otherwise, it is determined that it is not necessary to switch to software decoding mode.

[0127] Optionally, the real-time decoding capability data includes the real-time decoding speed of the terminal device and the amount of data to be decoded buffered; the judgment module 602 is specifically used for:

[0128] Determine whether the real-time decoding speed is less than the first decoding speed threshold, and whether the amount of data to be decoded in the buffer is greater than or equal to a preset quantity threshold;

[0129] If the real-time decoding speed is less than the first preset decoding speed threshold and the amount of data to be decoded in the buffer is greater than or equal to the preset quantity threshold, then it is determined that it is necessary to switch to software decoding mode; otherwise, it is determined that it is not necessary to switch to software decoding mode.

[0130] Optionally, the decoding module 603 is specifically used for:

[0131] If the terminal device meets the hardware decoding requirements, activate both the hardware decoder and the software decoder.

[0132] Acquire video data from cloud gaming and decode it using a hardware decoder;

[0133] Switching module 604 is specifically used for:

[0134] The hardware decoder is turned off, and the software decoder is used to decode the video data of cloud gaming.

[0135] Optionally, historical decoding capability data includes historical decoding speed; the judgment module 602 is specifically used for:

[0136] Determine whether the historical decoding speed is greater than or equal to the second decoding speed threshold;

[0137] If the historical decoding speed is greater than or equal to the second decoding speed threshold, then the terminal device is determined to meet the hardware decoding requirements.

[0138] If the historical decoding speed is less than the second decoding speed threshold, then the terminal device is determined to not meet the hardware decoding requirements.

[0139] Optionally, historical decoding capability data includes historical decoding speed and decoding stability indicators; the judgment module 602 is specifically used for:

[0140] Determine whether the historical decoding speed is greater than or equal to the second decoding speed threshold, and whether the decoding stability index is greater than or equal to the decoding stability threshold;

[0141] If the historical decoding speed is greater than or equal to the second decoding speed threshold, and the decoding stability index is greater than or equal to the decoding stability threshold, then the terminal device is determined to meet the hardware decoding requirements; otherwise, the terminal device is determined not to meet the hardware decoding requirements.

[0142] Optionally, the judgment module 602 is also used to determine whether the terminal device supports hardware decoding of video data of cloud games before obtaining the historical decoding capability data of the terminal device;

[0143] The judgment module 602 is specifically used to trigger the acquisition module 601 to perform the step of judging whether the terminal device meets the hardware decoding requirements based on historical decoding capability data when the terminal device supports hardware decoding of video data of cloud gaming.

[0144] The decoding module 603 is also used to decode the video data of cloud games using software decoding when the judgment module determines that the terminal device does not support hardware decoding of the video data of cloud games.

[0145] Optionally, the device further includes: a recording module and an updating module;

[0146] The recording module is used to record real-time decoding capability data;

[0147] The update module is used to update historical decoding capability data based on the real-time decoding capability data recorded by the recording module.

[0148] Optionally, the decoding module 603 is specifically used for:

[0149] The video data of cloud games is transmitted to the hardware decoder based on the real-time decoding speed of the terminal device, so that the hardware decoder can decode the video data of cloud games.

[0150] Optionally, the historical decoding capability data includes at least one of the following: historical decoding capability data corresponding to the model of the terminal device, historical decoding capability data corresponding to the device manufacturer to which the terminal device belongs, and historical decoding capability data of the terminal devices used by the account that logged into the cloud gaming.

[0151] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0152] Memory 703 is used to store computer programs;

[0153] When the processor 701 executes the program stored in the memory 703, it implements the steps of any of the cloud gaming data decoding methods described above.

[0154] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0155] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0156] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0157] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0158] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the cloud gaming data decoding methods described in the above embodiments.

[0159] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the cloud gaming data decoding methods described in the above embodiments.

[0160] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0161] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0162] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0163] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A cloud gaming data decoding method, characterized in that, Applied to terminal devices, including: When cloud gaming starts, historical decoding capability data of the terminal device is obtained. The historical decoding capability data is data that reflects the hardware decoding capability of the terminal device for video data. Based on the historical decoding capability data, determine whether the terminal device meets the hardware decoding requirements; If the terminal device meets the hardware decoding requirements, the video data of the cloud game is decoded using hardware decoding. During the process of decoding the video data of the cloud game using hardware decoding, it is determined whether to switch to software decoding based on the real-time decoding capability data of the terminal device; If so, then switch to software decoding to decode the video data of the cloud game; When the historical decoding capability data includes historical decoding speed, determining whether the terminal device meets the hardware decoding requirements based on the historical decoding capability data includes: Determine whether the historical decoding speed is greater than or equal to the second decoding speed threshold; If the historical decoding speed is greater than or equal to the second decoding speed threshold, then the terminal device is determined to meet the hardware decoding requirements. If the historical decoding speed is less than the second decoding speed threshold, then it is determined that the terminal device does not meet the hardware decoding requirements; Alternatively, if the historical decoding capability data includes historical decoding speed and decoding stability indicators, the step of determining whether the terminal device meets the hardware decoding requirements based on the historical decoding capability data includes: Determine whether the historical decoding speed is greater than or equal to the second decoding speed threshold, and whether the decoding stability index is greater than or equal to the decoding stability threshold; If the historical decoding speed is greater than or equal to the second decoding speed threshold, and the decoding stability index is greater than or equal to the decoding stability threshold, then the terminal device is determined to meet the hardware decoding requirements; otherwise, the terminal device is determined not to meet the hardware decoding requirements. The acquisition of historical decoding capability data of the terminal device includes: Obtain a local configuration file, and retrieve the historical decoding capability data of the terminal device from the local configuration file; Download the cloud configuration file and obtain the historical decoding capability data of the terminal device from the cloud configuration file; or, when the local configuration file cannot be obtained, or the amount of historical decoding capability data of the terminal device obtained from the local configuration file does not reach the specified amount of data, obtain the cloud configuration file and obtain the historical decoding capability data of the terminal device from the cloud configuration file; the cloud configuration file includes: historical decoding capability data obtained by statistically analyzing the historical decoding capability data of multiple terminal devices of the same model, the same device manufacturer, or the same series.

2. The method according to claim 1, characterized in that, The real-time decoding capability data includes the real-time decoding speed of the terminal device; The step of determining whether to switch to software decoding mode based on the real-time decoding capability data of the terminal device includes: Determine whether the real-time decoding speed is less than the first decoding speed threshold; If the real-time decoding speed is less than the first decoding speed threshold, it is determined that it is necessary to switch to software decoding mode; otherwise, it is determined that it is not necessary to switch to software decoding mode.

3. The method according to claim 1, characterized in that, The real-time decoding capability data includes the amount of data to be decoded cached by the terminal device; the step of determining whether to switch to software decoding mode based on the real-time decoding capability data of the terminal device includes: Determine whether the amount of data to be decoded in the buffer is greater than or equal to a preset threshold. If the amount of data to be decoded in the cache is greater than or equal to a preset threshold, then it is determined that it is necessary to switch to software decoding mode; otherwise, it is determined that it is not necessary to switch to software decoding mode.

4. The method according to claim 1, characterized in that, The real-time decoding capability data includes the real-time decoding speed of the terminal device and the amount of data to be decoded in the buffer; the step of determining whether to switch to software decoding mode based on the real-time decoding capability data of the terminal device includes: Determine whether the real-time decoding speed is less than a first decoding speed threshold, and whether the amount of data to be decoded in the buffer is greater than or equal to a preset quantity threshold; If the real-time decoding speed is less than a first preset decoding speed threshold and the amount of data to be decoded in the cache is greater than or equal to a preset quantity threshold, then it is determined that it is necessary to switch to software decoding mode; otherwise, it is determined that it is not necessary to switch to software decoding mode.

5. The method according to any one of claims 1-4, characterized in that, If the terminal device meets the hardware decoding requirements, the video data of the cloud game is decoded using hardware decoding, including: If the terminal device meets the hardware decoding requirements, the hardware decoder and software decoder are activated. The video data of the cloud game is acquired, and the video data of the cloud game is decoded by the hardware decoder; The switching to software decoding for decoding the cloud game's video data includes: The hardware decoder is turned off, and the software decoder is invoked to decode the video data of the cloud game.

6. The method according to claim 1, characterized in that, Before acquiring the historical decoding capability data of the terminal device, the method further includes: Determine whether the terminal device supports hardware decoding of the video data of the cloud game; If the terminal device supports hardware decoding of the video data of the cloud game, the step of obtaining the historical decoding capability data of the terminal device is performed. If the terminal device does not support hardware decoding of the video data of the cloud game, software decoding is used to decode the video data of the cloud game.

7. The method according to claim 1, characterized in that, The method further includes: Record the real-time decoding capability data; The historical decoding capability data is updated based on the real-time decoding capability data.

8. The method according to claim 5, characterized in that, The decoding of the cloud game's video data using the hardware decoder includes: The video data of the cloud game is transmitted to the hardware decoder based on the real-time decoding speed of the terminal device, so that the hardware decoder can decode the video data of the cloud game.

9. The method according to claim 1, characterized in that, The historical decoding capability data includes at least one of the following: historical decoding capability data corresponding to the model of the terminal device, historical decoding capability data corresponding to the device manufacturer to which the terminal device belongs, and historical decoding capability data of terminal devices used by the account that logged into the cloud gaming.

10. A cloud gaming data decoding device, characterized in that, Applied to terminal devices, including: The acquisition module is used to acquire historical decoding capability data of the terminal device when the cloud game starts. The historical decoding capability data is data that reflects the hardware decoding capability of the terminal device for video data. The judgment module is used to determine whether the terminal device meets the hardware decoding requirements based on the historical decoding capability data obtained by the acquisition module. The decoding module is used to decode the video data of the cloud game using hardware decoding when the judgment module determines that the terminal device meets the hardware decoding requirements; The judgment module is further configured to determine whether to switch to software decoding mode based on the real-time decoding capability data of the terminal device during the process of the decoding module decoding the video data of the cloud game using hardware decoding mode. The switching module is used to switch to software decoding mode to decode the video data of the cloud game when the judgment module determines that the software decoding mode has been switched. When the historical decoding capability data includes historical decoding speed, the determination module is specifically used for: Determine whether the historical decoding speed is greater than or equal to the second decoding speed threshold; If the historical decoding speed is greater than or equal to the second decoding speed threshold, then the terminal device is determined to meet the hardware decoding requirements. If the historical decoding speed is less than the second decoding speed threshold, then it is determined that the terminal device does not meet the hardware decoding requirements; Alternatively, if the historical decoding capability data includes historical decoding speed and decoding stability indicators, the judgment module is specifically used for: Determine whether the historical decoding speed is greater than or equal to the second decoding speed threshold, and whether the decoding stability index is greater than or equal to the decoding stability threshold; If the historical decoding speed is greater than or equal to the second decoding speed threshold, and the decoding stability index is greater than or equal to the decoding stability threshold, then the terminal device is determined to meet the hardware decoding requirements; otherwise, the terminal device is determined not to meet the hardware decoding requirements. The acquisition module is specifically used for: Obtain a local configuration file, and retrieve the historical decoding capability data of the terminal device from the local configuration file; Download the cloud configuration file and obtain the historical decoding capability data of the terminal device from the cloud configuration file; or, when the local configuration file cannot be obtained, or the amount of historical decoding capability data of the terminal device obtained from the local configuration file does not reach the specified amount of data, obtain the cloud configuration file and obtain the historical decoding capability data of the terminal device from the cloud configuration file; the cloud configuration file includes: historical decoding capability data obtained by statistically analyzing the historical decoding capability data of multiple terminal devices of the same model, the same device manufacturer, or the same series.

11. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-9.

Images