Picture coding method, apparatus, device, medium and program product of cloud application

Abstract

The application discloses a picture coding method, device, equipment, medium and program product of cloud application. The method comprises the following steps: acquiring a first coding gear, the first coding gear corresponding to a first code rate range and a first coding resolution; acquiring code rate prediction data; in response to the code rate prediction data exceeding the first code rate range, acquiring a sampling rate corresponding to the code rate prediction data, the sampling rate being used for sampling the application picture to adapt to the code rate prediction data; coding the application picture based on the first coding resolution and the sampling rate to obtain coding data, and sending the coding data to a terminal. After setting the sampling rate, the application picture is coded using the sampling rate and the first coding resolution, which avoids the problem that the encoder needs to compress the picture more efficiently under the condition that the code rate decreases and the resolution remains unchanged, produces blocky blur and detail and color distortion of the application picture, improves the adaptability between the code rate and the resolution, and improves the picture fluency and performance effect.

Description

Technical Field

[0001] This application relates to the field of multimedia, and in particular to a method, apparatus, device, medium, and program product for encoding images in cloud applications. Background Technology

[0002] Cloud applications are a type of application where a cloud server generates the application screen and sends it to a terminal for display. Taking cloud gaming as an example, the cloud server is responsible for generating and rendering the cloud game screen, encoding the screen using a video encoder, and sending it to the terminal. The terminal is responsible for decoding and displaying the cloud game screen and receiving the player's game actions on the screen.

[0003] In related technologies, when the network bandwidth between the terminal and the cloud server is insufficient during cloud gaming, the cloud server sets the video encoder to a lower encoding bitrate.

[0004] However, a lower bitrate will result in lower image quality, leading to a blurry picture. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and program product for screen encoding in cloud applications. The technical solution is as follows:

[0006] On the one hand, a method for encoding images in cloud applications is provided, the method comprising:

[0007] Obtain the first encoding level, which is the level used by the application screen of the encoding cloud application, and the first encoding level corresponds to the first bitrate range and the first encoding resolution;

[0008] Obtain bitrate prediction data, which is used to express the encoding bitrate of the application screen by the cloud server;

[0009] In response to the bitrate prediction data exceeding the first bitrate range, the sampling rate corresponding to the bitrate prediction data is obtained, and the sampling rate is used to sample the application screen to adapt to the bitrate prediction data.

[0010] The application screen is encoded based on the first encoding resolution and the sampling rate corresponding to the bitrate prediction data to obtain encoded data, and the encoded data is sent to the terminal.

[0011] On the other hand, a screen encoding device for cloud applications is provided, the device comprising:

[0012] The acquisition module is used to acquire the first encoding level, which is the level used by the application screen of the encoding cloud application. The first encoding level corresponds to the first bitrate range and the first encoding resolution.

[0013] The acquisition module is also used to acquire bitrate prediction data, which is used to express the encoding bitrate of the application screen by the cloud server.

[0014] The acquisition module is further configured to, in response to the bitrate prediction data exceeding the first bitrate range, acquire the sampling rate corresponding to the bitrate prediction data, wherein the sampling rate is used to sample the application screen to adapt to the bitrate prediction data;

[0015] The encoding module is used to encode the application screen based on the first encoding resolution and the sampling rate corresponding to the bit rate prediction data to obtain encoded data;

[0016] The sending module is used to send the encoded data to the terminal.

[0017] In an optional embodiment, the acquisition module is further configured to, in response to the bitrate prediction data exceeding the first bitrate range, acquire the second bitrate range in which the bitrate prediction data is located; and acquire the sampling rate corresponding to the second bitrate range, wherein the cloud server pre-stores at least two sampling rates corresponding to the bitrate ranges respectively.

[0018] In an optional embodiment, the acquisition module includes:

[0019] A determining unit is used to determine the second encoding bit corresponding to the second bitrate range, wherein the cloud server pre-stores at least two bitrate ranges corresponding to each encoding bitrate.

[0020] The acquisition unit is used to acquire the sampling rate corresponding to the second encoding level, wherein the cloud server pre-stores the sampling rates corresponding to at least two encoding levels respectively.

[0021] In an optional embodiment, the acquisition module is further configured to, in response to the bitrate prediction data exceeding the first bitrate range, input the bitrate prediction data into a sampling rate prediction model and output the sampling rate corresponding to the bitrate prediction data, wherein the sampling rate prediction model is a pre-trained machine learning model for predicting sampling rates.

[0022] In an optional embodiment, the acquisition module is further configured to acquire the encoding working parameters of the cloud server, wherein the encoding working parameters are load parameters related to encoding the application screen;

[0023] The acquisition module is further configured to input the bitrate prediction data and the encoding working parameters into the sampling rate prediction model, and output the sampling rate corresponding to the bitrate prediction data.

[0024] In an optional embodiment, the acquisition module is further configured to, in response to receiving screen feedback information sent by the terminal, acquire information keywords corresponding to the screen feedback information, wherein the screen feedback information is used to express the display effect of the application screen on the interface of the terminal, and the information keywords are keywords extracted from the screen feedback information to express the display effect;

[0025] The acquisition module is further configured to use the information keywords as tags corresponding to the bitrate prediction data and the encoding working parameters, and to store the bitrate prediction data, the encoding working parameters and the sampling rate as sample data in a sample database, wherein the sample data is used to train the sampling rate prediction model.

[0026] In an optional embodiment, the acquisition module is further configured to acquire at least two candidate sampling rates arranged in ascending order, wherein the first encoding level corresponds to the first sampling rate among the at least two candidate sampling rates;

[0027] The acquisition module is further configured to, in response to the bitrate prediction data exceeding the first bitrate range, determine a second sampling rate corresponding to the bitrate prediction data from the at least two candidate sampling rates based on the comparison relationship between the bitrate prediction data and the first bitrate range, wherein the second sampling rate is the sampling rate among the at least two candidate sampling rates that is adjacent to the value of the first sampling rate.

[0028] In an optional embodiment, the encoding module is further configured to encode the application screen based on the first encoding resolution and the sampling rate when the duration between the most recent resolution adjustment time and the current time within the historical time period reaches a preset duration threshold.

[0029] In an optional embodiment, the acquisition module is further configured to sample at least two bitrate data points within a time window; obtain a weighted average of the at least two bitrate data points to obtain the bitrate prediction data.

[0030] In an optional embodiment, the acquisition module is further configured to acquire bitrate change information, the bitrate change information being used to express the trend of bitrate change;

[0031] The acquisition module is further configured to sample the at least two bitrate data in a first time window when the bitrate change information indicates that the encoded bitrate is decreasing; and to sample the at least two bitrate data in a second time window when the bitrate change information indicates that the encoded bitrate is increasing, wherein the first time window is smaller than the second time window.

[0032] In an optional embodiment, the acquisition module is further configured to acquire bitrate change information, which is used to express the trend of bitrate change; and acquire operation frequency information, which is used to express the frequency of operations received by the terminal for the cloud application within a historical time period.

[0033] The acquisition module is further configured to determine the time window based on the bitrate change information and the operation frequency, and to sample the at least two bitrate data within the time window;

[0034] The rate change information is negatively correlated with the size of the time window; and the operation frequency information is negatively correlated with the size of the time window.

[0035] In an optional embodiment, the acquisition module is further configured to acquire the weight values ​​corresponding to the at least two bitrate data respectively; and acquire the weighted sum of the at least two bitrate data based on the weight values ​​corresponding to the at least two bitrate data respectively;

[0036] The acquisition module is further configured to acquire the weight and value of the weight values ​​corresponding to the at least two bitrate data respectively;

[0037] The acquisition module is further configured to acquire the ratio between the weighted sum and the weighted sum, thereby obtaining the bitrate prediction data.

[0038] In an optional embodiment, the acquisition module is further configured to assign weight values ​​to the at least two bitrate data points according to a preset weight distribution rule, wherein the weight values ​​and the duration between the acquisition time of the bitrate data and the current time are negatively correlated; or...

[0039] The acquisition module is further configured to acquire bitrate-related information corresponding to the acquisition time of the i-th bitrate data, wherein the bitrate-related information is information related to the encoded bitrate; input the bitrate-related information and the i-th bitrate data into the weight prediction model, and output the weight value corresponding to the i-th bitrate data, wherein the weight prediction model is a pre-trained machine learning model for predicting the weight of bitrate data.

[0040] In an optional embodiment, the acquisition module is further configured to acquire screen data of the application screen corresponding to the first encoding resolution; sample the screen data according to the sampling rate to obtain sampled data; and encode the sampled data to obtain the encoded data.

[0041] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the screen encoding method of cloud application as described in any of the above embodiments of this application.

[0042] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the screen encoding method for cloud applications as described in any of the embodiments of this application above.

[0043] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the screen encoding method for the cloud application described in any of the above embodiments.

[0044] The beneficial effects of the technical solutions provided in this application include at least the following:

[0045] After determining the first encoding level corresponding to the resolution configuration, if the network environment deteriorates and the bitrate decreases, the sampling rate corresponding to the bitrate is obtained. The application screen is then sampled using the sampling rate, so that the sampled application screen can adapt to the decreased bitrate. The application screen is then encoded based on the sampling rate and the first encoding resolution of the first encoding level. By setting the sampling rate and using the sampling rate and the first encoding resolution to encode the application screen, the encoder is avoided from needing to compress the screen more efficiently when the bitrate decreases but the resolution remains unchanged. This avoids issues such as blocky blurring and loss of detail and color in the application screen, thus improving the compatibility between bitrate and resolution, and improving the smoothness and performance of the screen. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the structure of a screen encoding system for a cloud application provided in an exemplary embodiment of this application;

[0048] Figure 2 This is a schematic diagram illustrating the interaction process between a cloud server and a terminal, provided in an exemplary embodiment of this application.

[0049] Figure 3 This is a flowchart of a screen encoding method for a cloud application provided in an exemplary embodiment of this application;

[0050] Figure 4 Based on Figure 3 A schematic diagram of the resolution setting process provided in the illustrated embodiment;

[0051] Figure 5 This is a flowchart of a screen encoding method for a cloud application provided in another exemplary embodiment of this application;

[0052] Figure 6 This is a flowchart of a screen encoding method for a cloud application provided in another exemplary embodiment of this application;

[0053] Figure 7 This is a flowchart of a screen encoding method for a cloud application provided in another exemplary embodiment of this application;

[0054] Figure 8 This is a schematic diagram of a screen feedback process provided in an exemplary embodiment of this application;

[0055] Figure 9 This is a schematic diagram of the overall cloud gaming operation process provided in an exemplary embodiment of this application;

[0056] Figure 10 This is a schematic diagram of the downsampling rate curve provided in an exemplary embodiment of this application;

[0057] Figure 11 This is a schematic diagram of the bandwidth curve provided in an exemplary embodiment of this application;

[0058] Figure 12 This is a schematic diagram of the stuttering rate curve provided in an exemplary embodiment of this application;

[0059] Figure 13 This is a schematic diagram of the overall cloud conferencing process provided in an exemplary embodiment of this application;

[0060] Figure 14 This is a schematic diagram of the overall cloud gaming operation process provided by another exemplary embodiment of this application;

[0061] Figure 15 This is a structural block diagram of a screen encoding device for a cloud application provided in another exemplary embodiment of this application;

[0062] Figure 16 This is a structural block diagram of a screen encoding device for a cloud application provided in another exemplary embodiment of this application;

[0063] Figure 17 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0065] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0066] A cloud application is a software application model built and run on cloud computing technology. It provides application services to users over the network, enabling users to use various computing resources, software functions and data storage on demand, without having to install and maintain complex software systems on local devices.

[0067] Cloud applications can be used in a variety of scenarios, such as: 1. Enterprise office: Cloud office applications, such as online document editing, project management, and team collaboration tools, enable employees to collaborate anytime, anywhere, improving work efficiency. 2. E-commerce: E-commerce platforms utilize cloud applications to process massive amounts of order, user information, and product data. 3. Cloud gaming: By performing calculations related to game resources on cloud servers and sending the game screen to the terminal, the terminal displays the game screen and receives player input, realizing the gaming process. 4. Education: Online education platforms leverage cloud applications to provide functions such as course learning, online examinations, and teacher-student interaction.

[0068] It is worth noting that the above application scenarios are merely illustrative examples. In this embodiment, cloud gaming is used as an example for explanation. Currently, the main form of cloud gaming involves running on a cloud server while encoding and compressing the game screen and sound effects, and transmitting them to the terminal in real time using Real-Time Communication (RTC) technology. The terminal receives the encoded and compressed audio and video streams, decodes and plays them to achieve the display of game screens and playback of game sound effects. The terminal receives game operations and transmits the game operation information to the cloud server in real time. The cloud server updates and renders the screen based on the game operation information and then transmits it back to the terminal in real time, repeating this process to achieve a cloud gaming mode where the screen and operations are displayed locally and the game screen is rendered in the cloud.

[0069] During cloud gaming, the computation and rendering take place on the cloud server side. Users do not need high-performance client devices, but they do need a good network to ensure the real-time transmission of audio and video streams. However, the network conditions of terminals are usually diverse, such as wired networks, local area networks, and data networks, and the network conditions are easily affected by various factors, resulting in fluctuations in network quality and bandwidth, such as network congestion and changes in routing status.

[0070] RTC transmission addresses network fluctuations by estimating bandwidth and adjusting the coding rate in a timely manner to match changing network bandwidth, thereby ensuring the quality of transmission service as much as possible.

[0071] However, when network bandwidth is insufficient, the encoder's bitrate usually needs to be set to a lower bitrate to meet the network bandwidth requirements. A lower bitrate means that the amount of data transmitted in the same amount of time is reduced, and the encoder needs to compress the video more efficiently. High dynamic scenes or areas containing fine textures are difficult to keep clear at low bitrates, resulting in loss of image details and a deterioration in overall visual effect. In addition, in order to adapt to the lower bitrate, some encoders may reduce the frame rate, which affects the smoothness of the video, especially in fast-moving scenes.

[0072] This application provides a cloud application-based screen encoding method. After determining the first encoding level corresponding to the resolution configuration, if the network environment deteriorates and the bitrate decreases, the sampling rate corresponding to the bitrate is obtained. The application screen is then sampled using the sampling rate so that the sampled application screen can adapt to the decreased bitrate.

[0073] It should be noted that all information (including but not limited to image information, user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the application screens, account information, and other content involved in this application were obtained with full authorization.

[0074] Figure 1 This is a schematic diagram of the structure of a screen encoding system for a cloud application provided in an exemplary embodiment of this application, which is illustrative. Figure 1 As shown, the system involves a terminal 110 and a cloud server 120. The terminal 110 is used to display the cloud application screen, and the cloud server 120 is used to run the cloud application. The cloud application is implemented as a cloud game, cloud office program, cloud conferencing program, cloud teaching program, etc. The specific application type of the cloud application is not limited in this embodiment.

[0075] In this embodiment, cloud gaming is used as an example to illustrate the implementation of a cloud application. The cloud game runs on cloud server 120. For example, terminal 110 logs into the cloud game on cloud server 120 using a first account and sends resolution configuration information to cloud server 120, such as configuring the cloud game resolution to 1080P. Alternatively, cloud server 120 can provide game footage to terminal 110 with default resolution configuration information. Cloud server 120 obtains the game data of the first account in the cloud game and generates the game footage of the first account in the cloud game. Cloud server 120 encodes the game footage according to the resolution, obtains encoded data, and sends the encoded data to terminal 110. Terminal 110 decodes the encoded data to obtain the game footage and displays the game footage on its interface.

[0076] Terminal 110 can also receive game operations, such as operations input by the player through peripheral devices or operations generated by the player through touch on the terminal interface, and send the operation information of the game operation to cloud server 120. For example, if the game operation is input through a peripheral device, the operation shortcut key information of the game operation is sent to cloud server 120; or if the game operation is implemented through touch, the operation coordinates of the touch operation are sent to cloud server 120. After receiving the operation information of the game operation, cloud server 120 updates the game screen based on the operation information and sends the updated game screen encoding to terminal 110.

[0077] Among them, the resolution configured in the above-mentioned terminal's resolution configuration information also corresponds to a bitrate range, such as: the bitrate range corresponding to a resolution of 1080P is 7-10Mbps.

[0078] In some embodiments, when the network bandwidth between terminal 110 and cloud server 120 deteriorates, cloud server 120 automatically reduces the encoding bitrate. In this embodiment, when the encoding bitrate is lower than the bitrate range corresponding to the resolution configuration information, cloud server 120 obtains the downsampling rate corresponding to the reduced encoding bitrate, downsamples the game screen based on the downsampling rate, and then encodes it to obtain encoded data. By reducing the resolution through downsampling, the adaptability between the resolution and the encoding bitrate is improved, thereby improving the smoothness and clarity of the game screen.

[0079] It is worth noting that the aforementioned cloud server 120 can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0080] Figure 2This is a schematic diagram illustrating the interaction process between a cloud server and a terminal, provided in an exemplary embodiment of this application. Figure 2 As shown, firstly, the cloud server encodes the game screen at a first resolution. At this time, the cloud server's encoder corresponds to a first encoding bitrate and sends the encoded data to the terminal. After receiving the encoded data, the terminal decodes and displays the game screen 210. The encoding bitrate range corresponding to the first resolution is 7-10Mbps.

[0081] When the network bandwidth between the terminal and the cloud server deteriorates, the cloud server automatically reduces the first encoding bitrate to the second encoding bitrate.

[0082] If the second encoding bitrate is lower than 7Mbps, the cloud server obtains the first sampling rate, which is used to downsample the game screen. After downsampling the game screen using the first sampling rate, the cloud server obtains the game screen corresponding to the second resolution, encodes the game screen based on the second resolution, and sends the encoded data to the terminal. After receiving the encoded data, the terminal decodes and displays the game screen 220.

[0083] Based on the above, the screen encoding method for cloud applications provided in the embodiments of this application will be described.

[0084] Figure 3 This is a flowchart of a screen encoding method for a cloud application provided in an exemplary embodiment of this application. The method is executed by a computer device; in this embodiment, the method is executed by a cloud server as an example. Figure 3 As shown, the method includes the following steps.

[0085] Step 320: Obtain the first encoding level. The first encoding level is the level used by the application screen of the encoding cloud application.

[0086] In some embodiments, cloud applications include at least one of the following applications: instant messaging applications, game programs, shopping programs, multimedia playback programs, financial management programs, medical programs, catering programs, office programs, organizational management programs, etc., and this application embodiment does not limit this.

[0087] The first encoding level corresponds to the first bitrate range and the first encoding resolution. Illustratively, the first encoding level corresponds to any of the following levels:

[0088] 1. Bitrate range 7Mbps~10Mbps, encoding resolution 1080P;

[0089] 2. Bitrate range 4Mbps~7Mbps, encoding resolution 720P;

[0090] 3. Bitrate range 1Mbps~4Mbps, encoding resolution 540P.

[0091] It is worth noting that the values ​​in the above-mentioned encoding positions are merely illustrative examples, and the specific values ​​in the encoding positions are not limited in this application embodiment; in addition, the number of the above-mentioned encoding positions is merely illustrative examples, and the number of encoding positions may be more or less, and this application embodiment does not limit this.

[0092] Optionally, the method for obtaining the first encoded bit includes at least one of the following methods:

[0093] First, the system receives resolution configuration information from the receiving terminal and obtains the first encoding level based on this information. Optionally, the terminal receives a resolution configuration operation and sends the corresponding resolution configuration information to the cloud server. The resolution configuration operation is a configuration operation triggered by the user on the terminal to specify the desired image resolution. For example, during the display of the cloud application's image, the user triggers the display of the resolution configuration area and performs a resolution configuration operation within that area.

[0094] To illustrate, when a terminal receives a resolution setting operation, it sends the operation information to the cloud server. Based on this operation information, the cloud server sends an application screen to the terminal. This application screen includes a resolution configuration area, which contains at least two candidate resolutions. The terminal receives a configuration operation for the first resolution within the resolution configuration area as a resolution configuration operation and sends the operation information of this operation to the cloud server. The cloud server then determines the first encoding level corresponding to the first resolution based on the operation information. For example, the encoding resolution is 1080P. Figure 4 This is a schematic diagram of the interface for the resolution setting process provided in an exemplary embodiment of this application, such as... Figure 4 As shown, the terminal displays a resolution configuration area 400, which includes at least two candidate resolutions. The terminal receives a configuration operation for the first resolution 410 in the resolution configuration area 400 and sends the operation information of the resolution configuration operation to the cloud server.

[0095] The resolution is set by the terminal, and the application screen is encoded based on the user's ideal resolution. Then, the resolution is adjusted by downsampling rate when the encoding bitrate is reduced. This takes into account both user expectations and the impact of network bandwidth environment, and improves the flexibility of determining the resolution based on the encoding bitrate.

[0096] In some embodiments, the resolution configuration information sent by the terminal is used to instruct the cloud server to provide the cloud application client with the upper limit of the resolution of the application screen. That is, even if the encoding bitrate is increased, the encoding resolution is not greater than the resolution configured in the resolution configuration information.

[0097] Second, the pre-set default encoding resolution is used as the first encoding resolution. That is, the cloud server pre-sets the encoding resolution of the terminal display application screen as the default encoding resolution, such as 1080P.

[0098] Third, the first encoding level is automatically determined based on the network bandwidth information between the cloud server and the terminal. For example, network latency data is acquired, and the first encoding level is automatically determined based on this data. The cloud server pre-stores the correspondence between network latency data and encoding levels.

[0099] It is worth noting that the above-described method for determining the first encoding level is merely an illustrative example, and the embodiments of this application do not limit the method for determining the first encoding level.

[0100] Step 340: Obtain bitrate prediction data. Bitrate prediction data is used to express the encoding bitrate of the application screen by the cloud server.

[0101] In some embodiments, the bitrate prediction data can also be used to express the transmission bitrate of the application screen between the cloud server and the terminal. To ensure that the encoded data of the application screen can be transmitted smoothly, the encoding bitrate usually needs to be less than or equal to the available transmission bitrate, and the encoding bitrate needs to be adjusted according to the available bandwidth of the network environment. For example, in a network environment with limited bandwidth, a lower encoding bitrate can be selected to accommodate a lower transmission bitrate.

[0102] In some embodiments, the bitrate prediction data is the encoded bitrate predicted based on network bandwidth. The bitrate prediction data is predicted by the cloud server and applied to the encoder to determine the encoded bitrate.

[0103] The methods for obtaining bitrate prediction data include at least one of the following:

[0104] First, directly obtain the bitrate prediction data corresponding to the current network bandwidth;

[0105] When acquiring bitrate prediction data corresponding to network bandwidth, an initial bitrate is first determined based on the first encoding tier. For example, if the first encoding tier is 1080P, then 8Mbps is selected from 7-10Mbps as the initial bitrate. Feedback information from the terminal, such as buffer status and packet loss reports, is received, and the encoding bitrate is predicted and adjusted in real time based on the feedback information. Alternatively, network bandwidth usage information, including upload / download speeds, round-trip latency, packet loss rate, and latency, is acquired, and bitrate prediction data is determined based on network bandwidth usage and the current bitrate.

[0106] For illustrative purposes, if the current encoding bitrate is 8Mbps, and the feedback information sent by the terminal indicates that the packet loss rate of the terminal display screen is 20% and the download speed is 6Mbps, then the bitrate is reduced based on the packet loss rate and download speed, and the bitrate prediction data is obtained as 5Mbps.

[0107] Obtaining bitrate prediction data based on network bandwidth improves the real-time performance and accuracy of bitrate prediction data, reduces the amount of data processing during bitrate prediction, reduces the consumption of computing resources, and improves the efficiency of bitrate prediction and subsequent downsampling rate prediction.

[0108] Second, sample at least two bitrate data points within a time window, obtain the weighted average of the at least two bitrate data points, and get the bitrate prediction data.

[0109] After sampling at least two bitrate data points within a time window, a weighted average is obtained to get the bitrate prediction data. This avoids the problem of inaccurate bitrate prediction data expressed at a single point in time. By expressing the bitrate change trend and bitrate prediction data within a time window, the accuracy of the bitrate prediction data is improved, as well as the accuracy of determining the subsequent sampling rate.

[0110] The time window can be implemented as a preset time window; or, the time window can be implemented as a dynamically determined time window. The following explanations will address these two scenarios:

[0111] 1. The time window is implemented as a preset time window.

[0112] In some embodiments, the preset time window varies depending on the bitrate change. In some embodiments, bitrate change information is acquired, and this bitrate change information is used to express the bitrate change trend.

[0113] Optionally, when acquiring the bitrate change trend, at least one of the following methods may be used:

[0114] 1. Obtain the trend of change of at least two bitrate data points collected within a historical time period as bitrate change information;

[0115] For illustrative purposes, if at least two bitrate data points collected within a historical time period show a decreasing trend, the bitrate change information is 1; if at least two bitrate data points collected within a historical time period show an increasing trend, the bitrate change information is 0.

[0116] 2. Obtain network bandwidth information and determine bitrate changes based on network bandwidth information;

[0117] Indicatively, feedback information from the terminal is obtained, including the packet loss rate. If the packet loss rate is greater than a preset threshold, the bit rate is determined to be decreasing; if the packet loss rate is less than the preset threshold, the bit rate is determined to be increasing.

[0118] 3. Obtain the first bitrate data collected at the current moment, and obtain the second bitrate data collected most recently within the historical time period. Determine the bitrate change based on the comparison between the first bitrate data collected at the current moment and the second bitrate data collected most recently within the historical time period.

[0119] If the first bitrate data is less than the second bitrate data, it indicates that the bitrate is decreasing; if the first bitrate data is greater than the second bitrate data, it indicates that the bitrate is increasing.

[0120] 4. First, collect at least two bitrate data points within a first time window. Based on the changing trends of these two bitrate data points within the first time window, determine whether to extend the first time window to a second time window. The length of the first time window is shorter than the length of the second time window.

[0121] By directly determining the trend of change through the first time window, at least two bitrate data points can be collected simultaneously for subsequent use, improving the execution efficiency and parallelism of time window and bitrate data acquisition.

[0122] To illustrate, firstly, one bitrate data point is collected every 1 second within a 5-second time window. Based on the trend of the bitrate data within the 5-second window, it's determined whether to extend the 5-second time window to 8 seconds. If extended to 8 seconds, bitrate data is collected for another 3 seconds, resulting in the bitrate data collected in the second time window. To illustrate, the trend of bitrate data within the 5-second time window is determined by comparing the first bitrate data point (the first collected bitrate data point) with the last bitrate data point. For example, if the first bitrate data point is greater than the last bitrate data point, then the bitrate data within the 5-second time window is determined to be decreasing.

[0123] When determining the time window based on bitrate variation information, the following situations are included:

[0124] 1.1 When the bitrate change information indicates that the encoded bitrate is decreasing, at least two bitrate data points shall be sampled within the first time window;

[0125] The first time window is a pre-defined time window for situations where the encoded bitrate decreases. In some embodiments, at least two bitrate data points are sampled within the first time window according to a preset sampling period.

[0126] For illustration, with a preset sampling period of 1 second and a first time window of 5 seconds, at least two bitrate data points are sampled within the 5-second time window at a sampling period of 1 second. Specifically, if the first sampling moment is taken from the start of the 5-second time window, a total of 6 bitrate data points are sampled within the 5-second time window. If the start of the 5-second time window is taken as the start timing moment and the first 1-second moment is taken as the first sampling moment, a total of 5 bitrate data points are sampled within the 5-second time window.

[0127] 1.2 When the bitrate change information indicates that the encoded bitrate has increased, at least two bitrate data points are resampled in the second time window.

[0128] The second time window is a pre-defined time window for situations where the encoded bitrate increases, and the first time window is smaller than the second time window, meaning the window length of the first time window is less than the window length of the second time window. In some embodiments, at least two bitrate data points are sampled within the second time window according to a preset sampling period.

[0129] For illustration, with a preset sampling period of 1 second and a first time window of 8 seconds, at least two bitrate data points are sampled within the 8-second time window at a sampling period of 1 second. Specifically, the first sampling moment is taken from the start of the 8-second time window, so a total of 9 bitrate data points are sampled within the 8-second time window. Alternatively, if the start of the 8-second time window is taken as the start timing moment and the first 1-second moment is taken as the first sampling moment, a total of 8 bitrate data points are sampled within the 8-second time window.

[0130] When the encoding bitrate decreases, a smaller first time window is used to sample at least two bitrate data points. This enables rapid resolution reduction to adapt to bitrate changes, improving the response speed to network bandwidth changes and enhancing the adaptability between bitrate and resolution. When the encoding bitrate increases, a larger time window is used to sample at least two bitrate data points. This avoids excessive resolution switching caused by frequent changes in network bandwidth, ensuring that the resolution is increased after the network bandwidth stabilizes, and reducing resolution fluctuations.

[0131] In this illustrative embodiment, to reduce frequent resolution switching caused by frequent network jitter, an asymmetric sliding window is designed, using a smaller time window W. down Statistical decrease in coding bitrate B down Determine whether to reduce the resolution to respond quickly to a decrease in network bandwidth and reduce lag; use a larger time window W up Statistical increase in coding bitrate B up This determines whether to increase the resolution to ensure that the resolution is increased after the network bandwidth has stabilized, thus reducing resolution fluctuations.

[0132] For example: If we judge B down If B down If the current bitrate is below the lower limit, switch to the next downsampling rate. As the downsampling rate increases, the encoding resolution decreases, until the lowest bitrate is reached. If B is judged... up If B up If the current bitrate reaches its upper limit, the system switches to the previous downsampling rate, decreasing the downsampling rate and increasing the resolution, until the highest bitrate is reached, at which point downsampling ceases. If neither of these two conditions is met, the downsampling rate remains unchanged, meaning the encoding resolution remains constant.

[0133] In some embodiments, bitrate change information is acquired, which is used to express the trend of bitrate change; operation frequency information is acquired, which is used to express the frequency of operations received by the terminal for cloud applications within a historical time period. A time window is determined based on the bitrate change information and the operation frequency, and at least two bitrate data are sampled within the time window; wherein the magnitude of the change expressed by the bitrate change information is negatively correlated with the size of the time window; and the operation frequency expressed by the operation frequency information is negatively correlated with the size of the time window.

[0134] In a schematic manner, bit rate change information is quantized into first information data, and operation frequency information is quantized into second information data. The window size of the time window is calculated based on the first information data and the second information data.

[0135] 2. The time window is implemented as a dynamically determined time window.

[0136] Optionally, the size of the bitrate data is positively correlated with the size of the time window; that is, the smaller the bitrate data, the smaller the time window.

[0137] In some embodiments, a correspondence between the bitrate data range and the window size is set, and the time window is dynamically determined based on the bitrate data. For example, the time window corresponding to bitrate data of 4Mbps-7Mbps is set to 5s, and the time window corresponding to 1Mbps-4Mbps is set to 3s.

[0138] By using dynamically determined time windows, when the encoding bitrate decreases, the resolution is adjusted more promptly, improving the resolution's adaptability to bitrate encoding. Similarly, when the encoding bitrate increases, the larger the bitrate, the longer the interval for adjusting the resolution, improving the stability of the resolution.

[0139] In other embodiments, a window prediction model is pre-trained; this model is a pre-trained machine learning model. At least two of the following are used as input parameters to the window prediction model: bitrate data, network environment data, and encoding information. The output is a dynamically determined time window, and at least two bitrate data points are collected according to the time window. The bitrate data input to the window prediction model includes bitrate data sampled at the current time, or historical bitrate data collected at a time prior to a preset duration, such as bitrate data collected 2 seconds ago. The network environment data includes packet loss rate, upload / download rate, round-trip time, latency, and other data from feedback information sent by the terminal. The encoding information includes encoding format, encoding algorithm, and other information, which are not limited in this embodiment.

[0140] After obtaining at least two bitrate data points through sampling, a weighted average of the at least two bitrate data points is obtained to obtain bitrate prediction data. Specifically, the weight values ​​corresponding to the at least two bitrate data points are obtained; a weighted sum of the at least two bitrate data points is obtained based on these weight values; a weighted sum of the weight values ​​corresponding to the at least two bitrate data points is obtained; and the ratio between the weighted sum and the weighted sum is obtained to obtain the bitrate prediction data.

[0141] In some embodiments, weight values ​​are assigned to at least two bitrate data according to a preset weight distribution rule. The weight value and the duration between the acquisition time of the bitrate data and the current time are negatively correlated. That is, the longer the duration between the acquisition time of the bitrate data and the current time, the smaller the weight value.

[0142] The preset weight distribution rules include pre-defined candidate weight values, such as 0.1, 0.3, 0.5, 0.7, and 0.9. Based on the number of at least two bitrate data points, weights are assigned starting with the smallest weight value and then to the bitrate data point with the longest duration between the acquisition time and the current time. For example, if five bitrate data points are sampled and ordered by acquisition time as b1, b2, b3, b4, and b5, then b1 is assigned a weight of 0.1, b2 0.3, b3 0.5, b4 0.7, and b5 0.9. By pre-setting candidate weight values, after acquiring at least two bitrate data points, weights are directly assigned to them according to the candidate weight values, improving weight acquisition efficiency and reducing computational latency and resource consumption during weight determination.

[0143] The preset weight distribution rules also include pre-defined weighting algorithms. For example, a pre-defined weighting algorithm might be a = 1 - 0.1t, where a represents the weight and t represents the duration between the acquisition time of the bitrate data and the current time. For instance, if the duration between the acquisition time of bitrate data b and the current time is 3 seconds, then the weight of bitrate data b is 0.7. By pre-setting weighting algorithms, after acquiring at least two bitrate data points, weights are assigned to these at least two bitrate data points. Even when the number of bitrate data samples is large, the weighting algorithm can still determine the weight of each bitrate data point, improving the adaptability and capability of bitrate data weighting and increasing the efficiency of weight determination.

[0144] The preset weight distribution rules also include a pre-trained weight prediction model. Bitrate-related information is obtained at the acquisition time of the i-th bitrate data point. This bitrate-related information is information related to the encoded bitrate. The bitrate-related information and the i-th bitrate data are input into the weight prediction model, which outputs the weight value corresponding to the i-th bitrate data. The weight prediction model is a pre-trained machine learning model used to predict the weights of the bitrate data. The bitrate-related information includes image complexity, resolution, encoding algorithm, encoding format, and color space. Resolution, encoding algorithm, encoding format, and color space are pre-defined information. Image complexity can be obtained through a complexity analysis algorithm on the application image. For example, image complexity can be expressed based on the variance between the pixel values ​​of each pixel in the application image. Through the weight prediction model, after acquiring at least two bitrate data points, weights are assigned to these at least two bitrate data points. In the weight determination process, in addition to the bitrate data itself, consideration of bitrate-related information is introduced, improving the confidence of the weights and the accuracy of the subsequently determined bitrate prediction data.

[0145] Third, obtain the bitrate impact information at the current moment, input the bitrate impact information into the bitrate prediction model, and output bitrate prediction data.

[0146] Among them, bitrate impact information refers to relevant information that affects the encoding bitrate.

[0147] To illustrate, bitrate-related information includes network environment data, image complexity, resolution, encoding algorithm, encoding format, color space, and other information.

[0148] Bitrate impact information is input into the bitrate prediction model, which outputs bitrate prediction data corresponding to the bitrate impact information. The bitrate prediction model is trained using sample data. Illustratively, sample bitrate impact information and corresponding reference bitrate data are obtained. This sample bitrate impact information is then incorporated into the prediction model to be trained, outputting predicted bitrate data. The prediction model is trained based on the difference between the reference and predicted bitrate data, resulting in a trained bitrate prediction model. This allows for direct input of bitrate impact information into the bitrate prediction model, outputting predicted bitrate data.

[0149] By using a bitrate prediction model, after obtaining information on factors affecting bitrate, the model analyzes the weights to obtain bitrate prediction data. This approach considers not only management environment data but also other information that may affect bitrate during the bitrate prediction process, thus improving the accuracy of the bitrate prediction data. Furthermore, since the bitrate prediction model is a pre-trained machine learning model, simply inputting the bitrate impact information into the model will output the bitrate prediction data, thereby improving the efficiency of bitrate prediction data acquisition.

[0150] It is worth noting that the above-described method for obtaining bitrate prediction data is merely an illustrative example, and the embodiments of this application do not limit it.

[0151] Step 360: In response to the bitrate prediction data exceeding the first bitrate range, obtain the sampling rate corresponding to the bitrate prediction data.

[0152] The sampling rate is used to sample the application screen to fit the bitrate prediction data.

[0153] In some embodiments, after obtaining the first encoding level, the application screen is encoded based on the resolution and encoding bitrate range of the first encoding level. When the bitrate prediction data is lower than the lowest bitrate of the first bitrate range or higher than the highest bitrate of the first bitrate range, the sampling rate corresponding to the bitrate prediction data is obtained.

[0154] In some embodiments, if the first bitrate range is the first bitrate range corresponding to the resolution level selected by the user, then the sampling rate corresponding to the bitrate prediction data is obtained only for the case where the bitrate is lower than the lowest bitrate of the first bitrate range. That is, the resolution of the resolution level corresponding to the first bitrate range is the upper limit of the resolution set by the user.

[0155] Among them, the bitrate prediction data is negatively correlated with the sampling rate; that is, the smaller the bitrate prediction data, the higher the sampling rate.

[0156] The higher the sampling rate, the sparser the sampling of the application screen.

[0157] In some embodiments, taking the lowest bitrate where the bitrate prediction data is less than the first bitrate range as an example, the method of obtaining the sampling rate includes: obtaining the next-order sampling rate corresponding to the current first bitrate range, that is, obtaining the maximum sampling rate in the sampling rate set that is less than the sampling rate corresponding to the first bitrate range; or, obtaining the sampling rate corresponding to the data value of the bitrate prediction data.

[0158] In some embodiments, there is a correspondence between the sampling rate and the encoding level, which is preset, or the correspondence is determined based on a first encoding level determined in real time.

[0159] For illustrative purposes, when the correspondence is pre-defined, a sampling rate of 1x corresponds to a 1080P resolution, a sampling rate of 1.5x corresponds to a 720P resolution, and a sampling rate of 2x corresponds to a 540P resolution. That is, the sampling rate corresponding to 1080P means no sampling of the application screen, maintaining 1080P resolution; the sampling rate corresponding to 720P means sampling the 1080P application screen, with the application screen resolution equivalent to 720P after 1.5x sampling; and the sampling rate corresponding to 540P means sampling the 1080P application screen, with the application screen resolution equivalent to 540P after 2x sampling. A 1.5x sampling rate can be considered as sampling 2 pixels out of every 3 pixels; a 2x sampling rate can be considered as sampling 1 pixel out of every 2 pixels.

[0160] For illustrative purposes, when the correspondence is determined based on the first encoding level, with the first encoding level being the level corresponding to a 720P resolution, the sampling rate corresponding to a 1080P resolution is 0.67 times, the sampling rate corresponding to a 720P resolution is 1 times, and the sampling rate corresponding to a 540P resolution is 1.33 times.

[0161] Step 380: Encode the application screen based on the sampling rate corresponding to the first encoding resolution and the bit rate prediction data to obtain encoded data, and send the encoded data to the terminal.

[0162] Optionally, the screen data of the application screen corresponding to the first encoding resolution is obtained; the screen data is sampled according to the sampling rate to obtain sampled data; and the sampled data is encoded to obtain encoded data.

[0163] In some embodiments, in response to the bitrate prediction data exceeding a first bitrate range, a second bitrate range in which the bitrate prediction data is located is obtained; the sampling rate corresponding to the second bitrate range is obtained; and the application screen is encoded based on the first encoding resolution and the sampling rate corresponding to the second bitrate range to obtain encoded data.

[0164] In an optional embodiment, a second encoding level corresponding to the second bitrate range is determined, a sampling rate corresponding to the second encoding level is obtained, and the application screen is encoded based on the first encoding resolution and the sampling rate corresponding to the second encoding level to obtain encoded data.

[0165] In some embodiments, if the encoding bitrate continues to decrease after encoding the application screen based on the sampling rate, it is determined whether the encoding bitrate exceeds the bitrate range of the corresponding sampling rate level. If it exceeds the bitrate range of the corresponding sampling rate level, the sampling rate is adjusted until the maximum or minimum sampling rate is reached.

[0166] In some embodiments, if the duration between the most recent time of resolution or sampling rate adjustment within a historical time period and the current time reaches a duration threshold, the application screen is encoded based on the first encoding resolution and sampling rate.

[0167] In summary, the method provided in this application, after determining the first encoding level corresponding to the resolution configuration, if the network environment deteriorates and the bitrate decreases, obtains the sampling rate corresponding to the bitrate, and then samples the application screen using the sampling rate, so that the sampled application screen can adapt to the decreased bitrate. The application screen is then encoded based on the sampling rate and the first encoding resolution of the first encoding level. By setting the sampling rate and using the sampling rate and the first encoding resolution to encode the application screen, the problem of the encoder needing to compress the screen more efficiently when the bitrate decreases but the resolution remains unchanged, which would cause blocky blurring and distortion of details and colors in the application screen, is avoided. This improves the adaptability between bitrate and resolution, and enhances the smoothness and performance of the screen.

[0168] In an optional embodiment, when determining the sampling rate corresponding to the bitrate prediction data, at least two candidate sampling rates are obtained, and the sampling rate corresponding to the bitrate prediction data is determined based on the candidate sampling rates. Figure 5 This is a flowchart of a screen encoding method for a cloud application provided in another exemplary embodiment of this application. This method is executed by a computer device; in this embodiment, the method is executed by a cloud server as an example. Figure 5 As shown, step 360 above includes steps 361 to 362.

[0169] Step 361: Obtain at least two candidate sampling rates arranged in ascending order.

[0170] The first encoding level corresponds to the first sampling rate among at least two candidate sampling rates; or, the sampling rate currently applied by the encoder is the first sampling rate. That is, when the sampling rate was most recently adjusted within the historical time period, the encoder adjusted the sampling rate of the application screen to the first sampling rate.

[0171] In some embodiments, there is a one-to-one correspondence between at least two candidate sampling rates and at least two encoding bits. That is, each encoding bit corresponds to one candidate sampling rate. For illustration, a sampling rate of 1x corresponds to a resolution of 1080P, a sampling rate of 1.5x corresponds to a resolution of 720P, and a sampling rate of 2x corresponds to a resolution of 540P.

[0172] In other embodiments, among at least two candidate sampling rates, some candidate sampling rates correspond to the encoding level, while the other candidate sampling rates are pre-set sampling rates corresponding to resolutions other than the encoding level. For example, 1080P corresponds to a sampling rate of 1. When the sampling rate is 1.2, the corresponding resolution is 1600×900; when the sampling rate is 1.5, the corresponding resolution is 1280×720, or 720P. The sampling rate can be any pre-set value.

[0173] In this embodiment, the example of a one-to-one correspondence between at least two candidate sampling rates and at least two encoding levels is used for illustration. The candidate sampling rates corresponding to the at least two encoding levels are arranged in ascending order. For example, the sampling rate corresponding to 1080P is 1x, the sampling rate corresponding to 720P is 1.5x, and the sampling rate corresponding to 540P is 2x, so they are arranged in ascending order as 1, 1.5, 2.

[0174] Step 362: In response to the bitrate prediction data exceeding the first bitrate range, based on the comparison relationship between the bitrate prediction data and the first bitrate range, determine the sampling rate corresponding to the bitrate prediction data from at least two candidate sampling rates.

[0175] In response to the bitrate prediction data exceeding the first bitrate range, a second sampling rate corresponding to the bitrate prediction data is determined from at least two candidate sampling rates based on the comparison relationship between the bitrate prediction data and the first bitrate range.

[0176] The second sampling rate is the sampling rate that is adjacent to the value of the first sampling rate among at least two candidate sampling rates.

[0177] In some embodiments, when the predicted bitrate is less than the minimum of a first bitrate range, i.e., the encoded bitrate decreases, a minimum sampling rate greater than the first sampling rate is determined from at least two candidate sampling rates. Optionally, if the first sampling rate is not the maximum of the at least two candidate sampling rates, a minimum sampling rate greater than the first sampling rate is determined from at least two candidate sampling rates.

[0178] For illustration, the first sampling rate is sampling rate 1, which corresponds to 1080P. When the bit rate prediction data is less than the minimum value of the first bit rate range, the minimum sampling rate greater than 1 times, i.e., 1.5 times, is obtained from at least two candidate sampling rates 1, 1.5, and 2 as the second sampling rate.

[0179] In some embodiments, when the predicted bitrate data is greater than the maximum value of a first bitrate range, i.e., the encoded bitrate increases, a maximum sampling rate less than the first sampling rate is determined from at least two candidate sampling rates. Optionally, if the first sampling rate is not the minimum value among at least two candidate sampling rates, a maximum sampling rate less than the first sampling rate is determined from at least two candidate sampling rates.

[0180] For illustration, the first sampling rate is sampling rate 2 corresponding to 540P. When the bit rate prediction data is greater than the maximum value of the first bit rate range, the maximum sampling rate less than 2 times, i.e., 1.5 times, is obtained from at least two candidate sampling rates 1, 1.5, and 2 as the second sampling rate.

[0181] In summary, the method provided in this embodiment, after determining the first encoding level corresponding to the resolution configuration, if a network environment deteriorates leading to a decrease in bitrate, obtains the sampling rate corresponding to the bitrate and encodes the application screen based on the sampling rate and the first encoding resolution of the first encoding level. This avoids the problem of the encoder needing to compress the screen more efficiently when the bitrate decreases while the resolution remains unchanged, thus preventing issues such as blocky blurring and distortion of details and colors in the application screen. This improves the adaptability between bitrate and resolution, enhancing screen smoothness and performance. Furthermore, to avoid large fluctuations in the sampling rate that could lead to poor screen stability, the method controls the resolution of each sampling rate switch to an adjacent level, improving the stability of resolution changes and further enhancing the smoothness of the application screen.

[0182] In an optional embodiment, when determining the sampling rate corresponding to the bitrate prediction data, the sampling rate is determined based on the second bitrate range in which the bitrate prediction data is located. Figure 6 This is a flowchart of a screen encoding method for a cloud application provided in another exemplary embodiment of this application. This method is executed by a computer device; in this embodiment, the method is executed by a cloud server as an example. Figure 6 As shown, step 360 above includes steps 363 to 364.

[0183] Step 363: In response to the bitrate prediction data exceeding the first bitrate range, obtain the second bitrate range in which the bitrate prediction data is located.

[0184] Because different encoding levels correspond to different bitrate ranges, the bitrate ranges corresponding to different encoding levels may overlap or not overlap.

[0185] In some embodiments, when the bitrate ranges corresponding to the encoding bits overlap, obtaining the second bitrate range in which the bitrate prediction data is located includes at least one of the following methods:

[0186] 1. In a downward trend, if the predicted bitrate is lower than the lower limit of the first bitrate range and enters the second bitrate range, the sampling rate corresponding to the second bitrate will be switched. Conversely, in an upward trend, if the predicted bitrate is higher than the upper limit of the second bitrate range and enters the first bitrate range, the sampling rate corresponding to the first bitrate will be switched.

[0187] 2. When the bitrate prediction data is in at least two bitrate ranges, randomly select one bitrate range from the at least two bitrate ranges as the second bitrate range;

[0188] For illustrative purposes, the bitrate range for 1080P encoding is 7Mbps-10Mbps, and for 720P encoding, it's 5Mbps-8Mbps. If the predicted bitrate is 7.5Mbps, it falls within both 1080P and 720P encoding. Therefore, the bitrate range of one of these two encoding levels is randomly selected as the second bitrate range for the predicted bitrate. Randomly determining the second bitrate range improves efficiency and reduces latency and computational resource consumption during this process.

[0189] 3. When the bitrate prediction data falls within at least two bitrate ranges, acquire bitrate change information and determine a second bitrate range from these two ranges based on this information. For example, if the bitrate change information indicates a decrease in the encoded bitrate, determine the bitrate range corresponding to the smaller resolution from these two ranges as the second bitrate range; similarly, if the bitrate encoding information indicates an increase in the encoded bitrate, determine the bitrate range corresponding to the larger resolution from these two ranges as the second bitrate range.

[0190] For illustrative purposes, the bitrate range for 1080P encoding is 7Mbps-10Mbps, and for 720P encoding, it's 5Mbps-8Mbps. If the predicted bitrate is 7.5Mbps, then it falls within both 1080P and 720P encoding ranges. If the bitrate change information indicates a decrease in bitrate, the bitrate range corresponding to the 720P encoding range is selected as the second bitrate range; if the bitrate change information indicates an increase in bitrate, the bitrate range corresponding to the 1080P encoding range is selected as the second bitrate range.

[0191] The second bitrate range is determined based on the bitrate change information, so that the second bitrate range can adapt to the changes in the encoded bitrate in the bitrate change information, thereby improving the efficiency of sampling rate adjustment and reducing the delay in the sampling rate adjustment process.

[0192] Step 364: Obtain the sampling rate corresponding to the second bit rate range.

[0193] The cloud server stores the sampling rates corresponding to at least two bitrate ranges.

[0194] In an optional embodiment, the cloud server pre-stores bitrate ranges corresponding to at least two encoding bits, and pre-stores sampling rates corresponding to at least two encoding bits. Then, the second encoding bit corresponding to the second bitrate range is determined, and the sampling rate corresponding to the second encoding bit is obtained.

[0195] In summary, the method provided in this embodiment, after determining the first encoding level corresponding to the resolution configuration, if a network environment deteriorates leading to a decrease in bitrate, obtains the sampling rate corresponding to the bitrate. Then, it encodes the application screen based on the sampling rate and the first encoding resolution of the first encoding level. This avoids the problem of the encoder needing to compress the screen more efficiently when the bitrate decreases while the resolution remains unchanged, thus preventing blocky blurring and distortion of details and colors in the application screen. This improves the adaptability between bitrate and resolution, and enhances the smoothness and performance of the image. Furthermore, by preserving the relationship between the bitrate range and the sampling rate, after determining the second bitrate range corresponding to the bitrate prediction data, it obtains the sampling rate corresponding to the second bitrate range. This allows for sampling of the application screen based on the sampling rate, improving the efficiency and accuracy of sampling rate determination and enhancing the harmony between the encoding bitrate and the application screen resolution.

[0196] In an optional embodiment, when determining the sampling rate corresponding to the bitrate prediction data, the sampling rate is predicted by a sampling rate prediction model. Figure 7 This is a flowchart of a screen encoding method for a cloud application provided in another exemplary embodiment of this application. This method is executed by a computer device; in this embodiment, the method is executed by a cloud server as an example. Figure 7 As shown, step 360 above includes the following step 365.

[0197] Step 365: In response to the bitrate prediction data exceeding the first bitrate range, input the bitrate prediction data into the sampling rate prediction model and output the sampling rate corresponding to the bitrate prediction data.

[0198] Among them, the sampling rate prediction model is a machine learning model that is pre-trained to predict the sampling rate.

[0199] In some embodiments, the encoding working parameters of the cloud server are obtained. The encoding working parameters are load parameters related to the encoding application screen. The bitrate prediction data and the encoding working parameters are input into the sampling rate prediction model, and the sampling rate corresponding to the bitrate prediction data is output.

[0200] Among them, the encoding working parameters refer to the relevant parameters that affect the encoder load. These parameters are illustrative and include parameters such as image complexity, resolution, encoding algorithm, encoding format, and color space. The specific content of the encoding working parameters is not limited in the embodiments of this application.

[0201] In some embodiments, the sampling rate prediction model is a machine learning model trained using sample encoding parameters, which are of the same parameter type as the encoding working parameters.

[0202] To illustrate, sample encoding parameters are obtained, with a reference sampling rate labeled on them. The sample encoding parameters are then input into the sampling rate prediction model to be trained, and the predicted sampling rate is output. Based on the difference between the reference sampling rate and the predicted sampling rate, the sampling rate prediction model is trained to obtain the trained sampling rate prediction model.

[0203] In some embodiments, in response to receiving screen feedback information sent by the terminal, information keywords corresponding to the screen feedback information are obtained. The screen feedback information is used to express the display effect of the application screen on the terminal interface, and the information keywords are keywords extracted from the screen feedback information to express the display effect. The information keywords are used as labels corresponding to the bitrate prediction data and encoding parameters, and the bitrate prediction data and encoding parameters are stored as sample data in a sample database. The sample data is used to train the sampling rate prediction model.

[0204] Specifically, when the application screen for expressing information keywords is smooth, the bitrate prediction data, encoding parameters, and sampling rate are used as positive samples to train the sampling rate prediction model; when the application screen for expressing information keywords is choppy or blurry, the bitrate prediction data, encoding parameters, and sampling rate are used as negative samples to train the sampling rate prediction model.

[0205] In other words, for positive samples, the bitrate prediction data and encoding parameters are input into the sampling rate prediction model, and the predicted sampling rate is output. The sampling rate prediction model is trained with the goal of reducing the difference between the predicted sampling rate and the actual sampling rate. For negative samples, the bitrate prediction data and encoding parameters are input into the sampling rate prediction model, and the predicted sampling rate is output. The sampling rate prediction model is trained with the goal of increasing the difference between the predicted sampling rate and the actual sampling rate.

[0206] By acquiring screen feedback information as data labels for sample data, positive and negative sample data are generated based on the screen feedback information, and the sampling rate prediction model is further trained, which improves the prediction accuracy of the sampling rate prediction model. Furthermore, by combining the screen feedback information of users during the actual screen display process, the sampling rate predicted by the sampling rate prediction model can better match the user's expected results, thereby improving the adaptability of cloud applications.

[0207] Indicative, Figure 8 This is a schematic diagram illustrating a screen feedback process provided in an exemplary embodiment of this application. For example... Figure 8 As shown, after the cloud server samples the application screen according to the sampling rate, it sends the encoded application screen to the terminal. The terminal decodes and displays the application screen. When the user needs feedback on the screen status, they can perform an operation to display the screen feedback area 800. The screen feedback area 800 includes at least two candidate feedback items 810. Optionally, the screen feedback area 800 also includes a feedback content filling area 820. The user can select the information to be fed back from the at least two candidate feedback items 810, or they can enter the information to be fed back in the content filling area 820. For example, if the user fills in the content "The screen is blurry frequently" in the content filling area 820, the cloud server extracts the keyword information "blurry" and uses the sampling rate, bitrate prediction data, and encoding parameters as negative samples to train the sampling rate prediction model.

[0208] In summary, the method provided in this embodiment, after determining the first encoding level corresponding to the resolution configuration, if a network environment deteriorates leading to a decrease in bitrate, obtains the sampling rate corresponding to the bitrate and encodes the application screen based on the sampling rate and the first encoding resolution of the first encoding level. This avoids the problem of the encoder needing to compress the screen more efficiently when the bitrate decreases while the resolution remains unchanged, resulting in blocky blurring and distortion of details and colors in the application screen. This improves the adaptability between bitrate and resolution, and enhances the smoothness and performance of the image. Furthermore, by predicting the sampling rate corresponding to the bitrate prediction data using a sampling rate prediction model and using the historical sampling rate selection effect as a benchmark, the accuracy of determining the sampling rate is improved, and the adaptability between the sampling rate and the bitrate prediction data is enhanced.

[0209] Figure 9 This is a schematic diagram of the overall cloud gaming operation process provided in an exemplary embodiment of this application, such as... Figure 9 As shown, the overall process involves a cloud server and a client. The cloud server and client transmit encoded application screen data via RTC.

[0210] Taking cloud gaming as an example, the overall process includes the following steps.

[0211] Step 901: The cloud gaming client obtains the encoding bitrate range and downsampling rate configuration corresponding to different resolution levels.

[0212] In other words, after running the cloud gaming client, the cloud gaming client obtains multiple different resolution levels and the corresponding encoding bitrate range and downsampling rate configuration for each resolution level.

[0213] In some embodiments, the cloud gaming client directly configures multiple different resolution levels and the corresponding bitrate ranges and downsampling rates for each resolution level; alternatively, the cloud gaming client obtains multiple different resolution levels and the corresponding bitrate ranges and downsampling rates from the cloud gaming configuration server. In some embodiments, the configuration server and the cloud server are implemented as the same server, or they are implemented as different servers.

[0214] Optionally, the cloud gaming client and the cloud server synchronously obtain multiple different resolution levels, as well as the corresponding encoding bitrate range and downsampling rate, from the configuration server.

[0215] In some embodiments, the configuration server is configured with at least one resolution level, bitrate range, and downsampling rate corresponding to a cloud application, including the resolution level, bitrate, and downsampling rate corresponding to cloud gaming in this embodiment.

[0216] As an illustration, the resolution level, bitrate, and downsampling rate obtained by the cloud gaming client are as follows:

[0217] Ultra HD: Bitrate range 7-10Mbps, downsampling rate 1.0x, encoding resolution 1080P;

[0218] High Definition: Bitrate range 4-7Mbps, downsampling rate 1.5x, encoding resolution 720P;

[0219] Standard Definition: Bitrate range 1-4Mbps, downsampling rate 2.0x, encoding resolution 540P.

[0220] Since the downsampling rate of the ultra-high definition level is 1x, that is, based on the ultra-high definition game screen, if the encoding bitrate drops below 7Mbps, the downsampling rate is 1.5x or 2x to downsample the game screen.

[0221] The bitrate range, downsampling rate, and resolution in the aforementioned resolution levels are preset.

[0222] Step 902: The cloud gaming client sends the resolution level and encoding parameters to the cloud server.

[0223] In some embodiments, the cloud gaming client receives a resolution setting operation and sends the set resolution level and encoding parameters to the cloud server as the resolution setting result of the cloud gaming client.

[0224] In some embodiments, the resolution setting of the cloud gaming client is the upper limit of the resolution of the game screen provided by the cloud server to the cloud gaming client.

[0225] Encoding parameters include encoding format and other encoding parameters.

[0226] To illustrate, when a cloud gaming client receives a selection of an ultra-high definition (UHD) tier, the client sends the UHD tier identifier to the cloud server. In addition, the client also sends the encoding format (e.g., H264 or H265), frame rate, and the resolution and bitrate corresponding to the UHD tier to the cloud server.

[0227] Step 903: The cloud server updates the starting target bitrate and downsampling rate corresponding to the tier.

[0228] Optionally, the cloud server determines the resolution tier based on the tier identifier indicated by the cloud gaming client, and obtains the starting target bitrate and downsampling rate corresponding to the resolution tier.

[0229] The initial target bitrate can be implemented as the minimum bitrate of the resolution level, or as the maximum bitrate of the resolution level, or as the intermediate bitrate of the bitrate range of the resolution level. This application does not limit the selection method of the initial target bitrate in the embodiments.

[0230] The downsampling rate refers to the downsampling rate corresponding to the resolution level. For example, the downsampling rate corresponding to the resolution level of 1080P is 1.0 times.

[0231] Step 904: The cloud server dynamically updates the encoder bitrate based on network bandwidth estimates.

[0232] The bitrate must not exceed the upper limit of the corresponding resolution level.

[0233] In other words, when the network bandwidth between the cloud server and the cloud gaming client changes, the encoder's bitrate is updated based on the estimated network bandwidth. For example, when adjusting the bitrate for the first time, the target starting bitrate is used as a baseline. If network bandwidth conditions worsen, the bitrate is reduced by a preset adjustment increment, such as a 1Mbps decrease to lower the target starting bitrate. If network bandwidth conditions are good, the bitrate is increased by the preset adjustment increment until it reaches the highest bitrate within the bitrate range corresponding to the resolution level.

[0234] Step 905: The cloud server selects and updates the downsampling rate based on the sliding bandwidth statistics adaptive multi-resolution switching algorithm.

[0235] At least two coded bitrates, determined based on network bandwidth, are acquired within a sliding window. Bitrate change information is acquired to express the trend of bitrate change. When the bitrate change information indicates a decrease in the coded bitrate, at least two bitrate data points are sampled within a first time window. When the bitrate change information indicates an increase in the coded bitrate, at least two bitrate data points are sampled within a second time window. The first time window is shorter than the second time window.

[0236] The bitrate prediction data is obtained based on the weighted average of at least the bitrate data. When the bitrate prediction data decreases to the minimum value of the bitrate range corresponding to the current resolution level, the downsampling rate is updated. For example, the next lower resolution downsampling rate corresponding to the current resolution level is selected; that is, the downsampling rate corresponding to the next resolution level is obtained. For illustration, if the current resolution level is 1080P, then the downsampling rate corresponding to 720P is obtained.

[0237] Step 906: The video acquisition and encoding module updates the downsampling rate and the video encoding resolution.

[0238] The video capture and encoding module generates the video footage corresponding to the cloud game and transmits it via an encoder. After the RTC transmission module in the cloud server updates the downsampling rate, it sends the downsampling rate to the video capture and encoding module. The video capture and encoding module then downsamples the game footage based on the downsampling rate to adjust the encoding resolution and encodes the downsampled data to obtain encoded data. This encoded data is then sent to the RTC transmission module, which transmits it to the cloud game client using RTC technology.

[0239] Step 907: The video capture and encoding module updates the bitrate.

[0240] After the RTC transmission module dynamically updates the encoder bitrate based on network bandwidth estimation, it transmits the dynamically updated bitrate to the video capture and encoding module. The video capture and encoding module then updates the bitrate and encodes the game screen according to the updated bitrate.

[0241] Step 908: The cloud gaming client decoder updates the output width and height information.

[0242] When the cloud server provides encoded data to the cloud gaming client, it sends the downsampling rate of the game screen corresponding to the encoded data to the cloud gaming client. The cloud gaming client then outputs the updated width and height information corresponding to the resolution based on the downsampling rate and the resolution level.

[0243] Alternatively, when the cloud server provides encoded data to the cloud gaming client, it sends the game screen corresponding to the encoded data at a downsampled resolution to the cloud gaming client, so that the cloud gaming client can decode the game screen according to the downsampled resolution.

[0244] Step 909: Update the super-resolution and other parameter configurations in the cloud gaming client rendering module.

[0245] Since the resolution is obtained by downsampling, the cloud gaming client is also configured with super-resolution parameters, which are used to upsample the decoded data to improve the resolution of the game screen.

[0246] The application process of super-resolution parameters includes: 1. Selecting a super-resolution algorithm: Choose a suitable super-resolution algorithm according to the requirements, such as Super-Resolution CNN (SRCNN) or Very Deep SuperResolution (VDSR). 2. Preparing training data: Collect and prepare the dataset for training, which usually includes low-resolution and high-resolution image pairs. 3. Setting parameters: Set the parameters of the super-resolution algorithm, such as kernel size, learning rate, and number of iterations. 4. Training the model: Train the super-resolution model using the training data and adjust the parameters to optimize performance. 5. Applying the super-resolution model: Apply the trained super-resolution model to the real-world scenario. In this embodiment, after decoding the game screen, the game screen is upsampled and enhanced using the super-resolution model.

[0247] In summary, the method provided in this embodiment, in a cloud gaming scenario, after determining the resolution level, if the network environment deteriorates and the bitrate decreases, the sampling rate corresponding to the bitrate is obtained, and the game screen is encoded based on the sampling rate and the encoding resolution of the resolution level. This avoids the problem of the encoder needing to compress the screen more efficiently when the bitrate decreases but the resolution remains unchanged, resulting in blocky blurring of the game screen and distortion of details and colors, thus improving the smoothness and performance of the screen.

[0248] Indicative Figure 10 This is a schematic diagram of the downsampling rate curve provided in an exemplary embodiment of this application, as shown below. Figure 10 As shown in curve 1010, the average downsampling rate is approximately 1.23 times.

[0249] The 50th percentile downsampling rate, as shown in curve 1020, is approximately 1.1 times, indicating that half of the cloud gaming clients maintain a relatively constant resolution during runtime.

[0250] The 75th percentile downsampling rate, as shown in curve 1030, is approximately 1.4 times, indicating that most cloud gaming clients experience a resolution reduction of less than 1.5 times during runtime.

[0251] Figure 11 This is a schematic diagram of the bandwidth curve provided in an exemplary embodiment of this application, as shown below. Figure 11 As shown in Figure 1110, the average bandwidth usage of a cloud gaming client without the downsampling rate resolution switching function is 6.3 Mbps; while the average bandwidth usage of a cloud gaming client with the downsampling rate resolution switching function enabled is 5.5 Mbps, as shown in Figure 1120.

[0252] Figure 12 This is a schematic diagram of the stuttering rate curve provided in an exemplary embodiment of this application, such as... Figure 12 As shown in curve 1200, after enabling the downsampling rate switching resolution function at time node A, the stuttering rate of the cloud gaming client shows a downward trend.

[0253] Figure 13 This is a schematic diagram of the overall cloud conferencing process provided in an exemplary embodiment of this application, such as... Figure 13 As shown, the overall process involves a cloud server and a client. The cloud server and client transmit encoded application screen data via RTC.

[0254] Taking the operation of cloud conferencing as an example, the overall process includes the following steps.

[0255] Step 1301: The cloud conferencing client obtains the encoding bitrate range and downsampling rate configuration corresponding to different resolution levels.

[0256] In other words, after running the cloud conferencing client, the cloud conferencing client obtains multiple different resolution levels as well as the corresponding encoding bitrate range and downsampling rate configuration.

[0257] Optionally, the cloud conferencing client and the cloud server synchronously obtain multiple different resolution levels, as well as the corresponding encoding bitrate range and downsampling rate, from the configuration server.

[0258] In some embodiments, the configuration server is configured with at least one resolution level, bitrate range, and downsampling rate corresponding to a cloud application, including the resolution level, bitrate, and downsampling rate corresponding to cloud conferencing in this embodiment.

[0259] As an illustration, the resolution level, encoding bitrate, and downsampling rate obtained by the cloud conferencing client are as follows:

[0260] Ultra HD: Bitrate range 7-10Mbps, downsampling rate 1.0x, encoding resolution 1080P;

[0261] High Definition: Bitrate range 4-7Mbps, downsampling rate 1.5x, encoding resolution 720P;

[0262] Standard Definition: Bitrate range 1-4Mbps, downsampling rate 2.0x, encoding resolution 540P.

[0263] Since the downsampling rate of the ultra-high definition (UHD) is 1x, that is, based on the UHD meeting video, if the encoding bitrate drops below 7Mbps, the meeting video will be downsampled using a downsampling rate of 1.5x or 2x.

[0264] The bitrate range, downsampling rate, and resolution in the aforementioned resolution levels are preset.

[0265] Step 1302: The cloud conferencing client sends the resolution level and encoding parameters to the cloud server.

[0266] In some embodiments, the cloud conferencing client receives a resolution setting operation and sends the set resolution level and encoding parameters to the cloud server as the resolution setting result of the cloud conferencing client.

[0267] In some embodiments, the resolution setting of the cloud conferencing client is the upper limit of the resolution of the meeting screen provided by the cloud server to the cloud conferencing client.

[0268] In this embodiment, the cloud conferencing client refers to the client that receives the conference video feed in the cloud conferencing. The cloud conferencing also includes clients that output the conference video feed. The resolution setting of the client receiving the conference video feed is no higher than the resolution setting of the client outputting the conference video feed. In some embodiments, the client receiving the conference video feed can also be implemented as the client outputting the conference video feed. When multiple conference video feeds exist, if the account logged into the cloud conferencing client belongs to the management account in the cloud conferencing, the resolution setting operation is received on the cloud conferencing client. The resolution setting is applied to multiple cloud conferencing clients participating in the cloud conferencing.

[0269] Encoding parameters include encoding format and other encoding parameters.

[0270] As an illustration, when the cloud conferencing client receives a selection operation for the ultra-high definition (UHD) level, the cloud conferencing client sends the UHD level identifier to the cloud server. In addition, the cloud conferencing client also sends the encoding format, frame rate, resolution and bitrate corresponding to the UHD level to the cloud server.

[0271] Step 1303: The cloud server updates the starting target bitrate and downsampling rate corresponding to the meeting screen.

[0272] Optionally, the cloud server determines the resolution level based on the resolution identifier indicated by the cloud conferencing client, and obtains the starting target bitrate and downsampling rate corresponding to the resolution level.

[0273] The initial target bitrate can be implemented as the minimum bitrate of the resolution level, or as the maximum bitrate of the resolution level, or as the intermediate bitrate of the bitrate range of the resolution level. This application does not limit the selection method of the initial target bitrate in the embodiments.

[0274] The downsampling rate refers to the downsampling rate corresponding to the resolution level. For example, the downsampling rate corresponding to the resolution level of 1080P is 1.0 times.

[0275] Optionally, the cloud server obtains the first encoding level, which is the level used by the cloud conference to encode the conference screen. The first encoding level is the resolution level corresponding to the level identifier, including the first bitrate range and the first encoding resolution.

[0276] Step 1304: The cloud server dynamically updates the encoder bitrate based on network bandwidth estimation.

[0277] The bitrate must not exceed the upper limit of the corresponding resolution level.

[0278] In other words, when the network bandwidth between the cloud server and the cloud conferencing client changes, the encoder's bitrate is updated based on the estimated network bandwidth. For example, when adjusting the bitrate for the first time, the target starting bitrate is used as a baseline. If network bandwidth conditions worsen, the bitrate is reduced by a preset adjustment increment, such as a 1Mbps decrease to lower the target starting bitrate. If network bandwidth conditions are good, the bitrate is increased by the preset adjustment increment until it reaches the highest bitrate within the bitrate range corresponding to the resolution level.

[0279] Optionally, bitrate prediction data can be obtained, which is used to express the encoding bitrate of the conference screen by the cloud server.

[0280] Optionally, at least two bitrate data points are sampled within a time window; the weighted average of the at least two bitrate data points is obtained to get the bitrate prediction data.

[0281] Optionally, bitrate change information is acquired, which is used to express the trend of bitrate change; when the bitrate change information indicates that the encoded bitrate is decreasing, at least two bitrate data are sampled in a first time window; when the bitrate change information indicates that the encoded bitrate is increasing, at least two bitrate data are sampled in a second time window, wherein the first time window is smaller than the second time window.

[0282] Optionally, operation frequency information is obtained, which is used to express the frequency of operations received by the terminal for the cloud meeting within a historical time period. The operations for the cloud meeting include: meeting pause operation, mute operation, add member operation, etc.; a time window is determined based on bitrate change information and operation frequency, and at least two bitrate data are sampled in the time window.

[0283] Among them, the magnitude of the change expressed by the bit rate change information is negatively correlated with the size of the time window; and the operation frequency expressed by the operation frequency information is negatively correlated with the size of the time window.

[0284] Optionally, obtain the weight values ​​corresponding to at least two bitrate data points respectively; obtain the weighted sum of at least two bitrate data points based on the weight values ​​corresponding to the at least two bitrate data points respectively; obtain the weighted sum of the weight values ​​corresponding to the at least two bitrate data points respectively; obtain the ratio between the weighted sum and the weighted sum to obtain the bitrate prediction data.

[0285] Optionally, weight values ​​are assigned to at least two bitrate data points according to a preset weight distribution rule, wherein the weight values ​​and the duration between the acquisition time of the bitrate data and the current time are negatively correlated; or, bitrate-related information corresponding to the acquisition time of the i-th bitrate data point is obtained, wherein the bitrate-related information is information related to the encoded bitrate; the bitrate-related information and the i-th bitrate data point are input into the weight prediction model, and the weight value corresponding to the i-th bitrate data point is output, wherein the weight prediction model is a pre-trained machine learning model used to predict the weights of the bitrate data.

[0286] Step 1305: The cloud server selects and updates the downsampling rate based on the sliding bandwidth statistics adaptive multi-resolution switching algorithm.

[0287] At least two coded bitrates, determined based on network bandwidth, are acquired within a sliding window. Bitrate change information is acquired to express the trend of bitrate change. When the bitrate change information indicates a decrease in the coded bitrate, at least two bitrate data points are sampled within a first time window. When the bitrate change information indicates an increase in the coded bitrate, at least two bitrate data points are sampled within a second time window. The first time window is shorter than the second time window.

[0288] The bitrate prediction data is obtained based on the weighted average of at least the bitrate data. When the bitrate prediction data decreases to the minimum value of the bitrate range corresponding to the current resolution level, the downsampling rate is updated. For example, the next lower resolution downsampling rate corresponding to the current resolution level is selected; that is, the downsampling rate corresponding to the next resolution level is obtained. For illustration, if the current resolution level is 1080P, then the downsampling rate corresponding to 720P is obtained.

[0289] In response to the bitrate prediction data exceeding the first bitrate range, the sampling rate corresponding to the bitrate prediction data is obtained. The sampling rate is used to sample the conference screen to adapt to the bitrate prediction data.

[0290] Optionally, in response to the bitrate prediction data exceeding the first bitrate range, a second bitrate range in which the bitrate prediction data is located is obtained; the sampling rate corresponding to the second bitrate range is obtained, wherein the cloud server corresponding to the cloud meeting pre-stores at least two sampling rates corresponding to the bitrate ranges respectively.

[0291] Optionally, a second encoding bit corresponding to the second bitrate range is determined, and the cloud server pre-stores bitrate ranges corresponding to at least two encoding bits respectively;

[0292] Obtain the sampling rate corresponding to the second encoding level. The cloud server has pre-stored the sampling rates corresponding to at least two encoding levels.

[0293] Optionally, in response to the bitrate prediction data exceeding the first bitrate range, the bitrate prediction data is input into the sampling rate prediction model, and the sampling rate corresponding to the bitrate prediction data is output. The sampling rate prediction model is a pre-trained machine learning model for predicting the sampling rate.

[0294] Optionally, the encoding working parameters of the cloud server are obtained. These encoding working parameters are load parameters related to the encoding application screen. The bitrate prediction data and the encoding working parameters are input into the sampling rate prediction model, and the sampling rate corresponding to the bitrate prediction data is output.

[0295] Optionally, in response to receiving meeting feedback information sent by the cloud conferencing client, information keywords corresponding to the meeting feedback information are obtained. The meeting feedback information is used to express the display effect of the meeting screen on the terminal interface, and the information keywords are keywords extracted from the meeting feedback information to express the display effect.

[0296] Information keywords are used as labels for the bitrate prediction data and encoding parameters. The bitrate prediction data, encoding parameters, and sampling rate are stored as sample data in the sample database. The sample data is used to train the sampling rate prediction model.

[0297] Optionally, at least two candidate sampling rates are obtained in ascending order, wherein the first coding level corresponds to the first sampling rate among the at least two candidate sampling rates;

[0298] In response to the bitrate prediction data exceeding the first bitrate range, based on the comparison relationship between the bitrate prediction data and the first bitrate range, a second sampling rate corresponding to the bitrate prediction data is determined from at least two candidate sampling rates, wherein the second sampling rate is the sampling rate whose value is adjacent to the first sampling rate among the at least two candidate sampling rates.

[0299] Step 1306: The video acquisition and encoding module updates the downsampling rate and the video encoding resolution.

[0300] The video capture and encoding module generates the video frame corresponding to the cloud meeting and transmits it via an encoder. After the RTC transmission module in the cloud server updates the downsampling rate, it sends the downsampling rate to the video capture and encoding module. The video capture and encoding module then downsamples the meeting frame based on the downsampling rate to adjust the encoding resolution and encodes the downsampled data to obtain encoded data. This encoded data is then sent to the RTC transmission module, which transmits it to the cloud meeting client using RTC technology.

[0301] The conference screen is encoded based on the first encoding resolution and sampling rate to obtain encoded data, and the encoded data is sent to the cloud conferencing client.

[0302] Step 1307: The video capture and encoding module updates the bitrate.

[0303] After the RTC transmission module dynamically updates the encoder bitrate based on network bandwidth estimation, it transmits the dynamically updated bitrate to the video capture and encoding module. The video capture and encoding module then updates the bitrate and encodes the conference video according to the updated bitrate.

[0304] Step 1308: The cloud meeting client decoder updates the output width and height information.

[0305] When the cloud server provides encoded data to the cloud conferencing client, it sends the downsampling rate of the meeting screen corresponding to the encoded data to the cloud conferencing client. The cloud conferencing client then outputs the updated width and height information corresponding to the resolution based on the downsampling rate and the resolution level.

[0306] Alternatively, when the cloud server provides encoded data to the cloud conferencing client, it sends the corresponding meeting screen at a downsampled resolution to the cloud conferencing client, so that the cloud conferencing client can decode the meeting screen according to the downsampled resolution.

[0307] Step 1309: Update the super-resolution and other parameter configurations in the cloud meeting client rendering module.

[0308] Since the resolution is obtained through downsampling, the cloud conferencing client is also configured with a super-resolution parameter, which is used to upsample the decoded data to improve the resolution of the meeting screen.

[0309] In this embodiment of the application, after decoding the conference screen, the conference screen is upsampled and enhanced by a super-resolution model configured with super-resolution parameters.

[0310] In summary, the method provided in this embodiment, in a cloud conferencing scenario, after determining the resolution level, if the network environment deteriorates and the bitrate decreases, then the sampling rate corresponding to the bitrate is obtained, and the meeting screen is encoded based on the sampling rate and the encoding resolution of the resolution level. This avoids the problem of the encoder needing to compress the screen more efficiently when the resolution remains unchanged while the bitrate decreases, resulting in blocky blurring of the meeting screen and distortion of details and colors. It also avoids frequent disruptions to the meeting process and improves the smoothness and performance of the screen.

[0311] Figure 14 This is a schematic diagram of the overall cloud gaming operation process provided by another exemplary embodiment of this application, such as... Figure 14 As shown, the overall process involves a cloud server and a cloud gaming client. Specifically, the cloud server and the cloud gaming client transmit encoded application screen data via RTC.

[0312] Taking cloud gaming as an example, the overall process includes the following steps.

[0313] Step 1401: The cloud gaming client obtains the encoding bitrate range and downsampling rate configuration corresponding to different resolution levels.

[0314] In other words, after running the cloud gaming client, the cloud gaming client obtains multiple different resolution levels and the corresponding encoding bitrate range and downsampling rate configuration for each resolution level.

[0315] Optionally, the cloud gaming client and the cloud server synchronously obtain multiple different resolution levels, as well as the corresponding encoding bitrate range and downsampling rate, from the configuration server.

[0316] In some embodiments, the configuration server is configured with at least one resolution level, bitrate range, and downsampling rate corresponding to a cloud application, including the resolution level, bitrate, and downsampling rate corresponding to cloud gaming in this embodiment.

[0317] Step 1402: The cloud gaming client sends the resolution level and encoding parameters to the cloud server.

[0318] In some embodiments, the cloud gaming client receives a resolution setting operation and sends the set resolution level and encoding parameters to the cloud server as the resolution setting result of the cloud gaming client.

[0319] In some embodiments, the resolution setting of the cloud gaming client is the upper limit of the resolution of the game screen provided by the cloud server to the cloud gaming client.

[0320] In this embodiment, a cloud gaming client refers to a client that receives game footage in a cloud game. The cloud game also includes a client that outputs game footage. The resolution setting of the client receiving the game footage is no higher than that of the client outputting the game footage. In some embodiments, the client receiving the game footage can also be the client outputting the game footage. When multiple game footage streams exist, if the account logged into the cloud gaming client belongs to a management account in the cloud game, a resolution setting operation is received on the cloud gaming client. The resolution setting is applied to multiple cloud gaming clients participating in the cloud game.

[0321] Encoding parameters include encoding format and other encoding parameters.

[0322] To illustrate, when a cloud gaming client receives a selection of an ultra-high definition (UHD) tier, the client sends the UHD tier identifier to the cloud server. In addition, the client also sends the encoding format, frame rate, resolution, and bitrate corresponding to the UHD tier to the cloud server.

[0323] Step 1403: The cloud server updates the starting target bitrate and downsampling rate corresponding to the game screen.

[0324] Optionally, the cloud server determines the resolution tier based on the tier identifier indicated by the cloud gaming client, and obtains the starting target bitrate and downsampling rate corresponding to the resolution tier.

[0325] The initial target bitrate can be implemented as the minimum bitrate of the resolution level, or as the maximum bitrate of the resolution level, or as the intermediate bitrate of the bitrate range of the resolution level. This application does not limit the selection method of the initial target bitrate in the embodiments.

[0326] The downsampling rate refers to the downsampling rate corresponding to the resolution level. For example, the downsampling rate corresponding to the resolution level of 1080P is 1.0 times.

[0327] Optionally, the cloud server obtains the first encoding level, which is the level used for encoding the game screen of the cloud game. The first encoding level is the resolution level corresponding to the level identifier, including the first bitrate range and the first encoding resolution.

[0328] Step 1404: The cloud server dynamically updates the encoder bitrate based on network bandwidth estimation.

[0329] The bitrate must not exceed the upper limit of the corresponding resolution level.

[0330] In other words, when the network bandwidth between the cloud server and the cloud gaming client changes, the encoder's bitrate is updated based on the estimated network bandwidth. For example, when adjusting the bitrate for the first time, the target starting bitrate is used as a baseline. If network bandwidth conditions worsen, the bitrate is reduced by a preset adjustment increment, such as a 1Mbps decrease to lower the target starting bitrate. If network bandwidth conditions are good, the bitrate is increased by the preset adjustment increment until it reaches the highest bitrate within the bitrate range corresponding to the resolution level.

[0331] Optionally, bitrate prediction data can be obtained, which is used to express the bitrate of the game screen encoded by the cloud server.

[0332] Optionally, at least two bitrate data points are sampled within a time window; the weighted average of the at least two bitrate data points is obtained to get the bitrate prediction data.

[0333] Optionally, bitrate change information is acquired, which is used to express the trend of bitrate change; when the bitrate change information indicates that the encoded bitrate is decreasing, at least two bitrate data are sampled in a first time window; when the bitrate change information indicates that the encoded bitrate is increasing, at least two bitrate data are sampled in a second time window, wherein the first time window is smaller than the second time window.

[0334] Optionally, operation frequency information is obtained, which is used to express the frequency of operations received by the terminal for cloud gaming within a historical time period. The operations for cloud gaming include: attack operations, defense operations, skill release operations, item acquisition operations, etc.; a time window is determined based on bitrate change information and operation frequency, and at least two bitrate data are sampled within the time window.

[0335] Among them, the magnitude of the change expressed by the bit rate change information is negatively correlated with the size of the time window; and the operation frequency expressed by the operation frequency information is negatively correlated with the size of the time window.

[0336] Optionally, obtain the weight values ​​corresponding to at least two bitrate data points respectively; obtain the weighted sum of at least two bitrate data points based on the weight values ​​corresponding to the at least two bitrate data points respectively; obtain the weighted sum of the weight values ​​corresponding to the at least two bitrate data points respectively; obtain the ratio between the weighted sum and the weighted sum to obtain the bitrate prediction data.

[0337] Optionally, weight values ​​are assigned to at least two bitrate data points according to a preset weight distribution rule, wherein the weight values ​​and the duration between the acquisition time of the bitrate data and the current time are negatively correlated; or, bitrate-related information corresponding to the acquisition time of the i-th bitrate data point is obtained, wherein the bitrate-related information is information related to the encoded bitrate; the bitrate-related information and the i-th bitrate data point are input into the weight prediction model, and the weight value corresponding to the i-th bitrate data point is output, wherein the weight prediction model is a pre-trained machine learning model used to predict the weights of the bitrate data.

[0338] Step 1405: The cloud server selects and updates the downsampling rate based on the sliding bandwidth statistics adaptive multi-resolution switching algorithm.

[0339] At least two coded bitrates, determined based on network bandwidth, are acquired within a sliding window. Bitrate change information is acquired to express the trend of bitrate change. When the bitrate change information indicates a decrease in the coded bitrate, at least two bitrate data points are sampled within a first time window. When the bitrate change information indicates an increase in the coded bitrate, at least two bitrate data points are sampled within a second time window. The first time window is shorter than the second time window.

[0340] The bitrate prediction data is obtained based on the weighted average of at least the bitrate data. When the bitrate prediction data decreases to the minimum value of the bitrate range corresponding to the current resolution level, the downsampling rate is updated. For example, the next lower resolution downsampling rate corresponding to the current resolution level is selected; that is, the downsampling rate corresponding to the next resolution level is obtained. For illustration, if the current resolution level is 1080P, then the downsampling rate corresponding to 720P is obtained.

[0341] In response to the bitrate prediction data exceeding the first bitrate range, the sampling rate corresponding to the bitrate prediction data is obtained. The sampling rate is used to sample the game screen to adapt to the bitrate prediction data.

[0342] Optionally, in response to the bitrate prediction data exceeding the first bitrate range, a second bitrate range in which the bitrate prediction data is located is obtained; the sampling rate corresponding to the second bitrate range is obtained, wherein the cloud server corresponding to the cloud game pre-stores at least two sampling rates corresponding to the bitrate ranges respectively.

[0343] Optionally, a second encoding bit corresponding to the second bitrate range is determined, and the cloud server pre-stores bitrate ranges corresponding to at least two encoding bits respectively;

[0344] Obtain the sampling rate corresponding to the second encoding level. The cloud server has pre-stored the sampling rates corresponding to at least two encoding levels.

[0345] Optionally, in response to the bitrate prediction data exceeding the first bitrate range, the bitrate prediction data is input into the sampling rate prediction model, and the sampling rate corresponding to the bitrate prediction data is output. The sampling rate prediction model is a pre-trained machine learning model for predicting the sampling rate.

[0346] Optionally, the encoding working parameters of the cloud server are obtained. These encoding working parameters are load parameters related to the encoding application screen. The bitrate prediction data and the encoding working parameters are input into the sampling rate prediction model, and the sampling rate corresponding to the bitrate prediction data is output.

[0347] Optionally, in response to receiving game feedback information sent by the cloud gaming client, information keywords corresponding to the game feedback information are obtained. The game feedback information is used to express the display effect of the game screen on the terminal interface, and the information keywords are keywords extracted from the game feedback information to express the display effect.

[0348] Information keywords are used as labels for the bitrate prediction data and encoding parameters. The bitrate prediction data, encoding parameters, and sampling rate are stored as sample data in the sample database. The sample data is used to train the sampling rate prediction model.

[0349] Optionally, at least two candidate sampling rates are obtained in ascending order, wherein the first coding level corresponds to the first sampling rate among the at least two candidate sampling rates;

[0350] In response to the bitrate prediction data exceeding the first bitrate range, based on the comparison relationship between the bitrate prediction data and the first bitrate range, a second sampling rate corresponding to the bitrate prediction data is determined from at least two candidate sampling rates, wherein the second sampling rate is the sampling rate whose value is adjacent to the first sampling rate among the at least two candidate sampling rates.

[0351] Step 1406: The video acquisition and encoding module updates the downsampling rate and the video encoding resolution.

[0352] The video capture and encoding module generates the video footage corresponding to the cloud game and transmits it via an encoder. After the RTC transmission module in the cloud server updates the downsampling rate, it sends the downsampling rate to the video capture and encoding module. The video capture and encoding module then downsamples the game footage based on the downsampling rate to adjust the encoding resolution and encodes the downsampled data to obtain encoded data. This encoded data is then sent to the RTC transmission module, which transmits it to the cloud game client using RTC technology.

[0353] The game screen is encoded based on the first encoding resolution and sampling rate to obtain encoded data, and the encoded data is sent to the cloud gaming client.

[0354] Step 1407: The video capture and encoding module updates the bitrate.

[0355] After the RTC transmission module dynamically updates the encoder bitrate based on network bandwidth estimation, it transmits the dynamically updated bitrate to the video capture and encoding module. The video capture and encoding module then updates the bitrate and encodes the game screen according to the updated bitrate.

[0356] Step 1408: The cloud gaming client decoder updates the output width and height information.

[0357] When the cloud server provides encoded data to the cloud gaming client, it sends the downsampling rate of the game screen corresponding to the encoded data to the cloud gaming client. The cloud gaming client then outputs the updated width and height information corresponding to the resolution based on the downsampling rate and the resolution level.

[0358] Alternatively, when the cloud server provides encoded data to the cloud gaming client, it sends the game screen corresponding to the encoded data at a downsampled resolution to the cloud gaming client, so that the cloud gaming client can decode the game screen according to the downsampled resolution.

[0359] Step 1409: Update the super-resolution and other parameter configurations in the cloud gaming client rendering module.

[0360] Since the resolution is obtained by downsampling, the cloud gaming client is also configured with super-resolution parameters, which are used to upsample the decoded data to improve the resolution of the game screen.

[0361] In this embodiment of the application, after decoding the game screen, the game screen is upsampled and enhanced by a super-resolution model configured with super-resolution parameters.

[0362] In summary, the method provided in this embodiment, in a cloud gaming scenario, after determining the resolution level, if the network environment deteriorates and the bitrate decreases, the sampling rate corresponding to the bitrate is obtained, and the game screen is encoded based on the sampling rate and the encoding resolution of the resolution level. This avoids the problem of the encoder needing to compress the screen more efficiently when the bitrate decreases but the resolution remains unchanged, which would cause blocky blurring of the game screen and distortion of details and colors. It also avoids the game process being frequently affected, improving the smoothness and performance of the screen.

[0363] Figure 15 This is a structural block diagram of a screen encoding device for a cloud application provided in an exemplary embodiment of this application, such as... Figure 15 As shown, the device includes:

[0364] The acquisition module 1510 is used to acquire the first encoding level, which is the level used by the application screen of the encoding cloud application, and the first encoding level corresponds to the first bit rate range and the first encoding resolution.

[0365] The acquisition module 1510 is also used to acquire bitrate prediction data, which is used to express the encoding bitrate of the application screen by the cloud server.

[0366] The acquisition module 1510 is further configured to, in response to the bitrate prediction data exceeding the first bitrate range, acquire the sampling rate corresponding to the bitrate prediction data, wherein the sampling rate is used to sample the application screen to adapt to the bitrate prediction data.

[0367] Encoding module 1520 is used to encode the application screen based on the first encoding resolution and the sampling rate corresponding to the bit rate prediction data to obtain encoded data;

[0368] The sending module 1530 is used to send the encoded data to the terminal.

[0369] In an optional embodiment, the acquisition module 1510 is further configured to, in response to the bitrate prediction data exceeding the first bitrate range, acquire the second bitrate range in which the bitrate prediction data is located; and acquire the sampling rate corresponding to the second bitrate range, wherein the cloud server pre-stores at least two sampling rates corresponding to the bitrate ranges respectively.

[0370] In an optional embodiment, such as Figure 16 As shown, the acquisition module 1510 includes:

[0371] The determining unit 1511 is used to determine the second encoding level corresponding to the second bitrate range, wherein the cloud server pre-stores at least two bitrate ranges corresponding to the encoding levels respectively.

[0372] The acquisition unit 1512 is used to acquire the sampling rate corresponding to the second encoding level, wherein the cloud server pre-stores the sampling rates corresponding to at least two encoding levels respectively.

[0373] In an optional embodiment, the acquisition module 1510 is further configured to, in response to the bitrate prediction data exceeding the first bitrate range, input the bitrate prediction data into a sampling rate prediction model and output the sampling rate corresponding to the bitrate prediction data, wherein the sampling rate prediction model is a pre-trained machine learning model for predicting the sampling rate.

[0374] In an optional embodiment, the acquisition module 1510 is further configured to acquire the encoding working parameters of the cloud server, wherein the encoding working parameters are load parameters related to encoding the application screen;

[0375] The acquisition module 1510 is further configured to input the bitrate prediction data and the encoding working parameters into the sampling rate prediction model, and output the sampling rate corresponding to the bitrate prediction data.

[0376] In an optional embodiment, the acquisition module 1510 is further configured to, in response to receiving screen feedback information sent by the terminal, acquire information keywords corresponding to the screen feedback information, wherein the screen feedback information is used to express the display effect of the application screen on the interface of the terminal, and the information keywords are keywords extracted from the screen feedback information to express the display effect;

[0377] The acquisition module 1510 is further configured to use the information keywords as tags corresponding to the bitrate prediction data and the encoding working parameters, and to store the bitrate prediction data, the encoding working parameters and the sampling rate as sample data in a sample database, wherein the sample data is used to train the sampling rate prediction model.

[0378] In an optional embodiment, the acquisition module 1510 is further configured to acquire at least two candidate sampling rates arranged in ascending order, wherein the first encoding level corresponds to the first sampling rate among the at least two candidate sampling rates;

[0379] The acquisition module 1510 is further configured to, in response to the bitrate prediction data exceeding the first bitrate range, determine a second sampling rate corresponding to the bitrate prediction data from the at least two candidate sampling rates based on the comparison relationship between the bitrate prediction data and the first bitrate range, wherein the second sampling rate is the sampling rate among the at least two candidate sampling rates that is adjacent to the value of the first sampling rate.

[0380] In an optional embodiment, the encoding module 1520 is further configured to encode the application screen based on the first encoding resolution and the sampling rate when the duration between the most recent resolution adjustment time and the current time within the historical time period reaches a preset duration threshold.

[0381] In an optional embodiment, the acquisition module 1510 is further configured to sample at least two bitrate data points within a time window; obtain a weighted average of the at least two bitrate data points to obtain the bitrate prediction data.

[0382] In an optional embodiment, the acquisition module 1510 is further configured to acquire bitrate change information, the bitrate change information being used to express the trend of bitrate change;

[0383] The acquisition module 1510 is further configured to sample the at least two bitrate data in a first time window when the bitrate change information indicates that the encoded bitrate is decreasing; and to sample the at least two bitrate data in a second time window when the bitrate change information indicates that the encoded bitrate is increasing, wherein the first time window is smaller than the second time window.

[0384] In an optional embodiment, the acquisition module 1510 is further configured to acquire bitrate change information, which is used to express the trend of bitrate change; and acquire operation frequency information, which is used to express the frequency of operations received by the terminal for the cloud application within a historical time period.

[0385] The acquisition module 1510 is further configured to determine the time window based on the bit rate change information and the operation frequency, and to sample the at least two bit rate data within the time window;

[0386] The rate change information is negatively correlated with the size of the time window; and the operation frequency information is negatively correlated with the size of the time window.

[0387] In an optional embodiment, the acquisition module 1510 is further configured to acquire the weight values ​​corresponding to the at least two bitrate data respectively; and acquire the weighted sum of the at least two bitrate data based on the weight values ​​corresponding to the at least two bitrate data respectively;

[0388] The acquisition module 1510 is further configured to acquire the weight and value of the weight values ​​corresponding to the at least two bitrate data respectively;

[0389] The acquisition module 1510 is further configured to acquire the ratio between the weighted sum and the weighted sum, thereby obtaining the bitrate prediction data.

[0390] In an optional embodiment, the acquisition module 1510 is further configured to assign weight values ​​to the at least two bitrate data points according to a preset weight distribution rule, wherein the weight values ​​and the duration between the acquisition time of the bitrate data and the current time are negatively correlated; or,

[0391] The acquisition module 1510 is further configured to acquire bitrate-related information corresponding to the acquisition time of the i-th bitrate data, wherein the bitrate-related information is information related to the encoded bitrate; input the bitrate-related information and the i-th bitrate data into the weight prediction model, and output the weight value corresponding to the i-th bitrate data, wherein the weight prediction model is a machine learning model pre-trained for predicting the weight of the bitrate data.

[0392] In an optional embodiment, the acquisition module 1510 is further configured to acquire screen data of the application screen corresponding to the first encoding resolution; sample the screen data according to the sampling rate to obtain sampled data; and encode the sampled data to obtain the encoded data.

[0393] In summary, the apparatus provided in this application, after determining the first encoding level corresponding to the resolution configuration, if the network environment deteriorates and the bitrate decreases, obtains the sampling rate corresponding to the bitrate, and then samples the application screen using the sampling rate, so that the sampled application screen can adapt to the decreased bitrate. The application screen is then encoded based on the sampling rate and the first encoding resolution of the first encoding level. By setting the sampling rate and using the sampling rate and the first encoding resolution to encode the application screen, the problem of the encoder needing to compress the screen more efficiently when the bitrate decreases but the resolution remains unchanged, which would cause blocky blurring and distortion of details and colors in the application screen, is avoided. This improves the adaptability between bitrate and resolution, and enhances the smoothness and performance of the screen.

[0394] It should be noted that the cloud application screen encoding device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the cloud application screen encoding device and the cloud application screen encoding method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0395] Figure 17 This illustration shows a structural block diagram of a computer device 1700 provided in an exemplary embodiment of this application. The computer device 1700 may be a smartphone, tablet computer, Moving Picture Experts Group Audio Layer III (MP3) player, Moving Picture Experts Group Audio Layer IV (MP4) player, laptop computer, or desktop computer. The computer device 1700 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0396] Typically, computer device 1700 includes a processor 1701 and a memory 1702.

[0397] Processor 1701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1701 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 1701 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1701 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1701 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0398] The memory 1702 may include one or more computer-readable storage media, which may be non-transitory. The memory 1702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1702 is used to store at least one instruction, which is executed by the processor 1701 to implement the screen encoding method for cloud applications provided in the method embodiments of this application.

[0399] In some embodiments, the computer device 1700 also includes other components 1703, the type and number of which can be selected based on the functional needs of the computer device 1700. Those skilled in the art will understand that... Figure 17 The structure shown does not constitute a limitation on the computer device 1700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0400] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0401] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the screen encoding method for cloud applications as described in any of the above embodiments of this application.

[0402] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the screen encoding method for cloud applications as described in any of the above embodiments of this application.

[0403] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the screen encoding method for cloud applications described in any of the above embodiments.

[0404] It should be noted that the collection and processing of relevant data (such as dialogue content) in this application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0405] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0406] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A screen encoding method for cloud applications, characterized in that, The method includes: Obtain the first encoding level, which is the level used by the application screen of the encoding cloud application, and the first encoding level corresponds to the first bitrate range and the first encoding resolution; Obtain bitrate prediction data, which is used to express the encoding bitrate of the application screen by the cloud server; In response to the bitrate prediction data exceeding the first bitrate range, the sampling rate corresponding to the bitrate prediction data is obtained, and the sampling rate is used to sample the application screen to adapt to the bitrate prediction data. The application screen is encoded based on the first encoding resolution and the sampling rate corresponding to the bitrate prediction data to obtain encoded data, and the encoded data is sent to the terminal.

2. The method according to claim 1, characterized in that, The step of obtaining the sampling rate corresponding to the bitrate prediction data in response to the bitrate prediction data exceeding the first bitrate range includes: In response to the bitrate prediction data exceeding the first bitrate range, a second bitrate range in which the bitrate prediction data is located is obtained; Obtain the sampling rate corresponding to the second bitrate range, wherein the cloud server pre-stores at least two sampling rates corresponding to different bitrate ranges.

3. The method according to claim 2, characterized in that, The step of obtaining the sampling rate corresponding to the second bit rate range includes: Determine the second encoding level corresponding to the second bitrate range; the cloud server pre-stores at least two bitrate ranges corresponding to each encoding level. The sampling rate corresponding to the second encoding level is obtained, and the cloud server pre-stores the sampling rates corresponding to at least two encoding levels respectively.

4. The method according to claim 1, characterized in that, The step of obtaining the sampling rate corresponding to the bitrate prediction data in response to the bitrate prediction data exceeding the first bitrate range includes: In response to the bitrate prediction data exceeding the first bitrate range, the bitrate prediction data is input into the sampling rate prediction model, and the sampling rate corresponding to the bitrate prediction data is output. The sampling rate prediction model is a pre-trained machine learning model for predicting the sampling rate.

5. The method according to claim 4, characterized in that, The method further includes: Obtain the encoding parameters of the cloud server, wherein the encoding parameters are load parameters related to encoding the application screen; The step of inputting the bitrate prediction data into the sampling rate prediction model and outputting the sampling rate corresponding to the bitrate prediction data includes: The bitrate prediction data and the encoding working parameters are input into the sampling rate prediction model, and the sampling rate corresponding to the bitrate prediction data is output.

6. The method according to claim 5, characterized in that, The method further includes: In response to receiving screen feedback information sent by the terminal, information keywords corresponding to the screen feedback information are obtained. The screen feedback information is used to express the display effect of the application screen on the interface of the terminal, and the information keywords are keywords extracted from the screen feedback information to express the display effect. The information keywords are used as tags corresponding to the bitrate prediction data and the encoding working parameters. The bitrate prediction data, the encoding working parameters, and the sampling rate are stored as sample data in the sample database. The sample data is used to train the sampling rate prediction model.

7. The method according to claim 1, characterized in that, The step of obtaining the sampling rate corresponding to the bitrate prediction data in response to the bitrate prediction data exceeding the first bitrate range includes: Obtain at least two candidate sampling rates arranged in ascending order, wherein the first encoding level corresponds to the first sampling rate among the at least two candidate sampling rates; In response to the bitrate prediction data exceeding the first bitrate range, based on the comparison relationship between the bitrate prediction data and the first bitrate range, a second sampling rate corresponding to the bitrate prediction data is determined from the at least two candidate sampling rates, wherein the second sampling rate is the sampling rate whose value is adjacent to the first sampling rate among the at least two candidate sampling rates.

8. The method according to any one of claims 1 to 7, characterized in that, The encoding of the application screen based on the first encoding resolution and the sampling rate corresponding to the bitrate prediction data includes: If the duration between the most recent resolution adjustment and the current time within the historical time period reaches a preset duration threshold, the application screen is encoded based on the first encoding resolution and the sampling rate corresponding to the bitrate prediction data.

9. The method according to any one of claims 1 to 7, characterized in that, The acquisition of bitrate prediction data includes: Sample at least two bitrate data points within a time window; Obtain the weighted average of the at least two bitrate data to obtain the bitrate prediction data.

10. The method according to claim 9, characterized in that, The sampling of at least two bitrate data points within a time window includes: Obtain bitrate change information, which is used to express the trend of bitrate change; When the bitrate change information indicates that the encoded bitrate is decreasing, the at least two bitrate data are sampled in a first time window; When the bitrate change information indicates that the encoded bitrate has increased, the at least two bitrate data are sampled in a second time window, wherein the first time window is smaller than the second time window.

11. The method according to claim 9, characterized in that, The sampling of at least two bitrate data points within a time window includes: Obtain bitrate change information, which is used to express the trend of bitrate change; Obtain operation frequency information, which is used to express the frequency of operations received by the terminal for the cloud application within a historical time period; The time window is determined based on the bitrate change information and the operation frequency, and the at least two bitrate data are sampled within the time window. The rate change information is negatively correlated with the size of the time window; and the operation frequency information is negatively correlated with the size of the time window.

12. The method according to claim 9, characterized in that, The step of obtaining the weighted average of the at least two bitrate data to obtain the bitrate prediction data includes: Obtain the weight values ​​corresponding to the at least two bitrate data points respectively; The weighted sum of the at least two bitrate data is obtained based on the weight values ​​corresponding to the at least two bitrate data respectively. Obtain the weights and values ​​of the weights corresponding to the at least two bitrate data points respectively; The ratio between the weighted sum and the weighted sum is obtained to get the bitrate prediction data.

13. The method according to claim 12, characterized in that, The step of obtaining the weight values ​​corresponding to the at least two bitrate data points includes: The weight values ​​are assigned to the at least two bitrate data points according to a preset weight distribution rule, wherein the weight values ​​and the duration between the acquisition time and the current time of the bitrate data are negatively correlated; or... Obtain the bitrate-related information corresponding to the acquisition time of the i-th bitrate data, wherein the bitrate-related information is information related to the encoded bitrate; input the bitrate-related information and the i-th bitrate data into the weight prediction model, and output the weight value corresponding to the i-th bitrate data, wherein the weight prediction model is a pre-trained machine learning model for predicting the weights of bitrate data.

14. The method according to any one of claims 1 to 7, characterized in that, The process of encoding the application screen based on the first encoding resolution and the sampling rate corresponding to the bitrate prediction data to obtain encoded data includes: Obtain the screen data of the application screen corresponding to the first encoding resolution; The image data is sampled according to the sampling rate to obtain sampled data; The sampled data is encoded to obtain the encoded data.

15. A screen encoding device for cloud applications, characterized in that, The device includes: The acquisition module is used to acquire the first encoding level, which is the level used by the application screen of the encoding cloud application. The first encoding level corresponds to the first bitrate range and the first encoding resolution. The acquisition module is also used to acquire bitrate prediction data, which is used to express the encoding bitrate of the application screen by the cloud server. The acquisition module is further configured to, in response to the bitrate prediction data exceeding the first bitrate range, acquire the sampling rate corresponding to the bitrate prediction data, wherein the sampling rate is used to sample the application screen to adapt to the bitrate prediction data; The encoding module is used to encode the application screen based on the first encoding resolution and the sampling rate corresponding to the bit rate prediction data to obtain encoded data; The sending module is used to send the encoded data to the terminal.

16. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the screen encoding method for cloud applications as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the screen encoding method for cloud applications as described in any one of claims 1 to 14.

18. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the screen encoding method for a cloud application as described in any one of claims 1 to 14.

Images