Data processing method and device, electronic equipment and medium

Abstract

The embodiment of the application discloses a data processing method and device, electronic equipment and medium, which can be applied to the technical field of data processing. The method comprises the following steps: receiving frame storage information sent by a client, wherein the frame storage information is used for indicating a target frame storage quantity; obtaining a first encoding frame rate; when the target frame storage quantity meets a frame rate adjustment condition, adjusting the first encoding frame rate based on the target frame storage quantity to obtain a second encoding frame rate; performing video encoding processing on an application picture based on the second encoding frame rate to obtain a target video frame, and sending the target video frame to the client. By adopting the embodiment of the application, the encoding frame rate for video frame encoding can be adaptively adjusted based on the frame storage information of the client, which helps to reduce the delay feeling of picture display. The embodiment of the application can also be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation, auxiliary driving and intelligent household appliances.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to data processing methods, apparatus, electronic devices, and media. Background Technology

[0002] In some scenarios, such as real-time video applications like cloud gaming, servers can send encoded video frames to clients. The clients then decode these frames and display the resulting application screen. Generally, the higher the frame rate of the video frames sent by the server, the smoother the user experience. However, if the terminal device running the client has poor performance (e.g., poor video frame decoding capabilities), it may not be able to decode the video frames sent by the server in a timely manner, resulting in increased perceived latency for the user. Therefore, reducing this perceived latency is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This application provides a data processing method, apparatus, electronic device, and medium that can adaptively adjust the encoding frame rate of video frames based on the client's frame information, thereby helping to reduce the perceived latency in screen display.

[0004] On one hand, embodiments of this application disclose a data processing method, the method comprising:

[0005] The system receives frame-stacking information sent by the client, which indicates the target number of frame-stacking frames. The target number of frame-stacking frames is the difference between the number of video frames received by the client and the number of video frames decoded within a preset time range.

[0006] Obtain a first encoding frame rate, which is used to indicate the encoding frame rate of the video frames corresponding to the target number of frames;

[0007] When the target number of frames meets the frame rate adjustment condition, the first encoding frame rate is adjusted based on the target number of frames to obtain the second encoding frame rate;

[0008] The application screen is video encoded based on the second encoding frame rate to obtain the target video frame, and the target video frame is sent to the client.

[0009] On the one hand, embodiments of this application disclose another data processing method, the method comprising:

[0010] The number of video frames received and the number of video frames decoded within a preset time range are obtained; wherein, the video frames received within the preset time range are video frames obtained by the server performing video encoding processing based on a first encoding frame rate.

[0011] The frame-stacking information is determined based on the difference between the number of video frames received and the number of video frames decoded within the preset time range, and the frame-stacking information is sent to the server.

[0012] The system receives video frames sent by the server, which are obtained by video encoding based on a second encoding frame rate. It then performs video decoding on the received video frames to obtain the corresponding application screen and displays the application screen obtained by video decoding. The second encoding frame rate is obtained by the server adjusting the first encoding frame rate based on the frame information.

[0013] On one hand, embodiments of this application disclose a data processing apparatus, the apparatus comprising:

[0014] The receiving unit is used to receive frame-stacking information sent by the client. The frame-stacking information is used to indicate the target number of frame-stacking, which is the difference between the number of video frames received by the client and the number of video frames decoded within a preset time range.

[0015] An acquisition unit is used to acquire a first encoding frame rate, wherein the first encoding frame rate is used to indicate the encoding frame rate of the video frames corresponding to the target number of frames;

[0016] The processing unit is configured to adjust the first encoded frame rate based on the target number of frames when the target number of frames meets the frame rate adjustment condition, so as to obtain a second encoded frame rate.

[0017] The processing unit is further configured to perform video encoding processing on the application screen based on the second encoding frame rate to obtain a target video frame, and send the target video frame to the client.

[0018] On one hand, embodiments of this application disclose another data processing apparatus, the apparatus comprising:

[0019] The acquisition unit is used to acquire the number of video frames received and the number of video frames decoded within a preset time range; wherein, the video frames received within the preset time range are video frames obtained by the server performing video encoding processing based on a first encoding frame rate;

[0020] The processing unit is configured to determine the frame-stacking information based on the difference between the number of video frames received and the number of video frames decoded within the preset time range, and send the frame-stacking information to the server.

[0021] The receiving unit is configured to receive video frames sent by the server, which are obtained by video encoding processing based on a second encoding frame rate; perform video decoding processing on the received video frames to obtain the corresponding application screen; and display the application screen obtained by video decoding processing; wherein, the second encoding frame rate is obtained by the server adjusting the first encoding frame rate based on the frame information.

[0022] On one hand, embodiments of this application provide an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to perform the following steps:

[0023] The system receives frame-stacking information sent by the client, which indicates the target number of frame-stacking frames. The target number of frame-stacking frames is the difference between the number of video frames received by the client and the number of video frames decoded within a preset time range.

[0024] Obtain a first encoding frame rate, which is used to indicate the encoding frame rate of the video frames corresponding to the target number of frames;

[0025] When the target number of frames meets the frame rate adjustment condition, the first encoding frame rate is adjusted based on the target number of frames to obtain the second encoding frame rate;

[0026] The application screen is video encoded based on the second encoding frame rate to obtain the target video frame, and the target video frame is sent to the client.

[0027] On one hand, embodiments of this application provide another electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to perform the following steps:

[0028] The number of video frames received and the number of video frames decoded within a preset time range are obtained; wherein, the video frames received within the preset time range are video frames obtained by the server performing video encoding processing based on a first encoding frame rate.

[0029] The frame-stacking information is determined based on the difference between the number of video frames received and the number of video frames decoded within the preset time range, and the frame-stacking information is sent to the server.

[0030] The system receives video frames sent by the server, which are obtained by video encoding based on a second encoding frame rate. It then performs video decoding on the received video frames to obtain the corresponding application screen and displays the application screen obtained by video decoding. The second encoding frame rate is obtained by the server adjusting the first encoding frame rate based on the frame information.

[0031] On one hand, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, perform the following steps:

[0032] The system receives frame-stacking information sent by the client, which indicates the target number of frame-stacking frames. The target number of frame-stacking frames is the difference between the number of video frames received by the client and the number of video frames decoded within a preset time range.

[0033] Obtain a first encoding frame rate, which is used to indicate the encoding frame rate of the video frames corresponding to the target number of frames;

[0034] When the target number of frames meets the frame rate adjustment condition, the first encoding frame rate is adjusted based on the target number of frames to obtain the second encoding frame rate;

[0035] The application screen is video encoded based on the second encoding frame rate to obtain the target video frame, and the target video frame is sent to the client.

[0036] On the one hand, embodiments of this application provide another computer-readable storage medium storing computer program instructions, which, when executed by a processor, are used to perform the following steps:

[0037] The number of video frames received and the number of video frames decoded within a preset time range are obtained; wherein, the video frames received within the preset time range are video frames obtained by the server performing video encoding processing based on a first encoding frame rate.

[0038] The frame-stacking information is determined based on the difference between the number of video frames received and the number of video frames decoded within the preset time range, and the frame-stacking information is sent to the server.

[0039] The system receives video frames sent by the server, which are obtained by video encoding based on a second encoding frame rate. It then performs video decoding on the received video frames to obtain the corresponding application screen and displays the application screen obtained by video decoding. The second encoding frame rate is obtained by the server adjusting the first encoding frame rate based on the frame information.

[0040] On one hand, embodiments of this application provide a computer program product or computer program that includes computer instructions that, when executed by a processor, can implement the above-described method.

[0041] By employing the embodiments of this application, it is possible to obtain the frame information sent by the client, and then, when the frame information meets certain conditions, to adaptively adjust the encoding frame rate of the video frame encoding in the server based on the frame information, and then generate video frames based on the adjusted encoding frame rate. The frame information sent by the client can characterize whether the terminal device's ability to decode video frames within a preset time range can support the encoding frame rate. This allows the frame rate of the video frames generated by the server to match the client's ability to decode video frames, reducing the noticeable delay in the display caused by insufficient performance of the terminal device leading to delayed decoding and display of received video frames. Therefore, by employing this application, the encoding frame rate of the video frame encoding can be adaptively adjusted based on the client's frame information, helping to reduce the noticeable delay in the display. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0044] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0045] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of a frame rate adjustment method provided in an embodiment of this application;

[0047] Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0048] Figure 6a This is a schematic diagram of a counting method for the number of received frames provided in an embodiment of this application;

[0049] Figure 6b This is a schematic diagram of a method for counting the number of decoded frames provided in an embodiment of this application;

[0050] Figure 6c This is a schematic diagram illustrating a method for uploading a large number of frames according to an embodiment of this application;

[0051] Figure 7This is an interactive schematic diagram of a data processing method provided in an embodiment of this application;

[0052] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of another data processing device provided in an embodiment of this application;

[0054] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0055] Figure 11 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0057] This application proposes a data processing scheme that can acquire frame information sent by the client, and then, when the frame information meets certain conditions, adaptively adjust the encoding frame rate of the video frame encoding in the server based on the frame information, and then generate video frames based on the adjusted encoding frame rate. The frame information sent by the client can characterize whether the terminal device's ability to decode video frames within a preset time range can support the encoding frame rate. This allows the frame rate of the video frames generated by the server to match the client's decoding capability, reducing the noticeable delay in the display caused by insufficient performance of the terminal device leading to delayed decoding and display of received video frames. Therefore, by employing this application, the encoding frame rate of the video frame encoding can be adaptively adjusted based on the client's frame information, helping to reduce the noticeable delay in the display.

[0058] In one possible implementation, embodiments of this application can be applied to a data processing system. Please see [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application. For example... Figure 1 As shown, the data processing system may include a client 11 and a server 12. The client 11 can be a program providing local services to users. The client 11 can run on a terminal device, which may include, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, smart speakers, etc. The client 11 can also be a cloud application client, and cloud applications may include cloud gaming, cloud education, cloud security, cloud conferencing, cloud calling, and cloud social networking, etc., without limitation.

[0059] Server 12 can be the server corresponding to the client, used to provide services to the client, such as providing resources to the client and storing client data. Server 12 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. There are no restrictions here.

[0060] For example, this section uses a cloud gaming scenario to illustrate the data interaction process between client 11 and server 12. Cloud gaming, also known as gaming on demand, is a gaming technology based on cloud computing. In a cloud gaming scenario, the game itself is not on the user's terminal device; only the user client (also called the cloud application client) runs on the terminal device, while the actual game application (also called the virtual cloud client) runs on the server. Business objects can perform object operations through the client (such as clicking a control to indicate that a virtual game object in the game is moving left). The server can then receive object operation events sent by the cloud gaming client on the terminal device. The virtual cloud client on the server responds to these object operation events by generating the corresponding game screen (such as generating a game screen showing the virtual game object moving left), and performs video encoding processing on the game screen to obtain the corresponding video frames. These encoded video frames are then sent to the cloud gaming client. The cloud gaming client can then perform video decoding processing on the received video frames to obtain the corresponding game screen (such as a game screen showing the virtual game object moving left), and display the decoded game screen, allowing the business object to see the virtual game object moving left in the cloud game. In cloud gaming, virtual game objects can be virtual characters, virtual animals, cartoon characters, etc., that can be controlled by the user within a virtual scene of the cloud game. The virtual scene can be a virtual environment displayed (or provided) on the terminal device during cloud gaming, where the user can control the game objects to move. This virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional virtual environment, or a purely fictional virtual environment. The virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this application does not limit the dimension of the virtual scene.

[0061] Understandably, during the aforementioned data interaction process, the server can continuously generate video frames corresponding to the game screen based on a certain encoding frame rate, forming a video stream. Then, the client can perform video decoding processing on the received video frames based on a certain decoding frame rate to obtain the corresponding application screen and display it. The encoding frame rate indicates the number of video frames the server obtains per unit time through video encoding processing of the captured application screen, while the decoding frame rate indicates the number of video frames the client completes decoding per unit time. If the terminal device has good performance and capabilities, it can support a higher decoding frame rate on the client. If the server generates video frames at a higher encoding frame rate, and the terminal device also has good video decoding capabilities (or decoding and rendering capabilities, or rendering capabilities), then the user will experience a smoother game experience. However, if the server sends video frames to the client at an encoding frame rate exceeding the capabilities of the terminal device, and if the terminal device has many applications running simultaneously, its processing power decreases. The client cannot decode the received video frames in a timely manner, resulting in a buildup of video frames and significant latency in the game, leading to a poor user experience. If the server continues to send video frames to the client at an encoding frame rate exceeding the terminal device's capabilities, the number of buildup video frames will continue to accumulate. If the client does not have a frame dropping strategy, the latency of the application screen will continuously increase. Even with a frame dropping strategy, a video stream with an encoding frame rate exceeding the client's actual required frame rate will consume additional hardware and bandwidth resources, increasing business costs. Therefore, providing the client with a video stream at a suitable frame rate is essential. This application embodiment can upload the number of video frames buildup in the client (also called the number of buildup frames) to the server. The server can then adjust its encoding frame rate in a timely manner based on the number of buildup frames, preventing the buildup from continuously increasing. Furthermore, it allows the server's encoding frame rate to adapt to the performance of the terminal device, helping to reduce the perceived latency in the screen display.

[0062] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0063] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0064] Based on the above description, this application proposes a data processing method. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method can be executed by the aforementioned server. The data processing method may include the following steps.

[0065] S201. Receive the frame-stacking information sent by the client. The frame-stacking information is used to indicate the target number of frame-stackings. The target number of frame-stackings is the difference between the number of video frames received by the client and the number of video frames decoded within a preset time range.

[0066] The frame-stacking information can be a target number of frames, a parameter identifier indicating the target number of frames, or the number of video frames received and decoded by the client within a preset time range; there are no restrictions here. It is understood that if the frame-stacking information is the number of video frames received and decoded by the client within a preset time range, the server, upon receiving the frame-stacking information, can determine the target number of frames based on the difference between the number of video frames received and decoded by the client within the preset time range indicated by the frame-stacking information. If the frame-stacking information is a parameter identifier indicating the target number of frames, the server can determine the corresponding target number of frames based on the parameter identifier indicated by the frame-stacking information.

[0067] This preset time range is the period used to collect frame information. For example, it can be a time period before the client collects frame information. For instance, if the client determines the time point for collecting frame information as t1, then the preset time range can be the period from 2 seconds before t1 to t1.

[0068] Understandably, the server can continuously generate video frames at a certain encoding frame rate within a preset time range and send the video frames to the client. The client can perform video decoding (or decoding rendering, or rendering) on ​​the received video frames. The video frames that have undergone video frame decoding can be called decoded video frames.

[0069] For example, if the number of video frames received by the client within a preset time range is m, and the number of video frames decoded by the client is n, then the target number of frames to be stored can be mn. It's understandable that, generally speaking, if the number of video frames decoded by the client within the preset time range is similar to the number of video frames received by the client (i.e., m and n are similar), it means that the client's video decoding capability can support the server's encoding frame rate. If the number of video frames decoded by the client within the preset time range is much less than the number of video frames received by the client (i.e., mn is greater than a certain threshold), it means that the client's video decoding capability is insufficient to support the server's encoding frame rate. The value of mn is the number of frames to be stored corresponding to the preset time range (i.e., the target number of frames to be stored). Therefore, the server can receive frame storage information uploaded by the client indicating the target number of frames to be stored. This allows the server to reduce the encoding frame rate of the generated video frames when the number of frames to be stored exceeds a certain threshold, adapting to the client's corresponding video decoding capability and preventing the client from accumulating video frames and causing excessive latency.

[0070] In one embodiment, the hoarded frame information can be uploaded by the client at certain time intervals. For example, if the time interval is 2 seconds, the client can upload hoarded frame information corresponding to the number of hoarded frames generated within those 2 seconds every 2 seconds and then upload the hoarded frame information to the server. Alternatively, the hoarded frame information can be uploaded by the client when certain hoarded frame upload conditions are triggered. For example, the client uploads hoarded frame information only when it detects that the number of hoarded frames within a preset time range is greater than or equal to a certain threshold. In other words, the client determines the current hoarded frame situation. If the client detects that it cannot support the encoding frame rate corresponding to the currently received video frame, it can upload the hoarded frame information to the server so that the server can adjust the encoding frame rate based on the hoarded frame information. The hoarded frame information can also be triggered by the client or the terminal device running the client when it detects that the terminal's operating information meets the hoarded frame detection conditions. For example, it triggers hoarded frame information detection or acquisition when it detects dual-clienting or memory usage exceeding a threshold, and then uploads the acquired hoarded frame information to the server.

[0071] S202. Obtain the first encoding frame rate, which is used to indicate the encoding frame rate of the video frames corresponding to the target number of frames.

[0072] The video frames corresponding to the target number of frames can be the video frames received by the client when the target number of frames is determined, that is, the video frames received by the client within the preset time range (video frames sent by the server to the client). In other words, the first encoding frame rate can be the encoding frame rate of the video frames encoded by the server within the preset time range.

[0073] S203. When the target number of frames meets the frame rate adjustment condition, the first encoding frame rate is adjusted based on the target number of frames to obtain the second encoding frame rate.

[0074] The frame rate adjustment condition can be a condition that needs to be met to instruct the server to adjust the encoded frame rate. This frame rate adjustment condition can be a condition that increases or decreases the encoded frame rate. Therefore, adjusting the first encoded frame rate can be either increasing or decreasing the first encoded frame rate; there are no restrictions here.

[0075] The second encoding frame rate can be an adjusted encoding frame rate that is more suitable for the client's decoding capabilities. Then, video frames can be encoded based on the second encoding frame rate.

[0076] In one embodiment, when the target number of frames meets the frame rate adjustment condition, the first encoding frame rate is adjusted based on the target number of frames to obtain the second encoding frame rate. This may include: when the target number of frames is greater than a first threshold, determining the first frame rate adjustment value based on the target number of frames; and reducing the first encoding frame rate by the first frame rate adjustment value to obtain the second encoding frame rate.

[0077] The condition that the target number of stored frames exceeds a first threshold can be used as a frame rate adjustment condition to reduce the encoded frame rate. The first threshold can be the minimum value that the target number of stored frames must satisfy when the first encoded frame rate needs to be reduced. The first frame rate adjustment value can be a frame rate adjustment value determined based on the target number of stored frames for reducing the encoded frame rate. Generally, to fully utilize the client's decoding capabilities and the server's encoding capabilities, the first threshold can be set to 1, meaning that when the target number of stored frames is greater than 1, the first encoded frame rate is reduced.

[0078] Optionally, the first frame rate adjustment value can be determined based on the target number of stored frames. This can be achieved by determining the number of video frames that need to be reduced per unit time based on the target number of stored frames. In other words, the first frame rate adjustment value can be determined based on the target number of stored frames and the length of a preset time range. For example, the target number of stored frames can be divided by the length of the preset time range to obtain the first frame rate adjustment value. Optionally, if the value obtained by dividing the target number of stored frames by the length of the preset time range is not an integer, the first frame rate adjustment value can be determined by rounding, rounding up, or other rounding methods. There are no restrictions here. The first frame rate adjustment value can also be determined based on the range in which the target number of stored frames falls. For example, if the preset value for a target number of stored frames in the range of 1-3 is 3, then when the target number of stored frames falls within this range, the corresponding first frame rate adjustment value can be determined to be 3.

[0079] For example, the second coded frame rate can be determined using the following formula (Formula 1).

[0080]

[0081] Where R2 represents the second encoding frame rate, R1 represents the first encoding frame rate, d represents the target number of frames, T represents the length of the preset time range, such as the length of a time period, and round() represents rounding the contents in parentheses.

[0082] For example, if the target number of frames is 10, and the first threshold is 1, then the target number of frames is greater than the first threshold. Therefore, the first encoding frame rate can be adjusted based on the target number of frames. If the preset time range is 2 seconds, then the first frame rate adjustment value can be obtained as 5 according to 10 ÷ 2 = 5. When the first encoding frame rate is 80, the second encoding frame rate can be 80 - 5 = 75.

[0083] In one embodiment, when the target number of stored frames meets the frame rate adjustment condition, the first encoded frame rate is adjusted based on the target number of stored frames to obtain the second encoded frame rate. This may also include the following steps: obtaining the historical number of stored frames and determining the total number of stored frames based on the historical number of stored frames and the target number of stored frames, wherein the historical number of stored frames is the number of stored frames indicated by the stored frame information previously sent by the client; when the total number of stored frames is greater than a second threshold, determining the second frame rate adjustment value based on the total number of stored frames; and reducing the first encoded frame rate by the second frame rate adjustment value to obtain the second encoded frame rate.

[0084] The historical number of stored frames can be the number of stored frames indicated by the client's previous stored frame information, which is also the number of stored frames indicated by the server's previous stored frame information. For example, if the client sends stored frame information x1 at time t1 and then sends stored frame information x2 again at time t2 (time t2 is later than time t1), then the stored frame information x1 sent by the client at time t1 is the stored frame information sent before the stored frame information x2 sent by the client at time t2. When the server adjusts the encoded frame rate based on stored frame information x1, it can obtain the stored frame information x1 sent by the client at time t1 and use the stored frame information x1 sent by the client at time t1 as the historical number of stored frames.

[0085] Optionally, the historical frame count can be sent by the client along with information indicating the target frame count when sending frame count information, or it can be the frame count indicated by the previously received frame count information retrieved by the server. It is understood that after receiving frame count information, the server can store the frame count information or the target frame count indicated by the frame count information in a cache area. Optionally, after each receiving of frame count information, the server can overwrite the frame count information or the target frame count indicated by the frame count information in the cache area based on the latest frame count information, so that the client can synchronously send the historical frame count when sending frame count information again. Optionally, the server's cache area can store the current frame count and the historical frame count, and then after each receiving of frame count information, it can update the current frame count and the historical frame count in the cache area based on the latest frame count information, such as updating the historical frame count in the cache area based on the original current frame count stored in the cache area, and then updating the current frame count stored in the cache area based on the target frame count indicated by the latest received frame count information.

[0086] The condition that the total number of frames hoarded is greater than the second threshold can be used as another frame rate adjustment condition to reduce the encoded frame rate. The second threshold can be the minimum value that the total number of frames hoarded must meet when the first encoded frame rate needs to be reduced. The second threshold can be the same as or different from the first threshold, and there is no restriction in this case.

[0087] Optionally, a second frame rate adjustment value is determined based on the total number of stored frames. Refer to the description above regarding determining the first frame rate adjustment value based on the target number of stored frames. This second frame rate adjustment value can be a frame rate adjustment value used to reduce the encoded frame rate, determined based on the total number of stored frames. This second frame rate adjustment value can be the same as or different from the first frame rate adjustment value; there are no restrictions here. Determining the second frame rate adjustment value based on the total number of stored frames can mean determining the number of video frames to be reduced per unit time based on the total number of stored frames. In other words, the second frame rate adjustment value can be determined based on the total number of stored frames and the length of a preset time range, such as dividing the total number of stored frames by the length of the preset time range. Optionally, if the value obtained by dividing the total number of stored frames by the length of the preset time range is not an integer, the second frame rate adjustment value can be determined using rounding methods such as rounding to the nearest integer or rounding up; there are no restrictions here. The second frame rate adjustment value can be determined based on the total number of frames stored. Alternatively, a preset value can be determined based on the range of the total number of frames stored. For example, if the preset value is 3 for the total number of frames stored in the range of 1-3, then when the total number of frames stored is in the range of 1-3, the corresponding second frame rate adjustment value can be determined to be 3.

[0088] For example, if the target number of stored frames indicated by the stored frame information is 0, and the number of historical stored frames obtained is 12, then the total number of stored frames is 12. If the second threshold is 1, then the total number of stored frames is greater than the second threshold, and thus the adjustment of the first encoding frame rate can be determined based on the total number of stored frames. If the preset time range is 2 seconds, then the second frame rate adjustment value can be obtained as 6 according to 12 ÷ 2 = 6. When the first encoding frame rate is 80, the second encoding frame rate can be 80 - 6 = 74.

[0089] In one embodiment, when the target number of stacked frames meets the frame rate adjustment condition, the first encoding frame rate is adjusted based on the target number of stacked frames to obtain the second encoding frame rate. This may also include: when the number of stacked frames indicated by the stacked frame information received K times consecutively is less than or equal to a third threshold, obtaining a third frame rate adjustment value; where K is a positive integer; and increasing the first encoding frame rate by the third frame rate adjustment value to obtain the second encoding frame rate.

[0090] The condition that the number of cached frames indicated by K consecutive received cached frame information is less than or equal to a third threshold can be used as a frame rate adjustment condition to increase the encoded frame rate. K can be a value such as 1, 2, 3, etc. The third threshold can be the maximum value that the target cached frame number must satisfy when the first encoded frame rate needs to be increased. It is understood that the third threshold can be less than the first threshold. The third frame rate adjustment value can be the number of video frames for which the encoded frame rate needs to be increased. Optionally, the third frame rate adjustment value can be a preset value, for example, the third frame rate adjustment value can be 1. It is understood that in some scenarios, if the client does not have cached frames, it means that the client can fully support the server's current encoded frame rate, or even that the client's decoding capability is not being fully utilized. In this case, the server can increase the encoded frame rate, thereby enabling the server's encoded frame rate to adapt to the client's video frame decoding capability, maximizing the utilization of both the client's video frame decoding capability and the server's video frame encoding capability.

[0091] For example, if K=2 and the third threshold is 0, then the number of frames indicated by the frame information received by the server at time t1 is 0, and the number of frames indicated by the frame information received again at time t2 (i.e., the next frame information received after the frame information received at time t1) is 0. In other words, the number of frames indicated by two consecutive received frame information is less than or equal to 0, so the first coded frame rate can be increased. Specifically, if the third frame rate adjustment value is 1 and the first coded frame rate is 80, then the second coded frame rate can be 80+1=81.

[0092] S204. Perform video encoding processing on the application screen based on the second encoding frame rate to obtain the target video frame, and send the target video frame to the client.

[0093] The target video frame can be a video frame encoded based on the second encoding frame rate. It is understood that in the server corresponding to the cloud application, application frames can be continuously captured and video encoded to obtain video frames. It is understood that in the server, video encoding of the application frames based on the second encoding frame rate is used to indicate the number of target video frames generated per unit time. Optionally, in the server, application frames can also be captured at the same frequency as the second encoding frame rate, and encoded into target video frames based on the second encoding frame rate.

[0094] It is understandable that after receiving the target video frame, the client can perform video decoding processing on the received target video frame to obtain the corresponding application screen, and then display the application screen obtained from the video decoding processing.

[0095] In one possible implementation, when the target number of stored frames does not meet the frame rate adjustment condition, the application screen is video encoded based on the first encoding frame rate to obtain the target video frame, and the target video frame is sent to the client. This means that when the target number of stored frames does not meet the frame rate adjustment condition, the server's encoding frame rate is not adjusted, and video encoding is still performed according to the first encoding frame rate. For example, when the target number of stored frames is less than or equal to the first threshold, and does not meet the condition that the number of stored frames indicated by K consecutive received stored frame information is less than or equal to the third threshold, it is determined that the target number of stored frames does not meet the frame rate adjustment condition, and therefore the application screen is video encoded based on the first encoding frame rate, meaning the server's encoding frame rate is not adjusted. Similarly, when the total number of stored frames corresponding to the historical stored frame number and the target stored frame number is less than or equal to the second threshold, and does not meet the condition that the number of stored frames indicated by K consecutive received stored frame information is less than or equal to the third threshold, it is determined that the target number of stored frames does not meet the frame rate adjustment condition, and therefore the application screen is video encoded based on the first encoding frame rate, meaning the server's encoding frame rate is not adjusted.

[0096] Understandably, in practical applications, clients can continuously upload frame information to the server. The server then evaluates this information to determine whether to decrease, increase, or maintain the frame rate. Typically, during client operation, the server can adjust the encoding frame rate dynamically based on one or more of the frame information uploaded by the client: decreasing the frame rate, increasing the frame rate, or maintaining the frame rate.

[0097] By employing the embodiments of this application, it is possible to obtain the frame information sent by the client, and then, when the frame information meets certain conditions, to adaptively adjust the encoding frame rate of the video frame encoding in the server based on the frame information, and then generate video frames based on the adjusted encoding frame rate. The frame information sent by the client can characterize whether the terminal device's ability to decode video frames within a preset time range can support the encoding frame rate. This allows the frame rate of the video frames generated by the server to match the client's ability to decode video frames, reducing the noticeable delay in the display caused by insufficient performance of the terminal device leading to delayed decoding and display of received video frames. Therefore, by employing this application, the encoding frame rate of the video frame encoding can be adaptively adjusted based on the client's frame information, helping to reduce the noticeable delay in the display.

[0098] Furthermore, this section uses the example of a client uploading frame information at certain time intervals to illustrate a frame rate adjustment process. Please refer to [link / reference]. Figure 3 , Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. This method can be executed by the aforementioned server. The data processing method may include the following steps.

[0099] S301. Receive the frame-stacking information sent by the client. The frame-stacking information is used to indicate the target number of frame-stackings. The target number of frame-stackings is the difference between the number of video frames received by the client and the number of video frames decoded in the i-th time period.

[0100] This means that the client uploads frame information at certain time intervals, and the uploaded frame information is the frame information for the corresponding time period. The i-th time period is any time period where i is a positive integer, and the frame information is the frame information for the i-th time period. The time range indicated by the i-th time period is also equivalent to the preset time range mentioned above. For example, if the period length is 2 seconds, the client can send frame information corresponding to 2 seconds to the server every 2 seconds.

[0101] S302. Obtain the first encoding frame rate, which is used to indicate the encoding frame rate of the video frames corresponding to the target number of frames.

[0102] It can be understood that if the frame information is the frame information of the i-th time period, then the video frame corresponding to the target number of frame information is the video frame received by the client (or the video frame generated by the server) within the i-th time period. Therefore, the first encoding frame rate is the encoding frame rate of the video frame obtained by performing video encoding processing on the application screen within the i-th time period.

[0103] S303. When the target number of frames is greater than the first threshold, obtain the period length corresponding to the i-th time period, and determine the first frame rate adjustment value based on the period length and the target number of frames.

[0104] It is understandable that having a target number of frames greater than the first threshold can be a frame rate adjustment condition used to reduce the encoded frame rate. It is also understandable that each time period has the same period length; the period length corresponding to the i-th time period is the same as the period length of any other time period. The period length indicates the duration of a single period; for example, the period length of the i-th time period is 2 seconds.

[0105] Understandably, determining the first frame rate adjustment value based on the period length and the target number of frames can be achieved by first determining the number of frames per unit time based on the period length and the target number of frames, and then determining the first frame rate adjustment value based on the number of frames per unit time. For example, the target number of frames can be divided by the period length to obtain the first frame rate adjustment value. Optionally, if the value obtained by dividing the target number of frames by the period length is not an integer, the first frame rate adjustment value can be determined by rounding, rounding up, or other rounding methods; there are no restrictions here.

[0106] In one embodiment, determining the first frame rate adjustment value based on the target number of stored frames may further include the following steps: obtaining the historical number of stored frames, which indicates the difference between the number of video frames received by the client in the (i-1)th time period and the number of video frames decoded; obtaining the period length corresponding to the i-th time period, and determining the first frame rate adjustment value based on the period length, the historical number of stored frames, and the target number of stored frames; wherein, the second encoding frame rate is the encoding frame rate of the video frames obtained by performing video encoding processing on the application screen in the (i+1)th time period.

[0107] It can be understood that the (i-1)th time period can be the previous time period of the ith time period, and the (i+1)th time period can be the next time period of the ith time period. In other words, the server can determine the first frame rate adjustment value based on the number of frames stored in the current time period and the number of frames stored in the previous time period, and then determine the second encoding frame rate for video frame encoding processing in the next time period based on the first frame rate adjustment value.

[0108] It is understandable that the above-mentioned historical frame count refers to the number of frames indicated by the frame information previously sent by the client. When the client uploads frame information according to time periods, the number of frames indicated by the frame information corresponding to the (i-1)th time period is the number of frames indicated by the frame information previously sent by the client for the frame information of the i-th time period.

[0109] Optionally, the first frame rate adjustment value can be determined based on the period length, the historical number of frames hoarded, and the target number of frames hoarded. This can be achieved by determining the total number of frames hoarded based on the historical and target numbers, and then determining the first frame rate adjustment value based on the period length and the total number of frames hoarded. In other words, the number of frames hoarded per unit time can be determined based on the total number of frames hoarded. For example, the first frame rate adjustment value can be obtained by dividing the total number of frames hoarded by the period length. For instance, if the period length is 2 seconds, the historical number of frames hoarded is 8, and the target number of frames hoarded is 0, then the total number of frames hoarded is 8, and the first frame rate adjustment value is 8 ÷ 2 = 4.

[0110] S304. Reduce the first encoding frame rate by the first frame rate adjustment value to obtain the second encoding frame rate, wherein the second encoding frame rate is the encoding frame rate of the video frame obtained by performing video encoding processing on the application screen in the (i+1)th time period.

[0111] It is understandable that the second encoding frame rate can be used to encode video frames for the (i+1)th time period.

[0112] S305. Perform video encoding processing on the application screen based on the second encoding frame rate to obtain the target video frame, and send the target video frame to the client.

[0113] For example, the process of adjusting the frame rate based on the target number of stored frames indicated by the stored frame information uploaded periodically is described here with reference to the following table (Table 1).

[0114] Table 1

[0115]

[0116] Here's an explanation: A time period (also called a cycle) has a length of 2 seconds. In the first cycle, the encoding frame rate and the client decoding frame rate are the same (both 50), so the target number of stored frames is 0. Therefore, the first frame rate adjustment value can be set to 0, meaning the encoding frame rate is not reduced. In the second cycle, video frames are still generated at an encoding frame rate of 50. However, the terminal device's capabilities are reduced; if more programs are running, the client's decoding capability decreases, and the client decoding frame rate becomes 45. This results in 10 stored frames in the second time cycle, and since the target number of stored frames is greater than the first threshold, the first frame rate adjustment value can be 10 / 2 = 5, thus reducing the encoding frame rate to 45. In the third cycle, video frames can be generated at an encoding frame rate of 45. At this point, the server's encoding frame rate and the client's decoding capability are compatible (both encoding and decoding frame rates are 45). Since no new stored frames are added in the third cycle, the encoding frame rate does not need adjustment, and the number of stored frames in the client no longer increases.

[0117] For example, the process of adjusting the frame rate based on the target number of stored frames and the historical number of stored frames indicated by the stored frame information uploaded periodically is explained here with reference to the following table (Table 2).

[0118] Table 2

[0119]

[0120] Here, we can understand that the length of a time period (also called a cycle) is 2 seconds. In the first cycle, the encoding frame rate and the client decoding frame rate are the same (both are 50), so the corresponding target number of stored frames is 0. Therefore, the first frame rate adjustment value can be set to 0, that is, the encoding frame rate is not reduced. In the second cycle, video frames are still generated at an encoding frame rate of 50. At this time, the device capability of the terminal device is reduced. If more programs are opened, the client decoding capability is reduced, and the client decoding frame rate becomes 45. Thus, 10 stored frames are generated in the second time cycle, and the historical number of stored frames is 0 (that is, the number of stored frames in the first time cycle is 0). The total number of stored frames is 10 + 0 = 10. The total number of stored frames is greater than the second threshold, so the first frame rate adjustment value can be (10 + 0) / 2 = 5, thus reducing the encoding frame rate to 45. In the third cycle, video frames can be generated at an encoding frame rate of 45. At this point, the server's encoding frame rate is compatible with the client's decoding capability. However, since there were 10 stockpiled frames in the second cycle, if it is necessary to completely eliminate these stockpiled frames, the first frame rate adjustment value can be determined based on the target number of stockpiled frames in the third cycle and the historical number of stockpiled frames (i.e., the number of stockpiled frames in the second cycle was 10). For example, the first frame rate adjustment value could be (0+10) / 2 = 5, thus reducing the encoding frame rate by 5, i.e., lowering the encoding frame rate to 40. In the fourth cycle, the historical stockpiled frames have been eliminated, and no more stockpiled frames are generated, so the encoding frame rate does not need to be reduced. Therefore, after two encoding frame rate adjustments, the generated stockpiled frames can be eliminated. It is understandable that at this point, the encoding frame rate is lower than the client's decoding capability, so the client's decoding capability cannot be fully utilized. Therefore, the encoding frame rate can be gradually increased subsequently. For example, when the number of stockpiled frames indicated by the stockpiled frame information for K consecutive cycles is 0, the encoding frame rate can be increased by a certain value.

[0121] In one possible implementation, when the number of frames indicated by the frame information of K consecutive time periods is less than or equal to the third threshold, a third frame rate adjustment value is obtained, where K is a positive integer; the first encoding frame rate is increased by the third frame rate adjustment value to obtain a second encoding frame rate, which is the encoding frame rate of the video frame obtained by performing video encoding processing on the application screen in the (i+1)th time period.

[0122] For example, if K=2 and the third threshold is 0, then the number of frames indicated by the frame information received by the server in the first time period is 0, and the number of frames indicated by the frame information received in the second time period is also 0. That is, the number of frames indicated by the frame information received in two consecutive times is less than or equal to 0, so the first encoding frame rate can be increased. If the obtained third frame rate adjustment value is 1, and the first encoding frame rate is 80, then the second encoding frame rate can be obtained as 80+1=81. Therefore, in the third time period, video encoding processing can be performed based on the second encoding frame rate (i.e., 81).

[0123] In one possible implementation, when the number of stored frames does not meet the frame rate adjustment condition, video encoding processing of the application screen is performed based on the second encoding frame rate in the (i+1)th time period to obtain the target video frame, and the target video frame is sent to the client. This means that when the target number of stored frames does not meet the frame rate adjustment condition, the server's encoding frame rate is not adjusted, and video encoding processing is still performed according to the target number of stored frames in the (i+1)th time period. For example, when the target number of stored frames is less than or equal to a first threshold, and does not meet the condition that the number of stored frames indicated by the stored frame information for K consecutive time periods is less than or equal to a third threshold, it is determined that the target number of stored frames does not meet the frame rate adjustment condition, and therefore, video encoding processing of the application screen is performed based on the first encoding frame rate in the (i+1)th time period, meaning that the server's encoding frame rate is not adjusted. For example, when the total number of stored frames corresponding to the historical stored frame count and the target stored frame count is less than or equal to the second threshold, and the stored frame count indicated by the stored frame information for K consecutive time periods is less than or equal to the third threshold, it is determined that the target stored frame count does not meet the frame rate adjustment condition. Then, in the (i+1)th time period, the application screen is processed for video encoding based on the first encoding frame rate, that is, the encoding frame rate of the server is not adjusted.

[0124] Understandably, in practical applications, the client can upload frame-stacking information to the server at the end of each time period. The server then judges the frame-stacking information to determine whether to decrease, increase, or keep it unchanged. Typically, during client operation, the server can adjust the encoded frame rate dynamically based on one or more of the following: decreasing the encoded frame rate, increasing the encoded frame rate, or keeping the encoded frame rate unchanged.

[0125] This section explains the process of adjusting the server's frame rate, using illustrations. Please refer to [link / reference]. Figure 4 , Figure 4This is a schematic diagram of a frame rate adjustment method provided in an embodiment of this application. The server can detect whether the query interval is met (step S401). The query interval is the time interval used to query the number of stored frames. Detecting whether the query interval is met can also be detecting whether the end point of a time period has been reached. If the query interval is met, the server obtains the target number of stored frames indicated by the stored frame information sent by the client (step S402). For example, after receiving the stored frame information sent by the client, the server can store the target number of stored frames indicated by the stored frame information in the cache area. Then, when the query interval is met, the server can obtain the target number of stored frames indicated by the stored frame information sent by the client from the cache area. The server can then detect whether the target number of stored frames is greater than 1 (step S403). Here, the target number of stored frames being greater than 1 is a frame rate adjustment condition used to reduce the encoded frame rate, and 1 is the first threshold. If the target number of stored frames is greater than 1, the encoded frame rate can be reduced by a first frame rate adjustment value (step S404). The first encoded frame rate can be determined based on the period length of the time period and the target number of stored frames. If the target number of stored frames is not greater than 1, the process can proceed to the next cycle and wait for the next query interval to be met. Optionally, if the target number of stored frames is not greater than 1, but the number of target stored frames received for K consecutive time cycles is less than or equal to the third threshold, the third frame rate adjustment value can be determined, and the encoded frame rate can be increased by the third frame rate adjustment value.

[0126] By employing the embodiments of this application, it is possible to obtain the frame information sent by the client, and then, when the frame information meets certain conditions, to adaptively adjust the encoding frame rate of the video frame encoding in the server based on the frame information, and then generate video frames based on the adjusted encoding frame rate. The frame information sent by the client can characterize whether the terminal device's ability to decode video frames within a preset time range can support the encoding frame rate. This allows the frame rate of the video frames generated by the server to match the client's ability to decode video frames, reducing the noticeable delay in the display caused by insufficient performance of the terminal device leading to delayed decoding and display of received video frames. Therefore, by employing this application, the encoding frame rate of the video frame encoding can be adaptively adjusted based on the client's frame information, helping to reduce the noticeable delay in the display.

[0127] Please see Figure 5 , Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application. The method can be executed by a terminal device running a client. The data processing method may include the following steps.

[0128] S501. Obtain the number of video frames received and the number of video frames decoded within a preset time range; wherein, the video frames received within the preset time range are video frames obtained by the server performing video encoding processing based on the first encoding frame rate.

[0129] It can be understood that, within a preset time range, the server can generate video frames according to the first encoding frame rate, and the frame rate of the video frames received by the client is consistent with the first encoding frame rate.

[0130] Understandably, within a preset time range, the client can start counting the received video frames from zero and also start counting the decoded video frames from zero. For ease of understanding, the number of received video frames is also called the number of received frames, and the number of decoded video frames is also called the number of decoded frames. That is, the client increments the number of received frames by one for each video frame received and increments the number of decoded frames by one for each video frame decoded. Therefore, at the end of the preset time range, the client can statistically determine the number of video frames received and decoded within that time range.

[0131] S502. Determine the frame storage information based on the difference between the number of video frames received and the number of video frames decoded within a preset time range, and send the frame storage information to the server.

[0132] The process of determining the frame-stacking information based on the difference between the number of video frames received and the number of video frames decoded within a preset time range can be as follows: This can be achieved by determining the target frame-stacking number based on the difference between the number of received and decoded video frames, and then determining the frame-stacking information based on the target frame-stacking number; alternatively, it can be based on the corresponding parameter identifier determined by the target frame-stacking number, and then determining the corresponding frame-stacking information based on the parameter identifier; or it can be directly determined based on the number of video frames received and decoded by the client within the preset time range—no restriction is placed here. It is understood that if the frame-stacking information consists of the number of video frames received and decoded by the client within the preset time range, the server, upon receiving the frame-stacking information, can determine the target frame-stacking number based on the difference between the number of video frames received and decoded by the client within the preset time range indicated by the frame-stacking information. If the frame-stacking information is a parameter identifier indicating the target frame-stacking number, the server can determine the corresponding target frame-stacking number based on the parameter identifier indicated by the frame-stacking information.

[0133] S503: Receive video frames sent by the server, which are obtained by video encoding based on the second encoding frame rate; perform video decoding on the received video frames to obtain the corresponding application screen; and display the application screen obtained by video decoding. The second encoding frame rate is obtained by the server adjusting the first encoding frame rate based on the frame information.

[0134] It is understandable that the process by which the server adjusts the first encoded frame rate based on the stored frame information can be referred to the relevant description above, and will not be repeated here.

[0135] In one embodiment, after the client uploads the hoarded frame information, it can also receive video frames sent by the server, which are obtained through video encoding processing based on the first encoding frame rate. The received video frames are then decoded to obtain the corresponding application screen, which is then displayed. Receiving video frames sent by the server, which are obtained through video encoding processing based on the first encoding frame rate, is because the server detects that the target number of hoarded frames indicated by the hoarded frame information does not meet the frame rate adjustment conditions. Therefore, the server may not adjust the encoding frame rate and will still generate video frames according to the original encoding frame rate (i.e., the first encoding frame rate).

[0136] In one possible implementation, if the client determines that the frame information can be stored according to a certain time period, then the frame information is the frame information of the i-th time period, and the time range indicated by the i-th time period is equivalent to the aforementioned preset time range. For example, if the period length of this time period is 2 seconds, then the client can send frame information corresponding to 2 seconds to the server every 2 seconds.

[0137] Furthermore, at the end of the i-th time period, the terminal device can determine the number of received video frames and the number of decoded video frames received in the i-th time period; then, it can determine the target number of stored frames based on the difference between the number of received frames and the number of decoded frames, and determine the stored frame information for the i-th time period based on the target number of stored frames.

[0138] For example, see Figure 6a , Figure 6a This is a schematic diagram illustrating a method for counting the number of received frames provided in an embodiment of this application. The client can receive video frames sent by the server (step S601a), and then determine whether a video frame has been received (step S602a). If a video frame has been received, the number of received frames can be incremented by 1 (step S603a). When the statistical time point is reached (such as the end time point of the i-th time period), the number of video frames received by the client in a time period can be determined.

[0139] Please see Figure 6b , Figure 6bThis is a schematic diagram illustrating a method for counting the number of decoded frames provided in an embodiment of this application. The client can decode the received video frames (step S601b), and then determine whether a video frame has been decoded completely (step S602b). If a video frame has been decoded completely, the number of decoded frames can be incremented by 1 (step S603b). When the statistical time point is reached (such as the end time point of the i-th time period), the number of video frames decoded by the client in a time period can be determined.

[0140] Please see Figure 6c , Figure 6c This is a schematic diagram of a method for uploading the number of frames provided in an embodiment of this application. The client can wait in the communication thread (step S601c) and then determine whether the statistical time point has been reached (step S602c), such as whether the end time point of the i-th time period has been reached. If the statistical time point has been reached, the target number of frames can be calculated based on the number of received video frames and the number of decoded video frames (step S603c), and then the target number of frames can be sent to the server (step S604c). For example, frame information can be generated based on the target number of frames and then sent to the server. Then the number of received frames and the number of decoded frames can be set to 0 (step S605c), so that in the next time period, the number of received video frames and the number of decoded video frames can be recounted.

[0141] This section illustrates the data interaction process between the client and server using diagrams. Please refer to [link / reference]. Figure 7 , Figure 7This is an interactive schematic diagram of a data processing method provided in an embodiment of this application. First, the server can generate a first video frame based on a first encoding frame rate (step S701). This first video frame can be a general term for video frames generated based on the first encoding frame rate. Then, the server can send the first video frame to the client (step S702). The client in the terminal device can decode and display the received video frame, and can also count the number of received video frames and the number of decoded video frames. For example, when the statistical time point is reached, the client can count the number of target frames corresponding to a preset time range (step S703). For example, the statistical time point can be the end point of a time cycle, and the corresponding preset time range is the time range indicated by the time cycle. Or, the statistical time point can be the time point that meets the above-mentioned frame upload conditions or frame detection conditions. This application does not limit this. For example, the preset time range is the time range corresponding to 2 seconds before the statistical time point. It can be understood that reaching the statistical time point can also be called meeting the statistical conditions, which is not limited here. Then, the client can send the counted number of target frames to the server (step S704). Then, the server can adjust the first encoding frame rate based on the target number of frames to obtain a second encoding frame rate, and generate a second video frame based on the second encoding frame rate (step S705). The second encoding frame rate can be the encoding frame rate after adjusting the first encoding frame rate, and the second video frame can be a general term for video frames generated based on the second encoding frame rate. The method for determining the second encoding frame rate based on the target number of frames can be referred to the above description. Then, the server can send the second video frame to the client (step S706). It is understandable that the second encoding frame rate corresponding to the second video frame is an encoding frame rate adapted to the client's decoding capability. The client in the terminal device can decode and display the received video frames, and can also count the number of received video frames and the number of decoded video frames. Then, when the statistical time point is reached, the client can count the target number of stored frames corresponding to the preset time range (step S707). For example, the preset time range is the time range corresponding to 2 seconds before the statistical time point. Then repeat the steps S704-S706 above, and so on. The client can continuously count the number of stored frames and upload them to the server. For example, if the upload is performed according to a certain time period, the server can dynamically adjust the encoding frame rate based on the number of stored frames, reduce the accumulation of video frames in the client, and thus reduce the sense of latency in the application screen display.

[0142] By employing the embodiments of this application, it is possible to obtain the frame information sent by the client, and then, when the frame information meets certain conditions, to adaptively adjust the encoding frame rate of the video frame encoding in the server based on the frame information, and then generate video frames based on the adjusted encoding frame rate. The frame information sent by the client can characterize whether the terminal device's ability to decode video frames within a preset time range can support the encoding frame rate. This allows the frame rate of the video frames generated by the server to match the client's ability to decode video frames, reducing the noticeable delay in the display caused by insufficient performance of the terminal device leading to delayed decoding and display of received video frames. Therefore, by employing this application, the encoding frame rate of the video frame encoding can be adaptively adjusted based on the client's frame information, helping to reduce the noticeable delay in the display.

[0143] Please see Figure 8 , Figure 8 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. Optionally, this data processing apparatus can be installed in the aforementioned server. Figure 8 As shown, the data processing apparatus described in this embodiment may include:

[0144] The receiving unit 801 is used to receive frame-stacking information sent by the client. The frame-stacking information is used to indicate the target number of frame-stacking, which is the difference between the number of video frames received by the client and the number of video frames decoded within a preset time range.

[0145] The acquisition unit 802 is used to acquire a first encoding frame rate, wherein the first encoding frame rate is used to indicate the encoding frame rate of the video frame corresponding to the target number of frames;

[0146] Processing unit 803 is configured to adjust the first encoded frame rate based on the target number of frames when the target number of frames meets the frame rate adjustment condition, so as to obtain a second encoded frame rate.

[0147] The processing unit 803 is further configured to perform video encoding processing on the application screen based on the second encoding frame rate to obtain a target video frame, and send the target video frame to the client.

[0148] In one implementation, the processing unit 803 is specifically used for:

[0149] When the target number of frames is greater than the first threshold, a first frame rate adjustment value is determined based on the target number of frames.

[0150] The first encoded frame rate is reduced by the first frame rate adjustment value to obtain the second encoded frame rate.

[0151] In one implementation, the frame information is the frame information for the i-th time period, and the target frame count is used to indicate the difference between the number of video frames received by the client and the number of decoded video frames within the i-th time period; the first encoding frame rate is the encoding frame rate for obtaining video frames by performing video encoding processing on the application screen within the i-th time period; the processing unit 803 is specifically used for:

[0152] Obtain the period length corresponding to the i-th time period, and determine the first frame rate adjustment value based on the period length and the target number of frames;

[0153] The second encoding frame rate is the encoding frame rate of the video frame obtained by performing video encoding processing on the application screen in the (i+1)th time period.

[0154] In one implementation, the frame information is the frame information of the i-th time period, and the target frame number is used to indicate the difference between the number of video frames received by the client and the number of video frames decoded in the i-th time period; the first encoding frame rate is the encoding frame rate of the video frames obtained by performing video encoding processing on the application screen in the i-th time period.

[0155] The processing unit 803 is specifically used for:

[0156] Obtain the historical frame count, which is used to indicate the difference between the number of video frames received by the client and the number of video frames decoded in the (i-1)th time period;

[0157] Obtain the period length corresponding to the i-th time period, and determine the first frame rate adjustment value based on the period length, the number of historical frames, and the target number of frames.

[0158] The second encoding frame rate is the encoding frame rate of the video frame obtained by performing video encoding processing on the application screen in the (i+1)th time period.

[0159] In one implementation, the processing unit 803 is specifically used for:

[0160] Obtain the historical number of hoarded frames, and determine the total number of hoarded frames based on the historical number of hoarded frames and the target number of hoarded frames. The historical number of hoarded frames is the number of hoarded frames indicated by the hoarded frame information previously sent by the client.

[0161] When the total number of frames exceeds the second threshold, a second frame rate adjustment value is determined based on the total number of frames.

[0162] The second encoding frame rate is obtained by reducing the first encoding frame rate by the second frame rate adjustment value.

[0163] In one implementation, the processing unit 803 is specifically used for:

[0164] When the number of frames indicated by the K consecutive received frame information is less than or equal to the third threshold, the third frame rate adjustment value is obtained; K is a positive integer.

[0165] The first encoded frame rate is increased by the third frame rate adjustment value to obtain the second encoded frame rate.

[0166] Please see Figure 9 , Figure 9 This is a schematic diagram of another data processing apparatus provided in an embodiment of this application. Optionally, this data processing apparatus can be disposed in the aforementioned terminal device. Figure 9 As shown, the data processing apparatus described in this embodiment may include:

[0167] The acquisition unit 901 is used to acquire the number of video frames received and the number of video frames decoded within a preset time range; wherein, the video frames received within the preset time range are video frames obtained by the server performing video encoding processing based on a first encoding frame rate;

[0168] Processing unit 902 is used to determine frame-stacking information based on the difference between the number of video frames received and the number of video frames decoded within the preset time range, and to send the frame-stacking information to the server;

[0169] The receiving unit 903 is used to receive video frames sent by the server, which are obtained by video encoding processing based on a second encoding frame rate, perform video decoding processing on the received video frames to obtain the corresponding application screen, and display the application screen obtained by video decoding processing; wherein, the second encoding frame rate is obtained by the server adjusting the first encoding frame rate based on the frame information.

[0170] In one implementation, the frame information is the frame information of the i-th time period, and the processing unit 902 is specifically used for:

[0171] At the end of the i-th time period, determine the number of received video frames and the number of decoded video frames received during the i-th time period.

[0172] The target number of stored frames is determined based on the difference between the number of received frames and the number of decoded frames, and the stored frame information for the i-th time period is determined based on the target number of stored frames.

[0173] Please see Figure 10 , Figure 10This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device can be the server described above. The electronic device described in this embodiment includes: a processor 1001 and a memory 1002. Optionally, the electronic device may also include a network interface or a power supply module, etc. The processor 1001 and the memory 1002 can exchange data. The processor 1001 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0174] The aforementioned network interface may include input devices and / or output devices. For example, the input device may be a control panel, microphone, receiver, etc., and the output device may be a display screen, transmitter, etc., which will not be listed here.

[0175] The aforementioned memory 1002 may include read-only memory and random access memory, and provides program instructions and data to the processor 1001. A portion of the memory 1002 may also include non-volatile random access memory. The processor 1001 executes the program instructions by calling them:

[0176] The system receives frame-stacking information sent by the client, which indicates the target number of frame-stacking frames. The target number of frame-stacking frames is the difference between the number of video frames received by the client and the number of video frames decoded within a preset time range.

[0177] Obtain a first encoding frame rate, which is used to indicate the encoding frame rate of the video frames corresponding to the target number of frames;

[0178] When the target number of frames meets the frame rate adjustment condition, the first encoding frame rate is adjusted based on the target number of frames to obtain the second encoding frame rate;

[0179] The application screen is video encoded based on the second encoding frame rate to obtain the target video frame, and the target video frame is sent to the client.

[0180] In one implementation, the processor 1001 is specifically used for:

[0181] When the target number of frames is greater than the first threshold, a first frame rate adjustment value is determined based on the target number of frames.

[0182] The first encoded frame rate is reduced by the first frame rate adjustment value to obtain the second encoded frame rate.

[0183] In one implementation, the frame information is the frame information for the i-th time period, and the target frame count is used to indicate the difference between the number of video frames received by the client and the number of decoded video frames within the i-th time period; the first encoding frame rate is the encoding frame rate for obtaining video frames by performing video encoding processing on the application screen within the i-th time period; the processor 1001 is specifically used for:

[0184] Obtain the period length corresponding to the i-th time period, and determine the first frame rate adjustment value based on the period length and the target number of frames;

[0185] The second encoding frame rate is the encoding frame rate of the video frame obtained by performing video encoding processing on the application screen in the (i+1)th time period.

[0186] In one implementation, the frame information is the frame information of the i-th time period, and the target frame number is used to indicate the difference between the number of video frames received by the client and the number of video frames decoded in the i-th time period; the first encoding frame rate is the encoding frame rate of the video frames obtained by performing video encoding processing on the application screen in the i-th time period.

[0187] The processor 1001 is specifically used for:

[0188] Obtain the historical frame count, which is used to indicate the difference between the number of video frames received by the client and the number of video frames decoded in the (i-1)th time period;

[0189] Obtain the period length corresponding to the i-th time period, and determine the first frame rate adjustment value based on the period length, the number of historical frames, and the target number of frames.

[0190] The second encoding frame rate is the encoding frame rate of the video frame obtained by performing video encoding processing on the application screen in the (i+1)th time period.

[0191] In one implementation, the processor 1001 is specifically used for:

[0192] Obtain the historical number of hoarded frames, and determine the total number of hoarded frames based on the historical number of hoarded frames and the target number of hoarded frames. The historical number of hoarded frames is the number of hoarded frames indicated by the hoarded frame information previously sent by the client.

[0193] When the total number of frames exceeds the second threshold, a second frame rate adjustment value is determined based on the total number of frames.

[0194] The second encoding frame rate is obtained by reducing the first encoding frame rate by the second frame rate adjustment value.

[0195] In one implementation, the processor 1001 is specifically used for:

[0196] When the number of frames indicated by the K consecutive received frame information is less than or equal to the third threshold, the third frame rate adjustment value is obtained; K is a positive integer.

[0197] The first encoded frame rate is increased by the third frame rate adjustment value to obtain the second encoded frame rate.

[0198] Please see Figure 11 , Figure 11 This is a schematic diagram of another electronic device provided in an embodiment of this application. The electronic device can be the terminal device described above. The electronic device described in this embodiment includes: a processor 1101 and a memory 1102. Optionally, the electronic device may also include a network interface or a power supply module, etc. The processor 1101 and the memory 1102 can exchange data. The processor 1101 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0199] The aforementioned network interface may include input devices and / or output devices. For example, the input device may be a control panel, microphone, receiver, etc., and the output device may be a display screen, transmitter, etc., which will not be listed here.

[0200] The aforementioned memory 1102 may include read-only memory and random access memory, and provides program instructions and data to the processor 1101. A portion of the memory 1102 may also include non-volatile random access memory. The processor 1101 executes the program instructions when it calls them:

[0201] The number of video frames received and the number of video frames decoded within a preset time range are obtained; wherein, the video frames received within the preset time range are video frames obtained by the server performing video encoding processing based on a first encoding frame rate.

[0202] The frame-stacking information is determined based on the difference between the number of video frames received and the number of video frames decoded within the preset time range, and the frame-stacking information is sent to the server.

[0203] The system receives video frames sent by the server, which are obtained by video encoding based on a second encoding frame rate. It then performs video decoding on the received video frames to obtain the corresponding application screen and displays the application screen obtained by video decoding. The second encoding frame rate is obtained by the server adjusting the first encoding frame rate based on the frame information.

[0204] In one implementation, the frame information is the frame information of the i-th time period, and the processor 1101 is specifically used for:

[0205] At the end of the i-th time period, determine the number of received video frames and the number of decoded video frames received during the i-th time period.

[0206] The target number of stored frames is determined based on the difference between the number of received frames and the number of decoded frames, and the stored frame information for the i-th time period is determined based on the target number of stored frames.

[0207] Optionally, when the program instructions are executed by the processor, other steps of the method in the above embodiments can also be implemented, which will not be described in detail here.

[0208] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the above-described method, such as the method performed by the above-described electronic device, which will not be elaborated here.

[0209] Optionally, the storage medium involved in this application, such as a computer-readable storage medium, may be non-volatile or volatile.

[0210] Optionally, the computer-readable storage medium may primarily include a stored program area and a stored data area. The stored program area may store the operating system, at least one application program required for a given function, etc.; the stored data area may store data created based on the use of blockchain nodes, etc. Here, the blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. A blockchain is essentially a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain may include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.

[0211] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0212] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0213] This application also provides a computer program product or computer program that includes computer instructions that, when executed by a processor, can implement some or all of the steps in the methods described above. For example, the computer instructions are stored in a computer-readable storage medium. The processor of a computer device (i.e., the electronic device described above) reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps executed in the embodiments of the methods described above. For example, the computer device can be a terminal device or a server.

[0214] The data processing method, apparatus, electronic device, and medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, The method includes: The system receives frame-stacking information sent by the client. The frame-stacking information is used to indicate the target number of frame-stacking frames. The target number of frame-stacking frames is the difference between the number of video frames received by the client and the number of video frames decoded within a preset time range. The preset time range is the time range indicated by a time period. Obtain a first encoding frame rate, which is used to indicate the encoding frame rate of the video frames corresponding to the target number of frames; Obtain the historical number of hoarded frames, and determine the total number of hoarded frames based on the historical number of hoarded frames and the target number of hoarded frames. The historical number of hoarded frames is the number of hoarded frames indicated by the hoarded frame information previously sent by the client. Obtain the period length corresponding to the preset time range; When the total number of frames exceeds the second threshold, a second frame rate adjustment value is determined based on the total number of frames and the period length. The first encoding frame rate is reduced by the second frame rate adjustment value to obtain the second encoding frame rate; The application screen is video encoded based on the second encoding frame rate to obtain the target video frame, and the target video frame is sent to the client.

2. The method according to claim 1, characterized in that, The frame information is the frame information of the i-th time period, and the target frame number is used to indicate the difference between the number of video frames received by the client and the number of video frames decoded in the i-th time period; the first encoding frame rate is the encoding frame rate of the video frames obtained by performing video encoding processing on the application screen in the i-th time period. The second encoding frame rate is the encoding frame rate of the video frame obtained by performing video encoding processing on the application screen in the (i+1)th time period.

3. The method according to claim 1, characterized in that, The historical frame count is used to indicate the difference between the number of video frames received by the client and the number of video frames decoded in the (i-1)th time period; the period length is the period length corresponding to the i-th time period.

4. The method according to claim 1, characterized in that, The method further includes: When the number of frames indicated by the K consecutive received frame information is less than or equal to the third threshold, the third frame rate adjustment value is obtained; K is a positive integer. The third encoding frame rate is obtained by adding the third frame rate adjustment value to the second encoding frame rate.

5. A data processing method, characterized in that, The method includes: The number of video frames received and the number of video frames decoded within a preset time range are obtained; wherein, the video frames received within the preset time range are video frames obtained by the server performing video encoding processing based on a first encoding frame rate; and the preset time range is the time range indicated by a time period. The frame-stacking information is determined based on the difference between the number of video frames received and the number of video frames decoded within the preset time range, and the frame-stacking information is sent to the server. The system receives video frames from the server, processed by video encoding based on a second encoding frame rate. It then performs video decoding on the received video frames to obtain the corresponding application screen and displays the decoded application screen. The second encoding frame rate is determined as follows: The system obtains the historical number of stored frames and determines the total number of stored frames based on the historical number of stored frames and the target number of stored frames indicated by the stored frame information. The historical number of stored frames is the number of stored frames indicated by the stored frame information previously sent by the client. The system also obtains the period length corresponding to the preset time range. When the total number of stored frames exceeds a second threshold, a second frame rate adjustment value is determined based on the total number of stored frames and the period length. Finally, the system reduces the first encoding frame rate by the second frame rate adjustment value to obtain the second encoding frame rate.

6. The method according to claim 5, characterized in that, The frame information is the frame information for the i-th time period. The step of obtaining the number of video frames received and the number of video frames decoded within a preset time range includes: At the end of the i-th time period, determine the number of received video frames and the number of decoded video frames received during the i-th time period. The step of determining the frame information based on the difference between the number of video frames received and the number of video frames decoded within the preset time range includes: The target number of stored frames is determined based on the difference between the number of received frames and the number of decoded frames, and the stored frame information for the i-th time period is determined based on the target number of stored frames.

7. A data processing apparatus, characterized in that, The device includes: The receiving unit is used to receive frame-stacking information sent by the client. The frame-stacking information is used to indicate the target number of frame-stacking frames. The target number of frame-stacking frames is the difference between the number of video frames received by the client and the number of video frames decoded within a preset time range. The preset time range is the time range indicated by a time period. An acquisition unit is used to acquire a first encoding frame rate, wherein the first encoding frame rate is used to indicate the encoding frame rate of the video frames corresponding to the target number of frames; A processing unit is configured to: acquire the historical number of hoarded frames and determine the total number of hoarded frames based on the historical number of hoarded frames and the target number of hoarded frames, wherein the historical number of hoarded frames is the number of hoarded frames indicated by the hoarded frame information previously sent by the client; acquire the period length corresponding to the preset time range; when the total number of hoarded frames is greater than a second threshold, determine a second frame rate adjustment value based on the total number of hoarded frames and the period length; and reduce the first encoded frame rate by the second frame rate adjustment value to obtain a second encoded frame rate. The processing unit is further configured to perform video encoding processing on the application screen based on the second encoding frame rate to obtain a target video frame, and send the target video frame to the client.

8. A data processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire the number of video frames received and the number of video frames decoded within a preset time range; wherein, the video frames received within the preset time range are video frames obtained by the server performing video encoding processing based on a first encoding frame rate; and the preset time range is the time range indicated by a time period. The processing unit is configured to determine the frame-stacking information based on the difference between the number of video frames received and the number of video frames decoded within the preset time range, and send the frame-stacking information to the server. A receiving unit is configured to receive video frames sent by the server, which are obtained by video encoding processing based on a second encoding frame rate; perform video decoding processing on the received video frames to obtain the corresponding application screen; and display the application screen obtained by video decoding processing. The second encoding frame rate is determined as follows: the number of historical stored frames is obtained, and the total number of stored frames is determined based on the historical stored frame number and the target stored frame number indicated by the stored frame information, wherein the historical stored frame number is the number of stored frames indicated by the stored frame information previously sent by the client; the period length corresponding to the preset time range is obtained; when the total number of stored frames is greater than a second threshold, a second frame rate adjustment value is determined based on the total number of stored frames and the period length; the first encoding frame rate is reduced by the second frame rate adjustment value to obtain the second encoding frame rate.

9. An electronic device, characterized in that, The system includes a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute the method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-6.

11. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

Images