Video coding method, device, equipment, medium and program product

Abstract

The embodiment of the invention provides a video coding method and device, equipment, a medium and a program product. The method comprises the following steps: obtaining a first code stream corresponding to a video to be coded, wherein the first code stream is obtained after coding based on a first coding mode; based on the first code stream feature information and a first frame rate prediction model, predicting a first decoding frame rate corresponding to a second code stream, the second code stream being obtained after encoding based on a second encoding mode, and the first decoding frame rate being a video frame rate obtained after decoding the second code stream, the coding and decoding complexity of the second coding mode is greater than the coding and decoding complexity of the first coding mode; and comparing the first decoding frame rate with a first frame rate threshold, and determining whether to encode the to-be-encoded video based on the second encoding mode based on a comparison result. Through the technical scheme of the embodiment of the invention, unnecessary coding calculation can be reduced, and the situation that the coded video cannot be normally played is avoided, so that the calculation resources are saved, and the user experience is improved.

Description

Technical Field

[0001] This disclosure relates to computer technology, and more particularly to a video encoding method, apparatus, device, medium, and program product. Background Technology

[0002] With the rapid development of computer technology, servers typically need to encode videos and then send the encoded stream to clients for decoding and playback. Currently, to achieve high compression rates and better video quality, video encoding methods with high encoding complexity are commonly used. However, in the process of implementing this disclosure, at least the following problems were found in the existing technology:

[0003] This encoding method, with its high complexity, consumes more computing resources and has a high decoding complexity. This can result in a video frame rate that may not meet the requirements for real-time playback, leading to video playback stuttering and severely impacting the user experience. Summary of the Invention

[0004] This disclosure provides a video encoding method, apparatus, device, medium, and program product to reduce unnecessary encoding calculations, avoid situations where encoded videos cannot be played normally, thereby saving computing resources and improving user experience.

[0005] In a first aspect, embodiments of this disclosure provide a video encoding method, including:

[0006] Obtain the video to be encoded and the first bitstream corresponding to the video to be encoded, wherein the first bitstream is obtained by encoding the video to be encoded based on a first encoding method;

[0007] Based on the first bitstream feature information and the first frame rate prediction model corresponding to the first bitstream, the first decoding frame rate corresponding to the second bitstream is predicted. The second bitstream is obtained by encoding the video to be encoded based on the second encoding method. The first decoding frame rate is the video frame rate obtained by decoding the second bitstream based on the decoding method corresponding to the second encoding method. The encoding and decoding complexity of the second encoding method is greater than that of the first encoding method.

[0008] The first decoding frame rate is compared with the first frame rate threshold, and based on the comparison result, it is determined whether to encode the video to be encoded based on the second encoding method.

[0009] Secondly, embodiments of this disclosure also provide a video encoding apparatus, comprising:

[0010] The first bitstream acquisition module is used to acquire the video to be encoded and the first bitstream corresponding to the video to be encoded. The first bitstream is obtained by encoding the video to be encoded based on a first encoding method.

[0011] The first decoding frame rate prediction module is used to predict the first decoding frame rate corresponding to the second bitstream based on the first bitstream feature information and the first frame rate prediction model corresponding to the first bitstream. The second bitstream is obtained by encoding the video to be encoded based on the second encoding method, and the first decoding frame rate is the video frame rate obtained by decoding the second bitstream based on the decoding method corresponding to the second encoding method. The encoding and decoding complexity of the second encoding method is greater than that of the first encoding method.

[0012] The video encoding control module is used to compare the first decoding frame rate with the first frame rate threshold, and determine whether to encode the video to be encoded based on the comparison result.

[0013] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0014] One or more processors;

[0015] Storage device for storing one or more programs.

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the video encoding method as described in any of the embodiments of this disclosure.

[0017] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the video encoding method as described in any of the embodiments of this disclosure.

[0018] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the video encoding method as described in any of the embodiments of the present disclosure.

[0019] In this embodiment, a first bitstream corresponding to the video to be encoded is obtained. This first bitstream is obtained by encoding the video to be encoded using a first encoding method with low encoding and decoding complexity. Then, using the feature information of the first bitstream obtained by the first encoding method with low encoding and decoding complexity and a first frame rate prediction model, the first decoded frame rate of the second bitstream obtained by the second encoding method with high encoding and decoding complexity is predicted. The predicted first decoded frame rate is compared with a first frame rate threshold. Based on the comparison result, it can be determined whether the first decoded frame rate can meet the requirements of real-time playback. This determines whether it is necessary to encode the video to be encoded using the second encoding method with high encoding and decoding complexity. This reduces unnecessary encoding calculations and avoids situations where the encoded video cannot be played normally, thereby saving computing resources and improving the user experience. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 This is a schematic flowchart of a video encoding method provided in an embodiment of this disclosure;

[0022] Figure 2 This is a schematic flowchart of another video encoding method provided in an embodiment of this disclosure;

[0023] Figure 3 This is a schematic diagram of the structure of a video encoding device provided in an embodiment of this disclosure;

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0025] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0026] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0027] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0032] Figure 1 This is a flowchart illustrating a video encoding method provided in an embodiment of the present disclosure. This embodiment is applicable to situations where a decision is made on whether to use an encoding method with high encoding and decoding complexity for video encoding. The method can be executed by a video encoding device, which can be implemented in software and / or hardware, or optionally by an electronic device, such as a server.

[0033] like Figure 1 As shown, the video encoding method specifically includes the following steps:

[0034] S110. Obtain the video to be encoded and the first bitstream corresponding to the video to be encoded. The first bitstream is obtained after encoding the video to be encoded based on the first encoding method.

[0035] The video to be encoded can refer to the video that needs transcoding on the server. Transcoding refers to secondary encoding of the video so that the encoded stream can be sent to the client for playback. The first encoding method can be an encoding method with lower encoding / decoding complexity than the second encoding method. The second encoding method can be an encoding method with high encoding / decoding complexity, which severely consumes computing resources. It should be noted that both the first and second encoding methods can use existing encoding technologies; the second encoding method can refer to a high-complexity encoding method. Correspondingly, the first encoding method can refer to a low-complexity encoding method. The second encoding method can achieve a higher compression ratio and better picture quality than the first encoding method, but its encoding complexity is several times that of the first encoding method, consuming more computing resources, and its corresponding decoding complexity is also higher, resulting in a lower decoding frame rate. It should be noted that the higher the encoding complexity of the encoding method, the higher the corresponding decoding complexity, thus resulting in higher compression ratio and video picture quality, but also higher computing resource consumption and longer encoding / decoding time.

[0036] For example, the first encoding method may include a pre-encoding method or an existing encoding method. A pre-encoding method can refer to an encoding operation performed before the final encoding, i.e., a pre-encoding operation performed before the final encoding based on the second encoding method. An existing encoding method can refer to other final encoding methods besides the second encoding method. For instance, the same video often needs to be encoded multiple times using various different encoding methods. For example, in addition to using the second encoding method for final encoding, the first encoding method may also be used. In this case, the first encoding method can be used as an existing encoding method, allowing for direct use of the existing bitstream obtained from the existing encoding method for subsequent processing, eliminating the need for an additional pre-encoding and further saving computational resources.

[0037] For example, when the first encoding method is precoding, the encoding speed of the precoding method is greater than that of the second encoding method, and the video resolution of the precoding method is smaller than that of the second encoding method. For instance, if the second encoding method has a slow encoding speed and a video resolution of 1080p, then the precoding method has a veryfast encoding speed and a video resolution of 480p. This results in a shorter overall encoding process for the precoding method, improving encoding efficiency and further saving computational resources.

[0038] Specifically, the server parses the received transcoding request to obtain the video to be encoded and its target resolution. It can then quickly encode the video using a pre-encoding method to obtain the first bitstream. Alternatively, if an existing encoding method exists besides the second encoding method, the bitstream obtained by encoding the video using the existing method can be directly used as the first bitstream, eliminating the need for additional pre-encoding and further saving computational resources.

[0039] S120. Based on the first bitstream feature information and the first frame rate prediction model corresponding to the first bitstream, predict the first decoding frame rate corresponding to the second bitstream. The second bitstream is obtained after encoding the video to be encoded using the second encoding method. The first decoding frame rate is the video frame rate obtained after decoding the second bitstream using the decoding method corresponding to the second encoding method. The encoding and decoding complexity of the second encoding method is greater than that of the first encoding method.

[0040] The first decoding frame rate refers to the predicted decoding frame rate before video encoding based on the second encoding method. The decoding frame rate characterizes the actual decoding complexity of the second bitstream on the client side. For example, higher decoding complexity results in a lower decoded video frame rate. Since the decoding frame rates of different video content on the same client can vary by several times, the server cannot predict the decoding frame rate on the client side; the actual decoding frame rate can only be obtained during actual playback on the client. For example, video content with high motion and high texture has higher decoding complexity, resulting in a lower decoding frame rate, while static and simple video content has lower decoding complexity, resulting in a higher decoding frame rate. Therefore, some videos encoded using the second encoding method have a decoding frame rate lower than the frame rate required for real-time playback, leading to stuttering during playback and a reduced user experience. For example, 1080p HD and 4K Ultra HD videos encoded using the second encoding method have a decoding frame rate of less than 20fps, which cannot meet the requirements for real-time playback.

[0041] The first bitstream feature information refers to the bitstream feature information in the first bitstream that is related to the decoding frame rate. The first bitstream feature information can be used to characterize different video content. For example, the first bitstream feature information may include at least one of the following: coding mode feature information, bit type feature information, and distortion feature information. The coding mode feature information may include, but is not limited to: the proportion of intra-frame mode in P-frames and B-frames, the proportion of skip mode in P-frames and B-frames, the proportion of merge mode in P-frames and B-frames, the average size and type of CU (Coding Unit) in I-frames, P-frames, and B-frames, and the average length and direction of MV (Motion Vector) in I-frames, P-frames, and B-frames. The bit type feature information may include, but is not limited to: the proportion of residual bitrate in I-frames, P-frames, and B-frames to the total bitrate. The distortion feature information may include, but is not limited to: the sum of the absolute errors of the residuals in I-frames, P-frames, and B-frames, and the sum of the non-zero coefficients of the Discrete Cosine Transform (DCT) residuals in I-frames, P-frames, and B-frames.

[0042] The first frame rate prediction model is a network model that predicts the decoding frame rate of the second bitstream in the client based on the bitstream feature information obtained from a first encoding method with low encoding and decoding complexity. The network architecture of the first frame rate prediction model can be any regression network model, such as Support Vector Regression (SVR), decision trees, random forests, and deep neural network models. The first frame rate prediction model is obtained through supervised training based on the sample bitstream feature information of the first sample bitstream and the actual decoding frame rate of the second sample bitstream to ensure the prediction accuracy of the first frame rate prediction model. The first sample bitstream can be obtained by encoding a sample video using the first encoding method. The second sample bitstream can be obtained by encoding a sample video using the second encoding method. The actual decoding frame rate is the actual video frame rate obtained after the second sample bitstream is sent to the client for decoding, and it serves as the true value for model training.

[0043] Specifically, the feature information of the first bitstream using the second encoding method is input into a pre-trained first frame rate prediction model to predict the decoding frame rate of the second bitstream, and the first decoding frame rate output by the first frame rate prediction model is obtained. Thus, by using the first frame rate prediction model, based on the feature information of the first bitstream obtained by the first encoding method with low encoding and decoding complexity, the first decoding frame rate of the second bitstream obtained by the second encoding method with high encoding and decoding complexity can be automatically and quickly predicted, thereby allowing the decoding status of the second bitstream to be known in advance.

[0044] For example, besides the video content affecting the decoding frame rate, the required video resolution and client hardware performance during encoding can also influence the decoding frame rate. For instance, a higher video resolution or lower client hardware performance will result in a lower decoding frame rate. Based on this, different first frame rate prediction models can be trained for different video resolutions and client hardware performance information. In real-time use, based on the target resolution required for the video to be encoded and the hardware performance information of the client to be distributed, a target first frame rate prediction model capable of predicting the first decoding frame rate under the given hardware performance information and target resolution can be determined from multiple pre-trained first frame rate prediction models. The first bitstream feature information is then input into the target first frame rate prediction model for decoding frame rate prediction, thereby more accurately obtaining the first decoding frame rate corresponding to the second bitstream. This first decoding frame rate refers to the video frame rate obtained after decoding the second bitstream with the target resolution in the client to be distributed, further improving the accuracy of encoding decisions.

[0045] S130. Compare the first decoding frame rate with the first frame rate threshold, and determine whether to encode the video to be encoded based on the comparison result.

[0046] The first frame rate threshold is determined based on the minimum frame rate required for real-time video playback. The first frame rate threshold is greater than or equal to this minimum frame rate. To minimize transcoding and error tolerance in the second encoding method, the first frame rate threshold can be set to be greater than the minimum frame rate required for real-time video playback. For example, if real-time playback of a 60fps video is required, the first frame rate threshold can be set to 80fps.

[0047] Specifically, by comparing the first decoding frame rate with the first frame rate threshold, it can be determined whether the first decoding frame rate can meet the requirements of real-time playback, and then determine whether it is necessary to encode the video to be encoded using the second encoding method. This can effectively control the amount of encoding computation, reduce unnecessary encoding computation on the server side, save computing resources, and also avoid playback stuttering in the encoded video, thereby improving the user experience.

[0048] For example, step S130 may include: stopping encoding the video to be encoded based on the second encoding method in response to the first decoding frame rate being less than the first frame rate threshold; and allowing encoding of the video to be encoded based on the second encoding method to obtain the second bitstream in response to the first decoding frame rate being greater than or equal to the first frame rate threshold.

[0049] Specifically, if the first decoding frame rate is less than the first frame rate threshold, it indicates that the second bitstream encoded using the second encoding method cannot be decoded and played normally on the client. To avoid impacting user experience, the encoding of the video to be encoded using the second encoding method can be stopped, i.e., the formal encoding of the video using the second encoding method is skipped, thus saving computational resources. If the first decoding frame rate is greater than or equal to the first frame rate threshold, it indicates that the second bitstream encoded using the second encoding method can be decoded and played normally on the client. In this case, the video to be encoded can be encoded using the second encoding method to obtain the second bitstream. By skipping the transcoding of videos with high decoding complexity using the second encoding method, transcoding of videos with high decoding complexity can be avoided, thereby realizing dynamic decision-making in video encoding and greatly saving the computational resources consumed by the second encoding method.

[0050] The technical solution of this disclosure involves obtaining a first bitstream corresponding to the video to be encoded. This first bitstream is obtained by encoding the video to be encoded using a first encoding method with low encoding and decoding complexity. Then, using the feature information of the first bitstream obtained by the first encoding method with low encoding and decoding complexity and a first frame rate prediction model, the first decoded frame rate of the second bitstream obtained by the second encoding method with high encoding and decoding complexity is predicted. The predicted first decoded frame rate is compared with a first frame rate threshold. Based on the comparison result, it can be determined whether the first decoded frame rate can meet the requirements of real-time playback. This determines whether it is necessary to encode the video to be encoded using the second encoding method with high encoding and decoding complexity, thereby reducing unnecessary encoding calculations and avoiding situations where the encoded video cannot be played normally, thus saving computing resources and improving the user experience.

[0051] Based on the above technical solution, step S120 may include: obtaining target feature information corresponding to the video to be encoded, the target feature information including at least one of the spatiotemporal complexity of the video to be encoded, target resolution, and client hardware performance information; predicting the first decoding frame rate corresponding to the second bitstream based on the first bitstream feature information, target feature information, and first frame rate prediction model.

[0052] The spatiotemporal complexity can include the temporal perceptual information (TI) and spatial perceptual information (SI) of the video to be encoded. Temporal perceptual information characterizes the temporal changes in a video sequence. Video sequences with higher levels of motion typically have higher TI values. Spatial perceptual information characterizes the spatial changes in a video sequence. Spatially more complex video scenes have higher SI values. Target resolution refers to the required video resolution to which the video to be encoded will be encoded. Higher resolution requirements result in higher decoding complexity and lower decoding frame rates. Client hardware performance information refers to the hardware performance of the video playback device itself. Client hardware performance information reflects the client's decoding performance and also affects the video's decoding frame rate.

[0053] Specifically, the target resolution required for the video to be encoded can be obtained from the transcoding request. The time and space complexity of the video to be encoded is obtained by extracting its time and space complexity. For example, the time complexity extraction process is as follows: Differential processing is performed on every two adjacent video frames in the n-frame video to be encoded to obtain n-1 difference images, i.e., n-1 difference matrices. The average of each difference matrix is ​​taken to obtain n-1 difference mean values, and these n-1 difference mean values ​​are averaged again. The final average value is used as the time complexity of the video to be encoded. The space complexity extraction process is as follows: Sobel operator processing is performed on each video frame in the n-frame video to be encoded to obtain n space matrices. The average of each space matrix is ​​taken to obtain n space mean values, and these n space mean values ​​are averaged again. The final average value is used as the space complexity of the video to be encoded. The server can send a performance acquisition request to the client, so that the client can return client hardware performance information based on the request. Alternatively, the client can also send its hardware performance information to the server by adding it to the video playback request. After obtaining the spatiotemporal complexity, target resolution, and client hardware performance information of the video to be encoded, the server can simultaneously input the first bitstream feature information, spatiotemporal complexity, target resolution, and client hardware performance information into a pre-trained first frame rate prediction model to predict the decoding frame rate. This allows for a more accurate prediction of the first decoding frame rate corresponding to the second bitstream. It should be noted that the more types of input information the first frame rate prediction model receives, the higher its prediction accuracy. This first frame rate prediction model is obtained through supervised training based on the sample bitstream feature information and sample feature information of the first sample bitstream, as well as the actual decoding frame rate of the second sample bitstream, to ensure its prediction accuracy. Sample feature information can include at least one of the sample video's spatiotemporal complexity, target resolution, and client hardware performance information. The types of input information must remain consistent between the model training and model usage phases to ensure the accuracy of the model's predictions.

[0054] Figure 2 This is a flowchart illustrating another video encoding method provided in this disclosure. Based on the above-described embodiments, this disclosure provides a detailed description of the process for transmitting the second bitstream obtained by encoding using a second encoding method. Explanations of terms that are the same as or corresponding to those in the above-described embodiments are not repeated here.

[0055] like Figure 2 As shown, the video encoding method specifically includes the following steps:

[0056] S210. Obtain the video to be encoded and the first bitstream corresponding to the video to be encoded. The first bitstream is obtained after encoding the video to be encoded based on the first encoding method.

[0057] S220. Based on the first bitstream feature information and the first frame rate prediction model corresponding to the first bitstream, predict the first decoding frame rate corresponding to the second bitstream. The second bitstream is obtained after encoding the video to be encoded based on the second encoding method. The first decoding frame rate is the video frame rate obtained after decoding the second bitstream based on the decoding method corresponding to the second encoding method. The encoding and decoding complexity of the second encoding method is greater than that of the first encoding method.

[0058] S230, In response to the first decoding frame rate being greater than or equal to the first frame rate threshold, the video to be encoded is allowed to be encoded based on the second encoding method to obtain the second bitstream.

[0059] It should be noted that when the first decoding frame rate is less than the first frame rate threshold, the second bitstream cannot be obtained because the video to be encoded is not allowed to be encoded using the second encoding method. In this case, the first bitstream, which is formally encoded using an existing encoding method other than the second encoding, can be sent to the client. Since the encoding and decoding complexity of the existing encoding method is low, the client can decode and play the first bitstream normally, thereby meeting the requirements for real-time video playback.

[0060] S240. Based on the second bitstream feature information and the second frame rate prediction model corresponding to the second bitstream, predict the second decoding frame rate corresponding to the second bitstream.

[0061] The second decoding frame rate refers to the predicted decoding frame rate after video encoding based on the second encoding method. The difference between the first and second decoding frame rates lies in the different prediction stages. Second bitstream feature information refers to the bitstream feature information in the second bitstream that is related to the decoding frame rate. This second bitstream feature information can be used to characterize different video content. For example, the second bitstream feature information may include at least one of the following: encoding mode feature information, bit type feature information, and distortion feature information. The encoding mode feature information may include, but is not limited to: the proportion of intra-frame Intra mode in P-frames and B-frames, the proportion of skip mode in P-frames and B-frames, the proportion of merge mode in P-frames and B-frames, the average size and type of CU in I-frames, P-frames, and B-frames, and the average length and direction of MV in I-frames, P-frames, and B-frames. The bit type feature information may include, but is not limited to: the proportion of residual bitrate in I-frames, P-frames, and B-frames to the total bitrate. Distortion feature information may include, but is not limited to: the sum of the absolute errors of the residuals of I-frames, P-frames and B-frames, and the sum of the non-zero coefficients of the discrete cosine transform residuals of I-frames, P-frames and B-frames.

[0062] The second frame rate prediction model is a network model that predicts the decoding frame rate of the second bitstream in the client based on the feature information of the second bitstream. The network architecture of the second frame rate prediction model can be any regression network model, such as Support Vector Regression (SVR), decision trees, random forests, and deep neural network models. The second frame rate prediction model is obtained through supervised training based on the sample bitstream feature information and the actual decoding frame rate of the second sample bitstream to ensure the prediction accuracy. The second sample bitstream can be obtained by encoding sample videos using a second encoding method. The actual decoding frame rate is the actual video frame rate obtained after the second sample bitstream is sent to the client for decoding, and it serves as the true value for model training.

[0063] Specifically, after allowing the video to be encoded using a second encoding method to obtain a second bitstream, a second frame rate prediction model can be used to predict the second decoding frame rate of the second bitstream on the client side based on the second bitstream's own feature information. This allows for further determination of whether the second bitstream's decoding frame rate meets the requirements for real-time video playback. For example, the second bitstream's feature information can be input into a pre-trained second frame rate prediction model to predict the second bitstream's decoding frame rate, obtaining the second decoding frame rate output by the model. Because the second bitstream's feature information has a higher correlation with the actual decoding frame rate, the second frame rate prediction model can more accurately predict the decoding frame rate.

[0064] For example, different second frame rate prediction models can be trained for different video resolutions and client hardware performance information. In real-time use, based on the target resolution required for the video to be encoded and the hardware performance information of the client to be distributed, a target second frame rate prediction model that can predict the second decoding frame rate under the given hardware performance information and target resolution can be determined from multiple pre-trained second frame rate prediction models. The second bitstream feature information is then input into the target second frame rate prediction model for decoding frame rate prediction, thereby obtaining the second decoding frame rate corresponding to the second bitstream more accurately. This second decoding frame rate refers to the video frame rate obtained after decoding the second bitstream with the target resolution in the client to be distributed, which can further improve the accuracy of the distribution decision.

[0065] S250. Compare the second decoding frame rate with the second frame rate threshold, and determine whether to send the second bitstream based on the comparison result.

[0066] The second frame rate threshold is determined based on the minimum frame rate required for real-time video playback. The second frame rate threshold is greater than or equal to this minimum frame rate. When prioritizing computational resources, the first frame rate threshold can be set higher than the second frame rate threshold, thereby minimizing transcoding using the second encoding method and conserving encoding computational resources. Conversely, when prioritizing bitrate, the second frame rate threshold can be set higher than the first frame rate threshold, thereby maximizing transcoding using the second encoding method, reducing the encoded bitrate, and further saving transmission costs.

[0067] Specifically, by comparing the second decoding frame rate with the second frame rate threshold, it can be determined again whether the second decoding frame rate can meet the requirements of real-time playback, and then it can be determined whether it is necessary to send the second bitstream. This can achieve effective control of bitstream sending, avoid the situation where the sent bitstream cannot meet the requirements of real-time or speed-up playback, which would lead to video not being able to play or stuttering, thereby improving the user experience.

[0068] For example, step S250 may include: stopping the transmission of the second bitstream in response to the second decoding frame rate being less than the second frame rate threshold; and allowing the transmission of the second bitstream to the client in response to the second decoding frame rate being greater than or equal to the second frame rate threshold.

[0069] Specifically, if the second decoding frame rate is less than the second frame rate threshold, it indicates that the second bitstream encoded using the second encoding method cannot be decoded and played normally on the client. To avoid this impacting the user experience, the distribution of the second bitstream can be stopped, i.e., the distribution of the second bitstream can be skipped to prevent the video from failing to play properly due to the second bitstream. If the second decoding frame rate is greater than or equal to the second frame rate threshold, it indicates that the second bitstream encoded using the second encoding method can be decoded and played normally on the client. In this case, the distribution of the second bitstream to the client can be allowed, thus ensuring that the distributed second bitstream meets the requirements for real-time playback. By skipping the distribution of bitstreams for videos with high decoding complexity, the user viewing experience can be guaranteed.

[0070] It should be noted that predicting the first decoding frame rate enables dynamic decision-making for video transcoding, thus achieving initial video screening. Predicting the second decoding frame rate enables dynamic strategies for video delivery, thus achieving fine-tuning. By combining these two stages of prediction and decision-making, unnecessary additional computation can be minimized while ensuring that the video delivered to the user meets the requirements for real-time playback or playback at accelerated speeds.

[0071] The technical solution of this disclosure, after allowing the video to be encoded based on the second encoding method to obtain the second bitstream, predicts the second decoding frame rate corresponding to the second bitstream more accurately based on the second bitstream feature information and the second frame rate prediction model, and compares the second decoding frame rate with the second frame rate threshold. Based on the comparison result, it is determined whether the second bitstream is allowed to be sent, so as to ensure that the bitstream after being sent can be played normally, thereby ensuring the user experience.

[0072] Based on the above technical solution, step S240 may include: obtaining target feature information corresponding to the video to be encoded, the target feature information including at least one of the spatiotemporal complexity of the video to be encoded, target resolution and client hardware performance information; predicting the second decoding frame rate corresponding to the second bitstream based on the second bitstream feature information, the target feature information and the second frame rate prediction model.

[0073] Specifically, similar to the first frame rate prediction model, the second frame rate prediction model can also predict the second decoding frame rate simultaneously based on the second bitstream feature information and the target feature information, thereby further improving the prediction accuracy of the second decoding frame rate. For example, after the server obtains the spatiotemporal complexity, target resolution, and client hardware performance information of the video to be encoded, it can simultaneously input the second bitstream feature information, spatiotemporal complexity, target resolution, and client hardware performance information into the pre-trained second frame rate prediction model to predict the decoding frame rate, which can more accurately obtain the second decoding frame rate corresponding to the second bitstream. It should be noted that the more types of input information the second frame rate prediction model has, the higher the accuracy of the model prediction. This second frame rate prediction model is obtained through supervised training based on the sample bitstream feature information and sample feature information of the second sample bitstream, as well as the actual decoding frame rate of the second sample bitstream, to ensure the prediction accuracy of the second frame rate prediction model. The sample feature information may include at least one of the spatiotemporal complexity, target resolution, and client hardware performance information of the sample video. The types of information input during the model training phase and the model usage phase need to be consistent to ensure the accuracy of the model prediction.

[0074] Figure 3 This is a schematic diagram of the structure of a video encoding device provided in an embodiment of the present disclosure, as shown below. Figure 3 As shown, the device specifically includes: a first bitstream acquisition module 310, a first decoding frame rate prediction module 320, and a video encoding control module 330.

[0075] The system includes a first bitstream acquisition module 310, which acquires the video to be encoded and the corresponding first bitstream. The first bitstream is obtained by encoding the video to be encoded using a first encoding method. A first decoding frame rate prediction module 320 is used to predict the first decoding frame rate corresponding to the second bitstream based on the first bitstream feature information and the first frame rate prediction model. The second bitstream is obtained by encoding the video to be encoded using a second encoding method, and the first decoding frame rate is the video frame rate obtained by decoding the second bitstream using the decoding method corresponding to the second encoding method. The encoding and decoding complexity of the second encoding method is greater than that of the first encoding method. A video encoding control module 330 is used to compare the first decoding frame rate with a first frame rate threshold and determine whether to encode the video to be encoded using the second encoding method based on the comparison result.

[0076] The technical solution provided in this disclosure obtains a first bitstream corresponding to the video to be encoded. This first bitstream is obtained by encoding the video to be encoded using a first encoding method with low encoding and decoding complexity. Then, using the feature information of the first bitstream obtained by the first encoding method with low encoding and decoding complexity and a first frame rate prediction model, the first decoded frame rate of the second bitstream obtained by the second encoding method with high encoding and decoding complexity is predicted. The predicted first decoded frame rate is compared with a first frame rate threshold. Based on the comparison result, it can be determined whether the first decoded frame rate can meet the requirements of real-time playback. This determines whether it is necessary to encode the video to be encoded using the second encoding method with high encoding and decoding complexity, thereby reducing unnecessary encoding calculations and avoiding situations where the encoded video cannot be played normally, thus saving computing resources and improving the user experience.

[0077] Based on the above technical solutions, the first encoding method includes a pre-encoding method or an existing encoding method;

[0078] The encoding speed of the precoding method is greater than that of the second encoding method, while the video resolution of the precoding method is less than that of the second encoding method.

[0079] Based on the above technical solutions, the first decoding frame rate prediction module 320 is specifically used for:

[0080] Obtain target feature information corresponding to the video to be encoded. The target feature information includes at least one of the following: the spatiotemporal complexity of the video to be encoded, the target resolution, and the client hardware performance information. Based on the first bitstream feature information, the target feature information, and the first frame rate prediction model corresponding to the first bitstream, predict the first decoding frame rate corresponding to the second bitstream.

[0081] Based on the above technical solutions, the video encoding control module 330 is specifically used for:

[0082] In response to a first decoding frame rate being less than a first frame rate threshold, encoding of the video to be encoded based on the second encoding method is stopped; in response to a first decoding frame rate being greater than or equal to the first frame rate threshold, encoding of the video to be encoded based on the second encoding method is allowed to obtain a second bitstream.

[0083] Based on the above technical solutions, the device also includes:

[0084] The second decoding frame rate prediction module is used to predict the second decoding frame rate corresponding to the second bitstream after the video to be encoded is allowed to be encoded based on the second encoding method and the second bitstream is obtained.

[0085] The bitstream delivery control module is used to compare the second decoding frame rate with the second frame rate threshold, and determine whether to deliver the second bitstream based on the comparison result.

[0086] Based on the above technical solutions, the second decoding frame rate prediction module is specifically used for:

[0087] Obtain target feature information corresponding to the video to be encoded. The target feature information includes at least one of the following: the spatiotemporal complexity of the video to be encoded, the target resolution, and the client hardware performance information. Based on the second bitstream feature information, the target feature information, and the second frame rate prediction model, predict the second decoding frame rate corresponding to the second bitstream.

[0088] Based on the above technical solutions, the bitstream distribution control module is specifically used for:

[0089] If the second decoding frame rate is less than the second frame rate threshold, stop sending the second bitstream; if the second decoding frame rate is greater than or equal to the second frame rate threshold, allow the second bitstream to be sent to the client.

[0090] Based on the above technical solutions, both the first frame rate threshold and the second frame rate threshold are determined based on the minimum frame rate required for real-time video playback.

[0091] When prioritizing computing resources, the first frame rate threshold is greater than the second frame rate threshold.

[0092] The video encoding apparatus provided in this disclosure can execute the video encoding method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the video encoding method.

[0093] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0094] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 4 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 4 The diagram below shows the structure of the terminal device or server 500. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0095] like Figure 4 As shown, electronic device 500 may include a processing unit (e.g., central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An edit / output (I / O) interface 505 is also connected to bus 504.

[0096] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0097] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0098] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0099] The electronic device provided in this embodiment and the video encoding method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0100] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the video encoding method provided in the above embodiments.

[0101] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0102] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0103] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0104] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a video to be encoded and a first bitstream corresponding to the video to be encoded, wherein the first bitstream is obtained by encoding the video to be encoded based on a first encoding method; predict a first decoding frame rate corresponding to a second bitstream based on the first bitstream feature information and a first frame rate prediction model, wherein the second bitstream is obtained by encoding the video to be encoded based on a second encoding method, and the first decoding frame rate is the video frame rate obtained by decoding the second bitstream based on the decoding method corresponding to the second encoding method, wherein the encoding / decoding complexity of the second encoding method is greater than the encoding / decoding complexity of the first encoding method; compare the first decoding frame rate with a first frame rate threshold, and determine whether to encode the video to be encoded based on the second encoding method based on the comparison result.

[0105] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video encoding method provided in the above embodiments.

[0107] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0109] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0110] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0111] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0112] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0113] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0114] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A video encoding method, characterized in that, include: Obtain the video to be encoded and the first bitstream corresponding to the video to be encoded, wherein the first bitstream is obtained by encoding the video to be encoded based on a first encoding method; Based on the first bitstream feature information and the first frame rate prediction model corresponding to the first bitstream, the first decoding frame rate corresponding to the second bitstream is predicted. The second bitstream is obtained by encoding the video to be encoded based on the second encoding method. The first decoding frame rate is the video frame rate obtained by decoding the second bitstream based on the decoding method corresponding to the second encoding method. The encoding and decoding complexity of the second encoding method is greater than that of the first encoding method. The first decoding frame rate is compared with the first frame rate threshold, and based on the comparison result, it is determined whether to encode the video to be encoded based on the second encoding method.

2. The video encoding method according to claim 1, characterized in that, The first encoding method includes a pre-encoding method or an existing encoding method; The encoding speed of the precoding method is greater than that of the second encoding method, and the video resolution of the precoding method is less than that of the second encoding method.

3. The video encoding method according to claim 1, characterized in that, The step of predicting the first decoding frame rate corresponding to the second bitstream based on the first bitstream feature information and the first frame rate prediction model includes: Obtain target feature information corresponding to the video to be encoded, wherein the target feature information includes at least one of the following: the spatiotemporal complexity of the video to be encoded, the target resolution, and the client hardware performance information; Based on the first bitstream feature information corresponding to the first bitstream, the target feature information, and the first frame rate prediction model, the first decoding frame rate corresponding to the second bitstream is predicted.

4. The video encoding method according to claim 1, characterized in that, The step of comparing the first decoding frame rate with a first frame rate threshold and determining whether to encode the video to be encoded based on the comparison result includes: In response to the first decoding frame rate being less than the first frame rate threshold, the encoding of the video to be encoded based on the second encoding method is stopped; In response to the first decoding frame rate being greater than or equal to the first frame rate threshold, the video to be encoded is allowed to be encoded based on the second encoding method to obtain the second bitstream.

5. The video encoding method according to claim 4, characterized in that, After allowing encoding of the video to be encoded based on the second encoding method to obtain the second bitstream, the method further includes: Based on the second bitstream feature information and the second frame rate prediction model corresponding to the second bitstream, predict the second decoding frame rate corresponding to the second bitstream. The second decoding frame rate is compared with the second frame rate threshold, and a decision is made on whether to send the second bitstream based on the comparison result.

6. The video encoding method according to claim 5, characterized in that, The step of predicting the second decoding frame rate corresponding to the second bitstream based on the second bitstream feature information and the second frame rate prediction model includes: Obtain target feature information corresponding to the video to be encoded, wherein the target feature information includes at least one of the following: the spatiotemporal complexity of the video to be encoded, the target resolution, and the client hardware performance information; Based on the second bitstream feature information, the target feature information, and the second frame rate prediction model, the second decoding frame rate corresponding to the second bitstream is predicted.

7. The video encoding method according to claim 5, characterized in that, The step of comparing the second decoding frame rate with the second frame rate threshold and determining whether to send the second bitstream based on the comparison result includes: In response to the second decoding frame rate being less than the second frame rate threshold, the transmission of the second bitstream is stopped; In response to the second decoding frame rate being greater than or equal to the second frame rate threshold, the second bitstream is allowed to be sent to the client.

8. The video encoding method according to any one of claims 5-7, characterized in that, Both the first frame rate threshold and the second frame rate threshold are determined based on the minimum frame rate required for real-time video playback; When computing resources are given priority, the first frame rate threshold is greater than the second frame rate threshold.

9. A video encoding device, characterized in that, include: The first bitstream acquisition module is used to acquire the video to be encoded and the first bitstream corresponding to the video to be encoded. The first bitstream is obtained by encoding the video to be encoded based on a first encoding method. The first decoding frame rate prediction module is used to predict the first decoding frame rate corresponding to the second bitstream based on the first bitstream feature information and the first frame rate prediction model corresponding to the first bitstream. The second bitstream is obtained by encoding the video to be encoded based on the second encoding method, and the first decoding frame rate is the video frame rate obtained by decoding the second bitstream based on the decoding method corresponding to the second encoding method. The encoding and decoding complexity of the second encoding method is greater than that of the first encoding method. The video encoding control module is used to compare the first decoding frame rate with the first frame rate threshold, and determine whether to encode the video to be encoded based on the comparison result.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the video encoding method as described in any one of claims 1-8.

11. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the video encoding method as described in any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video encoding method as described in any one of claims 1-8.

Images