Automated encoding optimization methods, apparatus and computer equipment, readable storage media, and program products for encoders.

Abstract

This disclosure provides an automated encoding optimization method, apparatus, computer device, readable storage medium, and program product for an encoder, relating to the field of video processing technology. The implementation scheme is as follows: Video data is acquired by the encoder based on a target bitrate and encoded using a bitrate control algorithm. The video data includes multiple image frames, each with its own quantization parameters and actual bitrate. The bitrate control algorithm dynamically adjusts the quantization parameters based on the image complexity of each image frame to make the actual bitrate numerically closer to the target bitrate. Multiple parallel detections are performed on each of the multiple image frames. In response to any detection result being positive, the number of alarm items for the corresponding detection is increased. The number of alarm items corresponding to the multiple detections is counted to obtain the total alarm statistics for the video data. Based on the total alarm statistics, the encoder's bitrate control algorithm is adjusted for encoding optimization.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, particularly to the fields of video encoding and video detection, and especially to an automated encoding optimization method, apparatus, computer equipment, computer-readable storage medium, and computer program product for an encoder. Background Technology

[0002] In the design and iteration of encoders, such as video encoders, quality assessment has always been a critical step. Traditional processes often involve a significant amount of non-automated operations, such as relying on manual evaluation of video encoding quality, which struggles to meet the increasingly demanding requirements for testing efficiency. Furthermore, traditional encoding quality assessments often focus on a single dimension—either encoding quality or encoding efficiency—which is insufficient for certain application scenarios, such as online encoding. Therefore, a more effective system is urgently needed to serve encoder design and iteration, both in terms of the degree of automation in the operational process and the completeness of the evaluation framework. Summary of the Invention

[0003] This disclosure provides an automated encoding optimization method, apparatus, computer device, computer-readable storage medium, and computer program product for encoders.

[0004] According to one aspect of this disclosure, an automated encoding optimization method for an encoder is provided, comprising: acquiring video data encoded by the encoder based on a target bitrate and using a bitrate control algorithm, the video data comprising multiple image frames, each image frame having its own quantization parameters and actual bitrate, wherein the bitrate control algorithm dynamically adjusts the quantization parameters based on the image complexity of each image frame to make the actual bitrate numerically closer to the target bitrate; performing multiple parallel detections on each of the multiple image frames, including: detecting if the quantization parameter of the current image frame is greater than the quantization parameter of the previous image frame. The algorithm checks whether the actual bitrate is less than the product of the target bitrate and a first threshold; whether the absolute value of the difference between the quantization parameter of the current image frame and the quantization parameter of the previous image frame is greater than a second threshold; whether the number of image frames with an actual bitrate greater than a preset maximum bitrate in the first subset of image frames within a preset time window is greater than a third threshold; in response to any detection result being yes in multiple parallel detections, the number of alarm items for the corresponding detection is increased; the number of alarm items corresponding to multiple detections is counted to obtain the total alarm statistics for the video data; based on the total alarm statistics, the encoder's bitrate control algorithm is adjusted for encoding optimization.

[0005] In some embodiments, performing multiple detections in parallel on each of the multiple image frames further includes: detecting whether the quantization parameter of the current image frame is greater than a fourth threshold, wherein, in response to the current image frame being an I-frame, detecting whether the quantization parameter of the current image frame is greater than a fifth threshold, the fifth threshold being less than the fourth threshold; detecting whether the actual bitrate of the current image frame is greater than the product of the target bitrate and a sixth threshold, or whether the codeword of the current image frame is greater than the product of the target bitrate and a seventh threshold, wherein the codeword is the product of the duration of the current image frame and the actual bitrate.

[0006] In some embodiments, performing multiple detections in parallel on each of the multiple image frames further includes: in response to the current image frame being an I-frame, detecting whether the absolute value of the difference between the quantization parameter of the I-frame and the quantization parameter of the subsequent P-frame or B-frame is greater than a second threshold, or detecting whether the absolute value of the difference between the average quantization parameter of all I-frames and the average quantization parameter of all P-frames or B-frames is greater than the second threshold.

[0007] In some embodiments, multiple detections are configured to determine the coding stability of a rate control algorithm, and each of the multiple alarm items indicates an indicator associated with coding stability. The process of counting the multiple alarm items corresponding to the multiple detections to obtain a total alarm statistic for the video data includes: determining at least one indicator to be improved for optimizing coding stability based on the order of the number of the multiple alarm items; and indicating at least one indicator to be improved in the total alarm statistic.

[0008] In some embodiments, multiple alarm items corresponding to multiple detections are counted to obtain total alarm statistics for video data, including: displaying the total alarm statistics in the form of data visualization for data analysis based on the total alarm statistics.

[0009] In some embodiments, the rate control algorithm uses a rate-quantization model, which includes model coefficients that are updated in real time during the encoding process. Adjusting the rate control algorithm for encoding optimization includes adjusting the update rate of the model coefficients.

[0010] In some embodiments, an automated encoding optimization apparatus for an encoder is provided, comprising: a video data acquisition module configured to acquire video data encoded by the encoder based on a target bitrate and using a bitrate control algorithm, the video data including multiple image frames, each image frame having its own quantization parameters and actual bitrate, wherein the bitrate control algorithm dynamically adjusts the quantization parameters based on the image complexity of each image frame to make the actual bitrate numerically closer to the target bitrate; and a video detection module configured to perform multiple parallel detections on each of the multiple image frames, the video detection module including: a first detection module configured to detect whether the actual bitrate of the current image frame is small when the quantization parameter of the current image frame is greater than the quantization parameter of the previous image frame. The algorithm consists of: a target bitrate multiplied by a first threshold; a second detection module configured to detect whether the absolute value of the difference between the quantization parameter of the current image frame and the quantization parameter of the previous image frame is greater than the second threshold; a third detection module configured to detect whether the number of image frames with an actual bitrate greater than a preset maximum bitrate in a subset of first image frames within a preset time window is greater than a third threshold; an alarm triggering module configured to trigger an increase in the number of alarm items for the corresponding detection in response to any detection result being "yes" among multiple parallel detections; an information statistics module configured to count the number of each alarm item corresponding to multiple detections to obtain the total alarm statistics of the video data; and an information feedback module configured to adjust the encoder's bitrate control algorithm for encoding optimization based on the total alarm statistics.

[0011] According to another aspect of this disclosure, a computer device is provided, comprising: at least one processor; and a memory having a computer program stored thereon, wherein when executed by the at least one processor, the computer program causes the at least one processor to perform the methods provided above in this disclosure.

[0012] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0013] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0014] According to one or more embodiments of this disclosure, balanced improvements in various encoder functions can be achieved, and a fully automated encoding optimization process can be provided.

[0015] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0016] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0017] Figure 1 This is a flowchart illustrating an encoding optimization method according to an exemplary embodiment.

[0018] Figure 2 This is a schematic diagram illustrating the automated process of a coding optimization method according to an exemplary embodiment.

[0019] Figure 3 This is a schematic diagram illustrating a coding balance evaluation according to an exemplary embodiment.

[0020] Figure 4 This is a schematic diagram illustrating the rate control algorithm flow according to an exemplary embodiment.

[0021] Figure 5 This is a schematic block diagram illustrating an encoding optimization apparatus according to an exemplary embodiment.

[0022] Figure 6 This is a block diagram illustrating an exemplary computer device that can be applied to an exemplary embodiment. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0025] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0026] In related technologies, traditional encoding quality testing processes often involve numerous non-automated operations. Especially given the unique nature of subjective and objective evaluation of encoding quality, traditional methods typically require human intervention, making full automation difficult and hindering improved testing efficiency. Furthermore, with the increasing use of online encoding in recent years, such as e-commerce live streaming, video conferencing, and cloud gaming, higher demands are being placed on the encoding quality of video streams. Therefore, encoders not only need to find a balance between encoding quality and efficiency but also need to maintain stable encoding quality and efficiency between frames to avoid issues that directly impact user experience, such as dropped frames in live streaming, conference freezes, and soaring latency in cloud gaming.

[0027] Therefore, embodiments of this disclosure provide a more effective encoding optimization method that enables fully automated testing to serve the design optimization of the video encoder, promoting balanced improvement in various functions of the video encoder.

[0028] Figure 1 This is a flowchart illustrating an encoding optimization method 100 according to an exemplary embodiment.

[0029] like Figure 1 As shown, the encoding optimization method 100 includes steps S101, S102, S103, S104 and S105, wherein step S102 further includes three sub-steps S1021, S1022 and S1023 executed in parallel.

[0030] In step S101, video data is acquired by the encoder based on the target bitrate and encoded using a bitrate control algorithm. The video data includes multiple image frames, each with its own quantization parameters and actual bitrate. The bitrate control algorithm dynamically adjusts the quantization parameters based on the image complexity of each image frame to make the actual bitrate numerically closer to the target bitrate.

[0031] In the example, the target bitrate can be a bitrate constraint set by the bitrate control algorithm before encoding, typically measured in bps (bits per second). The target bitrate can be an average bitrate budget per second set for the entire video sequence. The instantaneous bitrate of a single frame, i.e., the actual bitrate, is allowed to fluctuate around the target bitrate, but the bitrate control algorithm will eventually pull the overall average back to the target bitrate. The actual bitrate numerically approaching the target bitrate means that the difference between the actual bitrate and the target bitrate is within an acceptable tolerance range; for example, the actual bitrate may be slightly higher or slightly lower than the target bitrate. This tolerance range can be based on video encoding requirements or preset based on empirical values.

[0032] In the example, the quantization parameter (QP) can be the primary tool used by the encoder to balance image quality and bitrate. A lower quantization parameter value means more image detail is preserved, but it also increases the encoded bitrate because more bits are needed to represent the image. Conversely, a higher quantization parameter value leads to a decrease in image quality but reduces the bitrate because fewer bits are needed to represent the image. In actual encoding, the encoder can dynamically adjust the quantization parameter based on the target bitrate requirement and the image complexity to achieve different encoding effects under different conditions.

[0033] In step S102, multiple detections are performed in parallel for each of the multiple image frames.

[0034] In sub-step S1021, it is detected whether the actual bit rate of the current image frame is less than the product of the target bit rate and the first threshold, given that the quantization parameter of the current image frame is greater than the quantization parameter of the previous image frame.

[0035] In the example, the detection in sub-step S1021 can be used to evaluate whether the relationship between the quantization parameter and the bitrate during the encoding process meets the requirements. The first threshold can be a number less than 1, such as 0.9, 0.85, 0.8, etc., to represent the degree to which the actual bitrate is lower than the target bitrate. Therefore, if the quantization parameter of the current image frame is greater than that of the previous image frame, it means that the bitrate control algorithm believes that the quantization parameter should be increased relative to the previous image frame to reduce the bitrate, so that the actual bitrate is close to the target bitrate. However, if the actual bitrate of the current image frame is more than 15% lower than the target bitrate (taking the first threshold of 0.85 as an example), it means that the bitrate control algorithm has adjusted the bitrate too low, resulting in a decrease in encoding quality.

[0036] In sub-step S1022, it is detected whether the absolute value of the difference between the quantization parameter of the current image frame and the quantization parameter of the previous image frame is greater than the second threshold.

[0037] In the example, the detection in sub-step S1022 can be used to evaluate whether the difference in quantization parameters between consecutive frames during the encoding process meets the requirements. The value of the second threshold can be used to reflect the fluctuation of the quantization parameters between consecutive frames; for example, it can be set to 7 based on experience. Therefore, if the absolute value of the difference between the quantization parameters of the current image frame and the quantization parameters of the previous image frame is larger than the second threshold, it indicates that the fluctuation of the quantization parameters between consecutive frames is large. This means that the bitrate control algorithm is too aggressive in adjusting the quantization parameters and fails to maintain a smooth change in the quantization parameters, which may lead to problems such as sudden changes in image quality and increased latency, affecting the viewing experience.

[0038] In sub-step S1023, it is detected whether the number of image frames with an actual bit rate greater than the preset maximum bit rate in the first subset of image frames located within the preset time window is greater than the third threshold.

[0039] In the example, the detection in sub-step S1023 can be used to evaluate whether the stability of the bitrate during the encoding process meets the requirements. The time window can be preset to a certain duration, such as 500ms, 1000ms, etc., and this duration can be increased or decreased according to the detection requirements. Taking 500ms as an example, assuming the frame rate is 30 fps (30 frames per second), there are approximately 15 image frames within the 500ms time window, which is the first subset of image frames. The third threshold can be half the number of image frames within the time window. In this example, it is to detect whether half of the 15 image frames have an actual bitrate greater than the preset maximum bitrate. This maximum bitrate, also known as the maximum instantaneous bitrate, can be preset to a value that is a certain proportion greater than the target bitrate, such as two or three times the target bitrate, thereby providing greater flexibility for complex image scenes. The value of the maximum bitrate can be changed according to the video encoding requirements. Therefore, if the number of image frames with an actual bitrate greater than the preset maximum bitrate in the first subset of image frames within the preset time window is greater than the third threshold, it means that the bitrate fluctuates greatly, which may affect the smoothness of video playback or lead to bandwidth waste.

[0040] In step S103, in response to any of the parallel detections being true, the number of alarm items triggered for the corresponding detection is increased.

[0041] In the example, each of the sub-steps S1021, S1022 and S1023 can have a corresponding alarm item for the detection. The initial value of each alarm item can be zero. Once the detection result is yes, the number of the corresponding alarm items can be accumulated, for example, starting from 1 and incrementing. If the detection result is no, the number of the corresponding alarm items remains unchanged.

[0042] In the example, the alarm item can also include various information such as alarm type, trigger threshold, anomaly value, frame number, etc.

[0043] In step S104, the number of each alarm item corresponding to the multiple detections is counted to obtain the total alarm statistics of the video data.

[0044] In the example, after detecting all image frames of the video data, some detections may have a large number of corresponding alarm items, while others may have a small number of alarm items, or even no alarm items at all, meaning the number of alarm items remains zero. Therefore, by counting the number of each of these alarm items, the total alarm statistics can be used to quantitatively evaluate anomalies in the encoding process, thereby optimizing the bitrate control algorithm.

[0045] In step S105, based on the total alarm statistics, the encoder's bit rate control algorithm is adjusted to optimize the encoding.

[0046] In the example, if the number of alarm items corresponding to the detection in sub-step S1021 is large, it may mean that the quantization parameter is adjusted too quickly, leading to a sudden change in quality. Therefore, the rate control algorithm can pay more attention to scene changes in the image and optimize the quantization parameter adjustment strategy. If the number of alarm items corresponding to the detection in sub-step S1022 is large, the rate control algorithm can consider smoothing changes in quantization parameters and reducing quantization parameter fluctuations. If the number of alarm items corresponding to the detection in sub-step S1023 is large, the rate control algorithm can pay attention to reducing instantaneous bitrate fluctuations and increasing bitrate stability.

[0047] Therefore, the encoding optimization method of this disclosure provides a mechanism for detection and optimization from the perspective of maintaining stable encoding quality and efficiency between frames, rather than considering only a single dimension of encoding quality or efficiency, or merely seeking a balance between encoding quality and efficiency. This enables a balanced evaluation of encoding quality, encoding efficiency, and stability, promoting balanced progress in various functions of the video encoder and avoiding functional neglect. Consequently, it can more effectively and flexibly address many application scenarios requiring online encoding (such as e-commerce live streaming, video cloud conferencing, cloud gaming, etc.), improving user experience. Simultaneously, the encoding optimization method of this disclosure also provides the possibility of full-process automation, enabling a closed loop of automatic test triggering, automatic evaluation, and automatic alarm, thereby saving labor costs and improving the efficiency of detection and optimization.

[0048] Figure 2 This is a schematic diagram illustrating the automated process of a coding optimization method according to an exemplary embodiment.

[0049] like Figure 2As shown, in the encoding optimization method of this embodiment, test data can be submitted to the backend using a test pipeline, and test results can be received from the backend. The backend can be, for example, a server, and the encoding optimization method of this embodiment can be executed on the backend. Test data can include, for example, video data encoded by the encoder, and test results can include, for example, total alarm statistics of the video data.

[0050] The front-end can be used for interaction with developers and testers, such as providing feedback data. This feedback data can include a visualization of total alert statistics for video data, such as using ECharts visualization tools. The front-end is also compatible with communication methods such as email or instant messaging software to provide alert feedback to developers and testers. Developers and testers can also submit data requests through the front-end, such as performing custom data analysis and processing based on total alert statistics.

[0051] In addition, the backend can also connect to a database to achieve data management on a certain scale, such as using Doris and MySQL database management systems.

[0052] Figure 3 This is a schematic diagram illustrating a coding balance evaluation according to an exemplary embodiment.

[0053] like Figure 3 As shown, the encoding optimization method of this disclosure proposes an evaluation mechanism for stationarity based on encoding quality and encoding efficiency. For example... Figure 3 The "stableness" circle indicates the relationship between quantization parameters (QP) and bitrate, the difference in quantization parameters between consecutive frames, and bitrate stability, as shown in the above text. Figure 1 Steps S1021 to S1023 are described above. In addition, the maximum value of the quantization parameter, the maximum value of the I-frame quantization parameter, the maximum instantaneous bit rate, the maximum value of the single-frame codeword, the difference between the I-frame quantization parameter and the quantization parameter of the subsequent P-frame or B-frame, and the difference between the average quantization parameter of the I-frame and the average quantization parameter of the P-frame or B-frame can also be detected, which will be described in detail below.

[0054] In some embodiments, the steps of performing multiple detections in parallel for each of a plurality of image frames (e.g., combining) Figure 1 Step S102 may further include: detecting whether the quantization parameter of the current image frame is greater than a fourth threshold, wherein, in response to the current image frame being an I-frame, detecting whether the quantization parameter of the current image frame is greater than a fifth threshold, the fifth threshold being less than the fourth threshold; and detecting whether the actual bitrate of the current image frame is greater than the product of the target bitrate and a sixth threshold, or whether the codeword of the current image frame is greater than the product of the target bitrate and a seventh threshold, wherein the codeword is the product of the duration of the current image frame and the actual bitrate.

[0055] In the example, the maximum value of the quantization parameter can be determined by the encoder design and coding standard. An excessively large quantization parameter value can lead to a significant decrease in image quality, resulting in blurring, loss of detail, and other distortions, while also negatively impacting the video viewing experience. According to some coding standards, the quantization parameter can range from 0 to 51; correspondingly, the fourth threshold can be set to, for example, 45 based on experience. That is, if the quantization parameter of the current image frame exceeds this fourth threshold, an alarm will be triggered, increasing the number of corresponding alarm entries.

[0056] In the example, I-frames, also known as keyframes, can be encoded independently, without depending on other frames. For I-frames, the maximum allowable value for quantization parameters is smaller than for non-I-frames. Assuming the fourth threshold is 45, the fifth threshold for I-frames could be, for example, 35. Therefore, if the current image frame is an I-frame, it is checked whether the quantization parameters of the current image frame exceed this fifth threshold. If they do, an alarm is triggered, increasing the number of corresponding alarm entries.

[0057] In the example, similar to detecting the maximum value of the quantization parameter or the maximum value of the I-frame quantization parameter, the maximum instantaneous bitrate or the maximum value of a single-frame codeword can also be detected. Detecting the maximum instantaneous bitrate is equivalent to detecting whether the actual bitrate of the current image frame is greater than the product of the target bitrate and a sixth threshold, which can be, for example, a value of 3, representing a certain multiple of the target bitrate that is allowed. If expressed in codewords, since a codeword is the product of the frame duration and the bitrate, the maximum value of a single-frame codeword can also be detected, that is, whether the codeword of the current image frame is greater than the product of the target bitrate and a seventh threshold, which can be a factor used to convert the bitrate to codewords, for example, a value of 0.5. Therefore, if the maximum instantaneous bitrate of an image frame or the maximum value of a single-frame codeword exceeds the threshold requirement, an alarm will be triggered, increasing the number of corresponding alarm entries.

[0058] In some embodiments, the steps of performing multiple detections in parallel for each of a plurality of image frames (e.g., combining) Figure 1 Step S102 may further include: in response to the current image frame being an I-frame, detecting whether the absolute value of the difference between the quantization parameter of the I-frame and the quantization parameter of the subsequent P-frame or B-frame is greater than a second threshold, or detecting whether the absolute value of the difference between the average quantization parameter of all I-frames and the average quantization parameter of all P-frames or B-frames in multiple image frames is greater than a second threshold.

[0059] In the example, P-frames, also known as prediction frames, require encoding based on the previous frame. B-frames, also known as bidirectional prediction frames, require encoding based on both the previous and subsequent frames. This embodiment can detect the difference between the quantization parameters of an I-frame and the quantization parameters of its subsequent P-frames or B-frames, or it can detect the difference between the average quantization parameters of an I-frame and the average quantization parameters of a P-frame or B-frame. The second threshold here can be used to reflect the volatility of the quantization parameters, as described above; for example, it can be set to 7 based on experience. Therefore, if the detection result exceeds the threshold requirement, an alarm will be triggered, increasing the number of corresponding alarm items.

[0060] Continue to refer to Figure 3 In some embodiments, such as Figure 3 The "Coding Quality" circle indicates that, for the coding quality evaluation dimension, PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index), and VMAF (Multi-Method Evaluation Fusion) can be detected. For example, a minimum PSNR value can be detected, i.e., whether the PSNR of the current frame is less than a predetermined threshold, such as 23. If it is less than this threshold, an alarm is triggered, increasing the number of alarm entries. A minimum SSIM value can be detected, i.e. whether the SSIM of the current frame is less than a predetermined threshold, such as 0.75. If it is less than this threshold, an alarm is triggered, increasing the number of alarm entries. A minimum VMAF value can be detected, i.e. whether the VMAF of the current frame is less than a predetermined threshold, such as 85. If it is less than this threshold, an alarm is triggered, increasing the number of alarm entries.

[0061] In some embodiments, such as Figure 3 The "coding efficiency" circle indicates that, for the evaluation dimension of coding efficiency, the average bitrate can be detected. For example, it can be detected whether the difference between the average bitrate of multiple image frames and the target bitrate is greater than the product of the target bitrate and a predetermined percentage, such as 15%, meaning whether the average bitrate of multiple image frames is greater than 1.15 times the target bitrate. If it is greater, an alarm is issued, increasing the number of corresponding alarm items.

[0062] In some embodiments, such as Figure 3 The circles indicating "Encoding Quality" and "Encoding Efficiency" indicate that, for these two evaluation dimensions, encoding quality and encoding efficiency, BD-rate (bit rate difference rate) can also be detected. When there is baseline data, such as 3, if the detected BD-rate is greater than 3, an alarm is triggered, increasing the number of corresponding alarm items.

[0063] Therefore, the coding optimization method of this disclosure achieves a balanced evaluation of coding quality, coding efficiency and stability by selecting multiple evaluation indicators, rather than considering only a single dimension of coding quality or coding efficiency, or only seeking a balance between coding quality and coding efficiency.

[0064] In some embodiments, the multiple detections described above can be configured to determine the coding stability of the rate control algorithm, with each of the multiple alarm items indicating an indicator associated with the coding stability. Accordingly, the step of statistically analyzing the multiple alarm items corresponding to the multiple detections to obtain the total alarm statistics for the video data (e.g.) Figure 1 Step S104 shown may include: determining at least one indicator to be improved for optimizing coding stability based on the sorting of the number of each of the multiple alarm items; and indicating the at least one indicator to be improved in the total alarm statistics.

[0065] In the example, since the types of alarms correspond to the types of detections, sorting all alarms based on their quantity (e.g., from most to least) can identify several alarms with a high frequency of anomalies, which are then identified as several metrics requiring improvement. Correspondingly, these metrics can be highlighted in the overall alarm statistics, for example, by providing a sorting function to pin them to the top.

[0066] By providing feedback by specifying the indicators to be improved in the overall alarm statistics, the location of the anomaly can be effectively quantified, the root cause of the problem can be located, and the bitrate control algorithm can be optimized in a targeted manner.

[0067] In some embodiments, the above-described step of statistically analyzing multiple alarm items corresponding to multiple detections to obtain the total alarm statistics of the video data (e.g.) Figure 1 The step S104 shown may also include: displaying the total alarm statistics in the form of data visualization for data analysis based on the total alarm statistics.

[0068] In the example, information can be conveyed by transforming data into charts, graphs, or other visual elements. This can be done via a front-end (e.g., Figure 2 (As shown) ECharts visualization tools provide developers and testers with total alarm statistics displayed in a data visualization format. It also offers an interactive interface for developers and testers to adjust parameters or perform data analysis. For example, developers and testers can select time windows of interest to view data, adjust relevant threshold parameters, and compare multiple sets of data. Additionally, it can provide curves showing the changes of specific metrics with bitrate, and support comparisons between the current version and a reference version for anomaly attribution by developers and testers. Furthermore, it can provide frame-by-frame data comparisons between versions before and after each bitrate.

[0069] Therefore, data visualization provides a more flexible information feedback mechanism, which can help to effectively interpret data and analysis results, and further improve the efficiency of coding optimization.

[0070] In some embodiments, the rate control algorithm may use a rate-quantization model, which may include model coefficients that are updated in real time during the encoding process. Accordingly, the steps described above for adjusting the rate control algorithm to optimize encoding (e.g.) Figure 1 Step S105 shown may include: adjusting the update rate of the model coefficients.

[0071] In the example, the rate-quantization model (RQ model) reflects the derivation relationship between the bitrate and the quantization parameters. Taking the first-order RQ model as an example, it can be expressed as: Rate = α × Comp / Qstep, where Rate represents the bitrate; Comp represents the image complexity, which can be calculated using MAD (Mean Absolute Difference); α represents the model coefficients, which can be continuously updated according to the actual bitrate during the encoding process; and Qstep represents the quantization step size, used to control the degree of data compression. The quantization parameters can be obtained based on the quantization step size, and the relationship between the two can be expressed as: Qstep = λ × (2 (QP / 6) ), where Qstep represents the quantization step size, QP represents the quantization parameter, and λ represents the adjustable scaling factor to adapt to different coding scenarios or coding strategies.

[0072] Figure 4 This is a schematic diagram illustrating the rate control algorithm flow according to an exemplary embodiment.

[0073] like Figure 4 As shown, the RQ model can be initialized according to the expression Rate = α×Comp / Qstep. In each iteration, the bitrate is first estimated based on the RQ model. Then, the quantization parameters are adjusted according to the target bitrate and the RQ model, and partial and full frame encoding is performed. Afterward, the RQ model is updated based on the actual bitrate, specifically updating the α×Comp part. Then, the next iteration can begin.

[0074] In the iterative process of the bitrate control algorithm, by adjusting the update rate of the model coefficients, the actual bitrate output by the encoder can be made more stable.

[0075] Embodiments of this disclosure also provide an automated encoding optimization apparatus for an encoder.

[0076] Figure 5This is a schematic block diagram illustrating an encoding optimization apparatus 500 according to an exemplary embodiment.

[0077] like Figure 5 As shown, the encoding optimization device 500 includes a video data acquisition module 501, a video detection module 502, an alarm triggering module 503, an information statistics module 504, and an information feedback module 505.

[0078] The video data acquisition module 501 is configured to acquire video data encoded by the encoder based on a target bitrate and using a bitrate control algorithm. The video data includes multiple image frames, each with its own quantization parameters and actual bitrate. The bitrate control algorithm dynamically adjusts the quantization parameters based on the image complexity of each image frame to make the actual bitrate numerically closer to the target bitrate.

[0079] The video detection module 502 is configured to perform multiple parallel detections on each of a plurality of image frames. In some embodiments, the video detection module 502 includes a first detection module 5021, a second detection module 5022, and a third detection module 5023. The first detection module 5021 is configured to detect whether the actual bitrate of the current image frame is less than the product of the target bitrate and a first threshold, given that the quantization parameter of the current image frame is greater than the quantization parameter of the previous image frame. The second detection module 5022 is configured to detect whether the absolute value of the difference between the quantization parameter of the current image frame and the quantization parameter of the previous image frame is greater than a second threshold. The third detection module 5023 is configured to detect whether the number of image frames with an actual bitrate greater than a preset maximum bitrate in a first subset of image frames located within a preset time window is greater than a third threshold.

[0080] The alarm triggering module 503 is configured to increase the number of alarm items for the corresponding detection in response to any one of the multiple parallel detections being true.

[0081] The information statistics module 504 is configured to count the number of each of the multiple alarm items corresponding to the multiple detections, so as to obtain the total alarm statistics of the video data.

[0082] The information feedback module 505 is configured to adjust the encoder's bit rate control algorithm for encoding optimization based on total alarm statistics.

[0083] The operations of the aforementioned video data acquisition module 501, video detection module 502, alarm triggering module 503, information statistics module 504, and information feedback module 505 can be combined. Figure 1 The operations of steps S101, S102, S103, S104, and S105 are the same. Simultaneously, the operations of the first detection module 5021, the second detection module 5022, and the third detection module 5023 can be combined. Figure 1The operations of steps S1021, S1022, and S1023 are the same. Therefore, the details of each aspect will not be elaborated here.

[0084] In some embodiments, the video detection module 502 may further include: a fourth detection module 5024, configured to detect whether the quantization parameter of the current image frame is greater than a fourth threshold, wherein, in response to the current image frame being an I-frame, it detects whether the quantization parameter of the current image frame is greater than a fifth threshold, the fifth threshold being less than the fourth threshold; and a fifth detection module 5025, configured to detect whether the actual bitrate of the current image frame is greater than the product of the target bitrate and a sixth threshold, or whether the codeword of the current image frame is greater than the product of the target bitrate and a seventh threshold, wherein the codeword is the product of the duration of the current image frame and the actual bitrate.

[0085] In some embodiments, the video detection module 502 may further include: a sixth detection module 5026, configured to detect whether the absolute value of the difference between the quantization parameter of the I-frame and the quantization parameter of the subsequent P-frame or B-frame is greater than a second threshold in response to the current image frame being an I-frame, or to detect whether the absolute value of the difference between the average quantization parameter of all I-frames and the average quantization parameter of all P-frames or B-frames in multiple image frames is greater than the second threshold.

[0086] In some embodiments, the multiple detections described above can be configured to determine the coding stability of the rate control algorithm, with each of the multiple alarm items indicating an indicator associated with the coding stability. Accordingly, the information statistics module 504 may include: a problem determination module 5041, configured to determine at least one indicator to be improved for optimizing coding stability based on a ranking of the respective numbers of the multiple alarm items; and a problem display module 5042, configured to indicate the at least one indicator to be improved in the total alarm statistics.

[0087] In some embodiments, the information statistics module 504 may further include a data visualization module 5043, configured to display total alarm statistics in a data visualization form for data analysis based on the total alarm statistics.

[0088] In some embodiments, the rate control algorithm may use a rate-quantization model, which may include model coefficients that are updated in real time during the encoding process. The information feedback module 505 may include a model adjustment module 5051, configured to adjust the update rate of the model coefficients.

[0089] While specific functions have been discussed above with reference to specific modules, it should be noted that the functions of the modules discussed herein can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module. The specific actions performed by the modules discussed herein include the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, a specific module performing an action can include the specific module performing the action itself and / or another module that performs the action, called or otherwise accessed by the specific module.

[0090] It should also be understood that the above regarding Figure 5 The described modules can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit. For example, these modules can be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (which includes one or more components in a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0091] According to one aspect of this disclosure, a computer device is provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0092] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0093] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0094] In the following text, combined with Figure 6 Illustrative examples describing such computer devices, non-transitory computer-readable storage media, and computer program products.

[0095] Figure 6 An example configuration of a computer device 600 that can be used to implement the methods described herein is shown.

[0096] Computer device 600 can be a variety of different types of devices. Examples of computer device 600 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0097] Computer device 600 may include at least one processor 602, memory 604, multiple communication interfaces 606, display device 608, other input / output (I / O) devices 610, and one or more mass storage devices 612 capable of communicating with each other, such as via system bus 614 or other suitable connections.

[0098] Processor 602 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 602 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 602 may be configured to acquire and execute computer-readable instructions stored in memory 604, mass storage device 612, or other computer-readable media, such as program code of operating system 616, program code of application program 618, program code of other program 620, etc.

[0099] Memory 604 and mass storage device 612 are examples of computer-readable storage media for storing instructions that are executed by processor 602 to perform the various functions described above. For example, memory 604 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 612 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 604 and mass storage device 612 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 602 as a specific machine configured to perform the operations and functions described in the examples herein.

[0100] Multiple programs can be stored on mass storage device 612. These programs include operating system 616, one or more application programs 618, other programs 620, and program data 622, and they can be loaded into memory 604 for execution. Examples of such application programs or program modules may include, for example, Figure 1 The computer program logic (e.g., computer program code or instructions) of the method 100 shown and / or other embodiments described herein.

[0101] Although Figure 6 The modules 616, 618, 620, and 622, or portions thereof, are illustrated as being stored in memory 604 of computer device 600; however, modules 616, 618, 620, and 622 may be implemented using any form of computer-readable medium accessible by computer device 600. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0102] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by computer devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0103] One or more communication interfaces 606 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces, near field communication (NFC) interfaces, etc. Communication interface 606 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 606 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0104] In some examples, a display device 608, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 610 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0105] The technologies described herein can be supported by these various configurations of computer device 600, and are not limited to specific examples of the technologies described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on a server remote from computer device 600. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computer device 600 to other computer devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality can be implemented partly on computer device 600 and partly through a platform that abstracts the functionality of the cloud.

Claims

1. An automated encoding optimization method for encoders, characterized in that, The method includes: Video data is obtained by an encoder encoding a target bitrate using a bitrate control algorithm. The video data includes multiple image frames, each with its own quantization parameters and actual bitrate. The bitrate control algorithm dynamically adjusts the quantization parameters based on the image complexity of each image frame to make the actual bitrate numerically closer to the target bitrate. The actual bitrate being numerically closer to the target bitrate means that the difference between the actual bitrate and the target bitrate is within an acceptable range. Perform multiple parallel detections on each of the plurality of image frames, including: Detect whether the actual bitrate of the current image frame is less than the product of the target bitrate and the first threshold, given that the quantization parameter of the current image frame is greater than the quantization parameter of the previous image frame. Detect whether the absolute value of the difference between the quantization parameters of the current image frame and the quantization parameters of the previous image frame is greater than a second threshold. Detect whether the number of image frames with an actual bitrate greater than a preset maximum bitrate in the first subset of image frames located within a preset time window is greater than a third threshold. In response to any one of the multiple parallel detections being true, the number of alarm items triggered for the corresponding detection increases; The number of each alarm item corresponding to the multiple detections is counted to obtain the total alarm statistics of the video data; Based on the total alarm statistics, the bit rate control algorithm of the encoder is adjusted to optimize the encoding.

2. The method according to claim 1, characterized in that, The method of performing multiple parallel detections on each of the plurality of image frames further includes: Detect whether the quantization parameter of the current image frame is greater than a fourth threshold, wherein, in response to the current image frame being an I-frame, detect whether the quantization parameter of the current image frame is greater than a fifth threshold, wherein the fifth threshold is less than the fourth threshold; Detect whether the actual bitrate of the current image frame is greater than the product of the target bitrate and the sixth threshold, or whether the codeword of the current image frame is greater than the product of the target bitrate and the seventh threshold, wherein the codeword is the product of the duration of the current image frame and the actual bitrate.

3. The method according to claim 2, characterized in that, The method of performing multiple parallel detections on each of the plurality of image frames further includes: In response to the current image frame being an I-frame, it detects whether the absolute value of the difference between the quantization parameter of the I-frame and the quantization parameter of the subsequent P-frame or B-frame is greater than the second threshold, or it detects whether the absolute value of the difference between the average quantization parameter of all I-frames and the average quantization parameter of all P-frames or B-frames is greater than the second threshold.

4. The method according to any one of claims 1 to 3, characterized in that, The plurality of detections are configured to determine the coding stability of the bitrate control algorithm, and each of the plurality of alarm items indicates an indicator associated with the coding stability. The step of statistically analyzing the plurality of alarm items corresponding to the plurality of detections to obtain the total alarm statistics for the video data includes: Based on the ranking of the number of each of the multiple alarm items, at least one indicator to be improved is determined for optimizing the coding stability. The total alarm statistics specify at least one indicator that needs improvement.

5. The method according to claim 4, characterized in that, The step of statistically analyzing the multiple alarm items corresponding to the multiple detections to obtain the total alarm statistics of the video data includes: displaying the total alarm statistics in the form of data visualization for data analysis based on the total alarm statistics.

6. The method according to any one of claims 1 to 3, characterized in that, The bitrate control algorithm uses a bitrate-quantization model, which includes model coefficients that are updated in real time during the encoding process. The step of adjusting the bitrate control algorithm for encoding optimization includes adjusting the update rate of the model coefficients.

7. An automated encoding optimization device for an encoder, characterized in that, The device includes: The video data acquisition module is configured to acquire video data encoded by the encoder based on a target bitrate and using a bitrate control algorithm. The video data includes multiple image frames, each image frame having its own quantization parameters and actual bitrate. The bitrate control algorithm dynamically adjusts the quantization parameters based on the image complexity of each image frame so that the actual bitrate is numerically closer to the target bitrate. The actual bitrate being numerically closer to the target bitrate means that the difference between the actual bitrate and the target bitrate is within an acceptable range. A video detection module is configured to perform multiple detections in parallel on each of the plurality of image frames, the video detection module comprising: The first detection module is configured to detect whether the actual bit rate of the current image frame is less than the product of the target bit rate and the first threshold when the quantization parameter of the current image frame is greater than the quantization parameter of the previous image frame. The second detection module is configured to detect whether the absolute value of the difference between the quantization parameter of the current image frame and the quantization parameter of the previous image frame is greater than a second threshold. The third detection module is configured to detect whether the number of image frames in the first subset of image frames located within a preset time window whose actual bit rate is greater than the preset maximum bit rate is greater than the third threshold. The alarm triggering module is configured to increase the number of alarm items for the corresponding detection in response to any one of the multiple parallel detections being true. The information statistics module is configured to count the number of each alarm item corresponding to the multiple detections, so as to obtain the total alarm statistics of the video data; The information feedback module is configured to adjust the bitrate control algorithm of the encoder for encoding optimization based on the total alarm statistics.

8. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-6.

Images