Code rate control method and device, data storage method and device, video processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides a code rate control method and device, a data storage method and device, a video processing method and device, a storage medium and a program product. In the method, content feature information and coding feature information can be extracted from a video clip, the content feature information reflects the content complexity of the video clip, and the coding feature information reflects the coding complexity of the video clip; performing code rate prediction according to the content feature information and the coding feature information to obtain a target code rate used for video coding; predicting a target quality factor used for video coding according to the content feature information and the target code rate; and with the target code rate as a code rate constraint condition, according to the target quality factor, carrying out code rate control with the specified visual quality constant as a target on the coding process of the video clip. In one aspect, encoding may be performed based on a variable code rate to save encoding bandwidth; on one hand, each frame of picture of the coding result can accord with the specified visual quality based on the specified visual quality constant target, so that the video quality and the transmission efficiency are balanced.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a bitrate control, data storage and video processing method, device, storage medium and program product. Background Technology

[0002] In video applications, video data is typically encoded to improve transmission and storage efficiency. This involves using compression techniques to remove redundant information in the temporal and spatial dimensions of the video data. In video encoding, bitrate refers to the amount of data used per unit of time; that is, the amount of data produced per unit of time after encoding. It is a major factor affecting transmission efficiency and video quality. At the same resolution, a higher bitrate results in a lower compression ratio, higher video quality, but lower transmission efficiency.

[0003] In practical applications, to ensure video quality as much as possible with limited resources while minimizing bitrate savings, a reasonable bitrate is typically configured based on the video data resolution. For example, a 1080P video (i.e., 1920*1080 pixels) might be configured with a bitrate of 3 Mbps (Megabits per second); the video data is then encoded based on this configured bitrate. However, given a specific visual quality requirement, this method of encoding video with a fixed bitrate may result in some videos failing to meet that quality requirement, while others may not require such a high bitrate, leading to wasted bandwidth costs and reduced transmission efficiency. Summary of the Invention

[0004] This application provides a bitrate control, data storage and video processing method, device, storage medium and program product to balance video quality and transmission efficiency in the bitrate control process and minimize bandwidth costs.

[0005] This application provides a bitrate control method, comprising: extracting content feature information and encoding feature information from a video segment to be encoded, wherein the content feature information reflects the content complexity of the video segment and the encoding feature information reflects the encoding complexity of the video segment; performing bitrate prediction based on the content feature information and the encoding feature information to obtain a target bitrate for encoding the video segment; performing quality factor prediction based on the content feature information and the target bitrate to obtain a target quality factor for encoding the video segment; and using the target bitrate as a bitrate constraint and the target quality factor, performing bitrate control on the encoding process of the video segment with the goal of maintaining a constant specified visual quality.

[0006] This application also provides a data storage method, comprising: receiving video data to be stored, the video data including at least one video segment; encoding the at least one video segment, and using the bitrate control method to control the bitrate during the encoding process to obtain at least one encoded video segment; and storing the at least one encoded video segment.

[0007] This application embodiment also provides a video processing method, including: receiving video data sent by a data source, the video data including at least one video segment; encoding the at least one video segment, and using the bitrate control method to control the bitrate during the encoding process to obtain at least one encoded video segment; receiving a data acquisition request sent by a data requesting end, the data acquisition request including identification information of the requested video segment; and sending the video segment corresponding to the identification information from the at least one encoded video segment to the data requesting end.

[0008] This application also provides an electronic device, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: perform the steps in the bit rate control method.

[0009] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the bitrate control method.

[0010] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, performs the steps in the bitrate control method.

[0011] In this embodiment, content feature information and encoding feature information can be extracted from the video segment to be encoded. The content feature information reflects the content complexity of the video segment, and the encoding feature information reflects the encoding complexity of the video segment. Based on the content feature information and encoding feature information, bitrate prediction is performed to obtain the target bitrate for video encoding. Based on the content feature information and the target bitrate, the target quality factor for video encoding is predicted. Using the target bitrate as a bitrate constraint, and based on the target quality factor, bitrate control is performed on the video segment encoding process with the goal of maintaining a constant specified visual quality. On the one hand, encoding based on a variable bitrate can save encoding bandwidth; on the other hand, based on the goal of maintaining a constant specified visual quality, each frame in the final encoded result can meet the specified visual quality, thereby balancing video quality and transmission efficiency. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0013] Figure 1a A schematic diagram of a video encoding and decoding process provided for an exemplary embodiment of this application;

[0014] Figure 1b Another flowchart illustrating a rate control method provided in an exemplary embodiment of this application;

[0015] Figure 2 Another flowchart illustrating a rate control method provided for an exemplary embodiment of this application;

[0016] Figure 3a A schematic diagram of the structure of a rate prediction network provided for an exemplary embodiment of this application;

[0017] Figure 3b Another schematic diagram of the structure of a rate prediction network provided for an exemplary embodiment of this application;

[0018] Figure 4a A schematic diagram of the structure of a quality factor prediction network provided for an exemplary embodiment of this application;

[0019] Figure 4b Another schematic diagram of the structure of a quality factor prediction network provided as an exemplary embodiment of this application;

[0020] Figure 5a A flowchart illustrating the training bit rate prediction network and quality factor prediction network provided for an exemplary embodiment of this application;

[0021] Figure 5b A schematic flowchart illustrating a data storage method provided in an exemplary embodiment of this application;

[0022] Figure 5c A schematic diagram of the interaction flow of a video processing method provided for an exemplary embodiment of this application;

[0023] Figure 6 A schematic diagram of the structure of a rate control device provided in an exemplary embodiment of this application;

[0024] Figure 7 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding access points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0027] In video encoding, to ensure video quality as much as possible with limited resources while minimizing bitrate, a reasonable bitrate is typically configured based on the video data resolution. For example, a 1080P video might be configured with a bitrate of 3 Mbps; the video data is then encoded based on this configured bitrate. However, given a specific video quality requirement, this fixed bitrate encoding method may result in some videos failing to meet that quality requirement, while others may not require such a high bitrate, leading to wasted bandwidth and reduced transmission efficiency.

[0028] To address the aforementioned technical problems, this application provides a solution for fusion encoding based on constant visual quality and variable bitrate. It predicts the target bitrate based on content and encoding feature information extracted from the video segment to be encoded, and further predicts the target quality factor based on the target bitrate and content feature information. Thus, under the constraint of the target bitrate, the encoding process of the video segment is controlled according to the target quality factor to achieve constant visual quality. On the one hand, encoding based on variable bitrate can save encoding bandwidth; on the other hand, based on the goal of constant visual quality, each frame in the final encoded result can meet the specified visual quality, thereby balancing video quality and transmission efficiency.

[0029] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0030] Figure 1a A schematic diagram of a video encoding and decoding process provided for an exemplary embodiment of this application, as shown below. Figure 1aAs shown, the video segment to be encoded is input to the encoder. The encoder includes a bitrate control module and an encoding module. The bitrate control module controls the bitrate used by the encoding module during video encoding to adjust the video encoding quality. The encoding module uses the bitrate provided by the bitrate control module to encode the video segment, obtaining the corresponding bitstream. Further, the bitstream can be packaged and sent to a target address, where the target address can be a storage address on other electronic devices or a server, or it can be a storage address on this electronic device; this embodiment does not impose any restrictions. Further, as... Figure 1a As shown, the decoder can obtain the encoded bitstream from the target address, decode the bitstream to obtain the decoded video segment, and send the decoded video segment to the player for playback.

[0031] Furthermore, a quality assessment module can be used to evaluate the quality of the video encoded by the encoder to obtain the encoding quality. Based on the encoding quality of the video segment, the bitrate control module in the encoder can be adjusted or optimized to provide a more reasonable bitrate. The video quality assessment can employ either objective or subjective methods, without limitation.

[0032] Objective evaluation methods refer to methods of evaluating the quality of decoded video segments based on objective metrics, such as, but not limited to, PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structure Similarity). PSNR quantifies distortion by calculating the average of the squared differences between the original and compressed video pixels and using the logarithm of the ratio of this average to the maximum signal strength. SSIM considers the similarity of brightness, contrast, and structural information, and reflects human perception of image quality better than PSNR. SSIM values ​​range from -1 to 1, with values ​​closer to 1 indicating higher visual quality.

[0033] Subjective evaluation methods refer to quality assessments based on the human eye's subjective perception of decoded video segments. Examples include, but are not limited to, VMAF (Video Multimethod Assessment Fusion). VMAF combines the advantages of multiple video quality assessment methods (such as detail loss, noise, and coding artifacts) and integrates them through machine learning algorithms to provide a single, comprehensive video quality score. This score ranges from 0 to 100, with higher scores indicating better video quality and more closely resembling human perception of video quality.

[0034] Figure 1bThis is a flowchart illustrating a rate control method provided in an exemplary embodiment of this application. The method can be executed by an electronic device, which can be a terminal device such as a mobile phone, computer, or tablet computer, or a server such as a cloud server or a regular server. This embodiment does not limit the type of electronic device. This rate control method can be applied to the above-mentioned... Figure 1a The bitrate control module in the encoder is used to control the bitrate of the video segment encoding process more accurately.

[0035] like Figure 1a As shown, the bitrate control method includes the following steps:

[0036] Step 11: Extract content feature information and encoding feature information from the video segment to be encoded. The content feature information reflects the content complexity of the video segment, and the encoding feature information reflects the encoding complexity of the video segment.

[0037] Step 12: Predict the bitrate based on the content feature information and encoding feature information to obtain the target bitrate for encoding the video segment.

[0038] Step 13: Based on the content feature information and the target bitrate, predict the quality factor to obtain the target quality factor used for encoding the video segment.

[0039] Step 14: Using the target bitrate as a bitrate constraint, and based on the target quality factor, perform bitrate control on the encoding process of the video segment with the goal of maintaining constant visual quality.

[0040] In this embodiment, the video segment to be encoded can be a single video segment or multiple video sub-segments obtained by segmenting the video content based on its content or length. This embodiment does not limit the content and type of the video segment to be encoded; for example, it can be a short video, a video segment from a live streaming application, or various advertising videos, film and television videos, etc. In this embodiment, feature extraction is first performed on the video segment to be encoded to obtain content feature information and encoding feature information. The method for extracting content feature information and encoding feature information is not limited; any method that can extract the content feature information and encoding feature information described below is applicable to this embodiment.

[0041] Content feature information reflects the complexity of a video segment's content, which directly impacts the efficiency and quality of video encoding. Content complexity refers to the spatial and temporal complexity of a video segment. Spatial complexity typically refers to intra-frame complexity, the complexity within a single image frame, primarily determined by intra-frame details and textures. Therefore, this content feature information includes, but is not limited to, various features reflecting intra-frame complexity, such as color, texture, or shape features. Temporal complexity typically refers to inter-frame complexity, the complexity of content changes between consecutive frames, often involving the motion of objects in a scene. Therefore, this content feature information can also include, but is not limited to, various features reflecting inter-frame complexity, such as motion features, scene features, spatiotemporal texture features (texture features combining temporal and spatial information), and motion field features (the motion vector field of the entire scene).

[0042] Encoding feature information reflects the encoding complexity of a video segment, i.e., how easily the video segment is encoded. This affects the size of the encoded video file, decoding complexity, and the smoothness of playback. Encoding complexity can be specifically reflected in the encoder's encoding behavior and results when encoding the video segment. This includes, but is not limited to, the encoder standard type, the block division method, the encoder mode, and the residual size. Therefore, encoding feature information includes, but is not limited to, the encoder standard type, the block division method, the encoder mode, and the residual size involved in encoding the video segment.

[0043] Encoder standards include, but are not limited to, H.263 (a coding standard for video telephony, video conferencing, and internet video), H.264 (a highly compressed digital video codec standard), and H.265 (a video compression standard). Different encoder standards correspond to different levels of coding difficulty. Code block partitioning refers to dividing each frame of image into multiple small processing units for efficient compression and encoding. Code block partitioning methods include, but are not limited to, macroblock partitioning, hierarchical coding partitioning, coding unit (CU) partitioning, and adaptive block partitioning. The code block partitioning method directly affects the encoding efficiency, compression ratio, and decoded video quality.

[0044] In video coding, encoders employ different coding modes. These modes refer to different techniques and strategies used to compress video frames, optimizing coding efficiency and video quality for different video characteristics. Available coding modes include, but are not limited to: Intra Coding Mode, Inter Coding Mode, DC Mode, Direct Mode, Quantization Mode, Entropy Coding Mode, and so on. The choice of coding mode directly impacts coding efficiency, decoding complexity, and video quality.

[0045] In video coding, residual refers to the difference between the original data and the predicted data during predictive coding. It utilizes the spatial and temporal redundancy of video data to reduce data volume through predictive coding, thereby achieving efficient data compression. Residual size refers to the size of the residual block, and the choice of residual size has a significant impact on coding efficiency and the quality of the decoded image. In this embodiment, a variable bitrate (VBR) coding concept is introduced into the video coding process. The VBR coding algorithm is a coding bitrate that dynamically adjusts the coding bitrate (i.e., bitrate) to adapt to changes in the content complexity of the video segment, aiming to maximize video quality under a given bitrate constraint. Therefore, in this embodiment, after extracting content feature information and coding feature information from the video segment to be encoded, bitrate prediction can be performed based on the content feature information and coding feature information to obtain the target bitrate used for encoding the video segment. In this embodiment, the bitrate during the encoding process of the video segment is dynamically changing, rather than fixed. The purpose of the target bitrate is to constrain the bitrate used during encoding, that is, to limit the range of bitrates used during encoding, for example, it can not exceed the maximum bitrate, and the maximum bitrate can be determined by the target bitrate. The method for determining the maximum bitrate based on the target bitrate is described in the following embodiments. In this way, bitrate control can be performed under bitrate constraints, thereby saving encoding bandwidth and avoiding excessively high bitrates during video encoding.

[0046] It should be noted that the target bitrate is predicted based on the VBR bitrate control algorithm. However, in this embodiment, the VBR bitrate control algorithm is not directly used for bitrate control under the constraint of the target bitrate. Instead, the ideas of VBR bitrate control algorithm and constant quality bitrate control algorithm are integrated to balance video quality and transmission efficiency when controlling the bitrate during the encoding process of video segments. The constant quality bitrate control algorithm can be, but is not limited to, CRF (Constant Rate Factor) bitrate control algorithm. This bitrate control algorithm is an algorithm that automatically adjusts the encoding bitrate (i.e., bitrate) according to the set video quality level during the video encoding process. In other words, the bitrate in the CRF bitrate control algorithm is dynamically changing. Unlike the bitrate control principle of VBR or CBR (Constant Bit Rate), CRF is a bitrate control algorithm that strives to achieve a better balance between video quality and encoded file size. That is, it uses a single quality factor (RateFactor, RF) to control the quality and file size of the encoded output video to maintain a constant overall video quality level as much as possible.

[0047] Based on the above, in this embodiment, after obtaining the target bitrate, a quality factor prediction is further performed based on content feature information and the target bitrate to obtain the target quality factor used for encoding the video segment. In this process, not only is the target quality factor predicted based on content feature information, but the target bitrate is also considered, initially demonstrating the fusion of variable bitrate control algorithms and constant quality control algorithms. The quality factor is used to control the quality and file size of the encoded video segment, and can be considered as an adjustment factor affecting the quality of the output video or image. The target quality factor refers to the quality factor required when encoding the video segment; it is the basis for bitrate control during the encoding process, aiming for constant visual quality.

[0048] Furthermore, based on the target bitrate and target quality factor obtained above, the target bitrate can be used as a bitrate constraint. According to the target quality factor, bitrate control is performed on the video segment encoding process with the goal of maintaining a constant visual quality. In this process, bitrate control based on the target quality factor with the goal of maintaining a constant visual quality embodies the idea of ​​a constant quality bitrate control algorithm. The bitrate is variable during this process. Furthermore, the target bitrate is used as a bitrate constraint in the "bitrate control process with the goal of maintaining a constant visual quality." That is, in this embodiment, the quality factor is constant, and the bitrate changes dynamically, subject to the bitrate constraint. This further demonstrates the fusion of variable bitrate bitrate control algorithms and constant quality bitrate control algorithms. The specified visual quality can be understood as a certain visual quality under a visual quality standard. For example, using the VMAF quality evaluation standard, the specified visual quality could be 90 or 85, etc. It should be noted that the visual quality standard can adopt a subjective quality assessment system, and this embodiment does not limit this. This section explains that the specified visual quality is matched with the target quality factor, which is a quality factor that can meet the specified visual quality.

[0049] In this embodiment, the bitrate control algorithm used for "bitrate control with the goal of maintaining constant visual quality" is not limited. For example, it can be, but is not limited to, the CRF bitrate control algorithm. When using the CRF bitrate control algorithm, the target quality factor can be implemented as the target RF.

[0050] In this embodiment, on the one hand, under the constraint of the target bitrate, the bitrate during the encoding of video segments is variable and will not exceed the target bitrate, thereby avoiding excessive bitrate and saving bandwidth. On the other hand, the specified visual quality can be kept constant during the bitrate control process, so that each frame of the video encoding result meets the expected visual quality. Based on these two aspects, a balance can be struck between video quality and transmission efficiency during the video encoding process, saving bandwidth and computing resources as much as possible while ensuring constant video quality.

[0051] The following will be based on the above embodiments, combined with Figure 2 The diagram shown illustrates the steps in the rate control process provided in this application embodiment:

[0052] like Figure 2As shown, video content can be segmented into various video segments. Any video segment can be used as the video segment to be encoded and input into a feature extraction network for feature extraction. In this embodiment, the feature extraction network includes a video content complexity analysis module and a video encoding complexity analysis module. On one hand, the video segment to be encoded is fed into the video content complexity analysis module for video content complexity analysis to obtain content feature information Fc reflecting the content complexity of the video segment; on the other hand, the video segment to be encoded is fed into the video encoding complexity analysis module for video encoding complexity analysis to obtain encoding feature information Fe reflecting the encoding complexity of the video segment.

[0053] In this embodiment, the content feature information Fc is not limited to the feature information it contains. In an exemplary embodiment, the content feature information includes, but is not limited to, at least one of the following: feature information based on the gray-level co-occurrence matrix (GLCM), feature information based on the Discrete Cosine Transform (DCT), and feature information based on the Normalized Cross-Correlation (NCC) matrix. The GLCM is used to describe the spatial relationship of pixel gray values ​​in any frame of a video segment. The DCT-based feature information is used to describe the DCT calculation result of any frame of a video segment. The NCC matrix is ​​used to determine the similarity or matching degree between adjacent frames of a video segment. Subsequently, video content complexity analysis can be performed on at least one of the GLCM-based, DCT-based, and NCC-based feature information, and the content feature information Fc can be obtained by fusing the results of each analysis.

[0054] Specifically, within the video content complexity analysis module, a target gray-level co-occurrence matrix (GLCM) can be calculated for any frame of the image. For each frame, a distance parameter [1, 3, 5] is selected to calculate at least one of the following primary features of the target GLCM: energy, entropy, homogeneity, correlation, and contrast. Energy is the sum of the squares of all elements in the GLCM, used to measure the uniformity or stability of the image texture. Entropy measures the uncertainty or randomness of the image texture; a high entropy value indicates a complex and varied texture with rich information, while a low entropy value indicates a more regular texture with less information. Homogeneity reflects the frequency of pixel pairs with similar gray values ​​in the image. Correlation assesses the linear dependence of pixels on gray values. Contrast characterizes the overall degree of gray-level differences in the image; high contrast indicates significant gray-level variations and a coarse texture, while low contrast indicates small gray-level variations and a fine texture. Furthermore, for all frames of the video segment, numerical calculations are performed on any one of the first feature information of all frames of the video segment to obtain the first analysis result. The numerical calculation methods include, but are not limited to, at least one of the following: calculating the mean and calculating the variance. Further, based on the first analysis results corresponding to various first feature information, GLCM-based feature information can be generated. It should be noted that the distance parameters [1, 3, 5] mentioned above are merely examples and are not limited to them. When the first feature information includes the five types listed above, the distance parameters are [1, 3, 5], and the numerical calculation includes both the mean and variance, the GLCM-based feature information is a (3*5*2) = 30-dimensional feature vector; in (3*5*2), 3 represents the dimension of the distance parameters, 5 represents the dimension of the first feature information, and 2 represents the dimension of the numerical calculation.

[0055] Specifically, within the video content complexity analysis module, any given frame image can be divided into blocks, and the DCT calculation results for each block can be calculated. Based on the DCT calculation results of the blocks, at least one of the following second feature information is calculated: pixel intensity information within the video frame block, pixel intensity variation information within adjacent video frame blocks, DC component of the luminance component, and mean value of the chrominance component. Further, a preset content analysis algorithm and at least one second feature information are used to analyze the video segment, obtaining at least one second analysis result. The preset content analysis algorithm can be either a VCA (Video Complexity Analyzer, an adaptive content analysis technique for video encoding and decoding) algorithm or a Video Summarization (VS) algorithm. The VCA algorithm is used to obtain a description of the spatiotemporal features of the video based on DCT transformation; the video summarization algorithm is used to extract keyframes from long videos or generate short video summaries for content analysis of video segments. Finally, DCT-based feature information is obtained based on at least one second analysis result. When there are 14 types of second feature information, the DCT-based feature information is implemented as a 14-dimensional feature vector.

[0056] Specifically, within the video content complexity analysis module, the NCC matrix of the current frame can be calculated for any two adjacent frames of a video segment, with the previous frame as a reference. Based on the NCC matrix, at least one of the following third feature information can be calculated: mean, standard deviation, entropy, skewness, and kurtosis. Furthermore, for all frames of the video segment, numerical calculations can be performed on any of the extracted third feature information to obtain the third analysis result. The numerical calculation methods include, but are not limited to, at least one of the following: calculating the mean and calculating the variance. Further, based on the third analysis results corresponding to various third feature information, NCC-based feature information can be generated. It should be noted that when the third feature information includes the five types listed above, and the numerical calculation includes both mean and variance, the NCC-based feature information is a (5*2) = 10-dimensional feature vector; in (5*2), 5 represents the dimension of the third feature information, and 2 represents the dimension of the numerical calculation.

[0057] Based on the above, the first, second, and third analysis results can be fused, that is, the feature information based on GLCM, DCT, and NCC can be fused to obtain the content feature information Fc. Fusion methods include, but are not limited to, concatenation, addition, and weighted summation. Continuing with the dimensions of the various vector features listed above (30, 14, and 10), in the concatenation method, further combining the frame rate of the video segment yields a 55-dimensional feature vector, i.e., the content feature information Fc.

[0058] In this embodiment, the feature information contained in the encoding feature information Fe is not limited, nor is the specific implementation method of the video encoding complexity analysis module for video encoding complexity analysis limited. In an exemplary embodiment, when performing video encoding complexity analysis, the video encoding complexity analysis module can pre-encode the video segment to obtain pre-encoding process information and result information, which serve as the encoding feature information Fe reflecting the encoding complexity of the video segment. In this embodiment, the specific implementation method of the pre-encoding process is not limited. In an exemplary embodiment, a preset encoder can be used to pre-encode the video segment. During this process, the encoder level, encoder encoding parameters, etc., can be selected. Any encoder level can be selected from multiple encoder levels. Optionally, a lower level can be selected. Taking the x265 open-source encoder as an example, the ultrafast level can be selected. The encoder encoding parameters can be default parameters or user-defined parameters; this embodiment does not limit either of these. x265 is an open-source H.265 / HEVC (High Efficiency Video Coding) video encoder library. Based on this, a preset encoder can be used to encode each frame in a video segment according to the selected encoder level and encoder parameters, and encoder result information can be obtained. The encoder result information includes, but is not limited to, at least one of the following: the proportion of intra-prediction modes, motion vector length, CU proportion, residual energy, etc.

[0059] Furthermore, based on the encoder results, the encoder's behavior during encoding can be analyzed to obtain encoder behavior features used to measure the encoding complexity of a video segment. These encoder behavior features include multiple numerical features for each frame of the video segment, such as frame rate, resolution, code block division method, encoder speed, and other related information. For example, assuming that encoder behavior analysis determines each frame has 68 numerical features, the mean, standard deviation, entropy, skewness, and kurtosis of the numerical features for all frames of the video can be calculated based on these encoder behavior features, resulting in (68*5) = 340-dimensional features, which is the encoding feature information Fe used to reflect the encoding complexity.

[0060] This application does not limit the implementation of bitrate prediction based on content feature information and encoding feature information. In some optional embodiments, a pre-trained neural network model, such as a bitrate prediction network, can be used for bitrate prediction. Furthermore, when performing bitrate prediction based on content feature information and encoding feature information to obtain the target bitrate for encoding video segments, the content feature information and encoding feature information can be preprocessed according to preset feature information processing rules to improve the quality of the input information of the bitrate prediction network, so that the bitrate prediction network can perform bitrate prediction more efficiently and stably.

[0061] Optionally, mean subtraction can be performed on the content feature information and the encoding feature information; noise can also be eliminated on the content feature information and the encoding feature information through filtering or smoothing, and removing or reducing noise in the input information can improve the accuracy of bitrate prediction by the bitrate prediction network; missing values ​​can also be filled in the content feature information and the encoding feature information through the average value or median.

[0062] Furthermore, such as Figure 2 As shown, content feature information and encoding feature information can be concatenated to obtain the first concatenated feature. The first concatenated feature refers to the concatenated result obtained by concatenating content feature information and encoding feature information. The term "first" is used to define the concatenated feature, only to distinguish it from the "second concatenated feature" mentioned later, and does not limit the concatenation method, type, or attribute of the concatenated feature. Specifically, the content feature information and encoding feature information can be vector-connected at the feature vector level to form a wider feature vector, i.e., the first concatenated feature; alternatively, weights can be assigned to the content feature information and the encoding feature information respectively, and the content feature information and the encoding feature information can be weighted and summed according to the assigned weights to obtain the first concatenated feature. This embodiment does not limit the concatenation method of the feature information. Then, the first concatenated feature can be input into a bitrate prediction network for bitrate prediction to obtain the target bitrate. The bitrate prediction network is trained with the goal of maintaining constant visual quality. In other words, the training objective during the training phase of the bitrate prediction network is the same as the control objective of bitrate control in the aforementioned embodiments, which is to maintain constant visual quality.

[0063] like Figure 3a As shown, the network structure of a bitrate prediction network may include a bitrate feature extraction layer and a bitrate prediction layer. Figure 3a This example only illustrates the case where there are multiple bitrate feature extraction layers and does not limit the network structure of the bitrate prediction network. The following will explain in detail the bitrate prediction process based on this network structure:

[0064] First, the first concatenated feature is input into the bitrate feature extraction layer for feature extraction to obtain the first target feature. There can be one or more bitrate feature extraction layers. When there are multiple bitrate feature extraction layers, the input of the subsequent bitrate feature extraction layer is the output of the previous bitrate feature extraction layer, used to extract higher-level features based on the previous bitrate feature extraction layer.

[0065] like Figure 3a As shown, since the multiple bitrate feature extraction layers are C1, C2, C3, and C4 from front to back, the levels of features 1, 2, 3, and 4 (i.e., the first target features) output by C1, C2, C3, and C4 respectively increase sequentially. Low-level features refer to the basic attributes extracted directly from the features, which are usually closely related to the physical characteristics of the data. High-level features are obtained through a series of nonlinear transformations, such as combination, transformation, and abstraction of low-level features, and usually contain more complex patterns and the inherent structure of the data. Unlike low-level features, high-level features reflect the higher-level semantics and concepts of the data. As the feature extraction level gradually increases, the complexity of the features also increases, and the semantic information they contain becomes increasingly richer.

[0066] Subsequently, the first target feature can be input into the bitrate prediction layer for regression analysis of the relationship between the feature and the bitrate to obtain the target bitrate. Through a pre-training process, the bitrate prediction network can learn the relationship between features and bitrate through its internal neurons and connection weights. In other words, the trained bitrate prediction network has the ability to determine the target bitrate based on the first target feature. Regression analysis refers to the process of using a neural network model to perform numerical prediction, used to determine the relationship between one or more independent variables (input features) and a continuous dependent variable (target variable). In this embodiment, when the first target feature is input into the bitrate prediction layer for regression analysis of the relationship between the feature and the bitrate, the bitrate prediction layer can determine the target bitrate corresponding to the first target feature based on the pre-learned relationship between the feature and the bitrate and the input first target feature.

[0067] Optionally, the bitrate prediction network in the above embodiments can be implemented as various types of networks, such as MLP (Multilayer Perceptron) networks, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or Generative Adversarial Networks (GANs), etc., and this embodiment does not impose any limitations. Among them, the MLP network is a feedforward artificial neural network consisting of an input layer, one or more hidden layers, and an output layer. All layers are fully connected, meaning that every neuron in each layer is connected to all neurons in the next layer.

[0068] Preferably, when the bitrate prediction network is implemented as an MLP network, the bitrate feature extraction layer and the bitrate prediction layer can each include a fully connected layer and a normalization layer from the MLP network, respectively. The number of fully connected layers is negatively correlated with the output dimension of the fully connected layers; that is, as the number of fully connected layers increases, the output dimension of the fully connected layers decreases. A fully connected layer is a neural network layer where each neuron is connected to all neurons in the previous layer. This means that each neuron in each layer receives the output of all neurons in the previous layer as input and generates an output signal, which is then passed to all neurons in the next layer or used as the final output of the network. The normalization layer is used to adjust the features output by the fully connected layers to maintain them within a preset range, ensuring that the input distribution of each fully connected layer remains relatively stable. Optionally, the normalization layer can be implemented as a batch normalization (BN) layer, which standardizes the features output by the fully connected layers to have zero mean and unit variance.

[0069] In this embodiment, the number of fully connected layers and the number of neurons in each fully connected layer are not limited; they can be flexibly set according to the application scenario and application requirements. The following is combined with... Figure 3b The implementation of the bitrate feature extraction layer is illustrated by an example, such as... Figure 3bAs shown, the bitrate feature extraction layer C1-bitrate feature layer C4 and the bitrate prediction layer are implemented as follows: a fully connected layer and a normalization layer containing 512 neurons (i.e., bitrate feature extraction layer C1), a fully connected layer and a normalization layer containing 512 neurons (i.e., bitrate feature extraction layer C2), a fully connected layer and a normalization layer containing 256 neurons (i.e., bitrate feature extraction layer C3), a fully connected layer and a normalization layer containing 128 neurons (i.e., bitrate feature extraction layer C4), and a fully connected layer and a normalization layer containing 64 neurons (i.e., bitrate prediction layer). After the first concatenated feature is input into the bitrate prediction network, it can be processed sequentially... Figure 3b As shown, the feature dimensions of the output of each fully connected layer gradually decrease, ultimately yielding the target bitrate. Figure 3b The illustration uses a 395-dimensional feature vector as an example of the first concatenation feature, but it is not limited to this feature dimension.

[0070] In addition to the above Figure 3b In another optional embodiment of the network structure shown, when the bitrate prediction network is implemented as an MLP network, the MLP network may include five bitrate feature extraction layers and one bitrate prediction layer. The five bitrate feature extraction layers are bitrate feature extraction layers C11 to C15. The bitrate prediction layers can be implemented as follows: a fully connected layer and a normalization layer containing 1024 neurons (i.e., bitrate feature extraction layer C11), a fully connected layer and a normalization layer containing 512 neurons (i.e., bitrate feature extraction layer C12), a fully connected layer and a normalization layer containing 256 neurons (i.e., bitrate feature extraction layer C13), a fully connected layer and a normalization layer containing 128 neurons (i.e., bitrate feature extraction layer C14), and a fully connected layer and a normalization layer containing 64 neurons (i.e., bitrate feature extraction layer C15). The bitrate prediction layer can be implemented as a fully connected layer and a normalization layer containing 32 neurons. Figure 3a and Figure 3b The number of bitrate feature extraction layers shown, as well as the number of fully connected layers and neurons contained in the bitrate feature extraction layers, allow those skilled in the art to deduce other implementation structures without question.

[0071] In this way, feature information can be concatenated to obtain more accurate input information suitable for the bitrate prediction network. Based on the bitrate prediction network's bitrate feature extraction layer, higher-level features are gradually extracted, thus performing feature extraction more accurately. Furthermore, regression analysis of the relationship between features and bitrate can be performed based on the bitrate prediction layer to obtain a more accurate target bitrate.

[0072] This application does not limit the implementation of quality factor prediction based on content feature information and target bitrate. In some optional embodiments, a pre-trained neural network model, such as a quality factor prediction network, can be used for quality factor prediction. When predicting the quality factor based on content feature information and target bitrate, the content feature information and target bitrate can be preprocessed according to preset feature information processing rules to improve the quality of the input information of the quality factor prediction network, so that the quality factor prediction network can perform quality factor prediction more efficiently and stably. Preprocessing includes various processes such as filtering, denoising, smoothing, or normalization.

[0073] Furthermore, such as Figure 2 As shown, when predicting the quality factor based on content feature information and target bitrate to obtain the target quality factor used for encoding video segments, the content feature information and target bitrate used to reflect content complexity Fc can be used. The features are concatenated to obtain a second concatenated feature; this second concatenated feature is then input into a quality factor prediction network to predict the target quality factor.

[0074] The second concatenation feature refers to the concatenation result obtained by concatenating content feature information and target bitrate. The term "second" is used to define the concatenation feature, distinguishing it only from the "first concatenation feature" mentioned earlier, and does not limit the concatenation method, type, or attribute of the concatenation feature. Specifically, the content feature information and target bitrate can be vector-concatenated at the feature vector level to form a wider feature vector, i.e., the second concatenation feature; alternatively, weights can be assigned to the content feature information and target bitrate respectively, and the content feature information and target bitrate can be weighted and summed according to the assigned weights to obtain the second concatenation feature. This embodiment does not limit the concatenation method. Then, the second concatenation feature can be input into a quality factor prediction network to predict the quality factor and obtain the target quality factor. The quality factor prediction network is trained with the goal of maintaining constant visual quality. In other words, the training objective during the training phase of the quality factor prediction network is the same as the control objective of bitrate control in the aforementioned embodiments, which is to maintain constant visual quality.

[0075] like Figure 4a As shown, the network structure of the quality factor prediction network may include a quality feature extraction layer and a quality factor prediction layer. Figure 4a This example only illustrates the case where there are multiple quality feature extraction layers and does not limit the network structure of the quality factor prediction network. The following will explain in detail the process of quality factor prediction using this network structure:

[0076] First, the second concatenated feature is input into the quality feature extraction layer for feature extraction, resulting in the second target feature. The term "second" is used to define the target feature, distinguishing it only from the first target feature extracted by the bitrate prediction network in the aforementioned embodiment. The quality feature extraction layer can be one or more. In the case of multiple quality feature extraction layers, the input of a subsequent quality feature extraction layer is the output of a previous quality feature extraction layer, used to extract higher-level features based on the previous layer.

[0077] like Figure 4a As shown, since the multiple quality feature extraction layers are, from front to back, quality feature extraction layer D1, quality feature extraction layer D2, quality feature extraction layer D3, and quality feature extraction layer D4, the hierarchical levels of features 11, 12, 13, and 14 (i.e., the second target features) output by quality feature extraction layers D1, D2, D3, and D4 respectively increase sequentially. For a detailed explanation of feature hierarchy, please refer to the aforementioned embodiments; further details will not be repeated here.

[0078] Subsequently, the second target feature can be input into the quality factor prediction layer for regression analysis of the correlation between the feature and the quality factor to obtain the target quality factor. Through a pre-training process, the quality factor prediction network can learn the correlation between features and quality factors through its internal neurons and connection weights. In other words, the trained quality factor prediction network has the ability to determine the target quality factor based on the second target feature. In this embodiment, when the second target feature is input into the quality factor prediction layer for regression analysis of the correlation between the feature and the quality factor, the quality factor prediction layer can determine the target quality factor corresponding to the second target feature based on the pre-learned correlation between the feature and the quality factor and the input second target feature.

[0079] Optionally, the quality factor prediction network in the above embodiments can be implemented as various types of networks, such as MLP networks, convolutional neural networks, recurrent neural networks, or generative adversarial networks, etc., and this embodiment does not impose any limitations. Preferably, when the quality factor prediction network is implemented as an MLP network, the quality feature extraction layer and the quality factor prediction layer can respectively include fully connected layers and normalization layers in the MLP network, and the number of fully connected layers is negatively correlated with the output dimension of the fully connected layers. That is, as the number of fully connected layers increases, the output dimension of the fully connected layers continuously decreases.

[0080] In this embodiment, the number of fully connected layers and the number of neurons in each fully connected layer are not limited; they can be flexibly set according to the application scenario and application requirements. The following is combined with... Figure 4bThe structure of the quality feature extraction layer is illustrated by an example, such as... Figure 4b As shown, the quality feature extraction layers D1-D4 and the quality factor prediction layer are implemented as follows: a fully connected layer and normalization layer containing 512 neurons (i.e., quality feature extraction layer D1), a fully connected layer and normalization layer containing 512 neurons (i.e., quality feature extraction layer D2), a fully connected layer and normalization layer containing 256 neurons (i.e., quality feature extraction layer D3), a fully connected layer and normalization layer containing 128 neurons (i.e., quality feature extraction layer D4), and a fully connected layer and normalization layer containing 64 neurons (i.e., quality factor prediction layer). After the second concatenated feature is input into the quality factor prediction network, it can be processed sequentially... Figure 4b As shown, the feature dimensions of each fully connected layer gradually decrease, ultimately yielding the target quality factor. Figure 4b The illustration uses a 56-dimensional feature vector as an example for the second concatenation feature, but it is not limited to this feature dimension.

[0081] In addition to the above Figure 4b In another optional embodiment of the network structure shown, when the quality factor prediction network is implemented as an MLP network, the MLP network may include five quality feature extraction layers and one quality factor prediction layer. The five quality feature extraction layers are quality feature extraction layers D11 to D15. The quality factor prediction layers may be implemented as follows: a fully connected layer and a normalization layer containing 1024 neurons (i.e., quality feature extraction layer D11), a fully connected layer and a normalization layer containing 512 neurons (i.e., quality feature extraction layer D12), a fully connected layer and a normalization layer containing 256 neurons (i.e., quality feature extraction layer D13), a fully connected layer and a normalization layer containing 128 neurons (i.e., quality feature extraction layer D14), and a fully connected layer and a normalization layer containing 64 neurons (i.e., quality feature extraction layer D15). The quality factor prediction layer may be implemented as a fully connected layer and a normalization layer containing 32 neurons. This embodiment does not impose any restrictions on the specific network structure of the MLP network, such as the number of quality feature extraction layers, the number of neurons in each quality feature extraction layer, the number of quality factor prediction layers, and the number of neurons in each quality factor prediction layer.

[0082] In this way, feature information splicing operations can be used to obtain input information more accurately that is adapted to the quality factor prediction network. Based on the quality feature extraction layer of the quality factor prediction network, higher-level features are gradually extracted, thereby performing feature extraction more accurately. Then, the quality factor prediction layer can be used to perform regression analysis on the relationship between features and quality factors to obtain a more accurate target quality factor.

[0083] Using the above Figure 2 Before the two-stage neural network model shown, supervised training can be performed on the initial model using a labeled sample set to obtain a complete neural network model containing a feature extraction network, a bitrate prediction network, and a quality factor prediction network. This complete neural network model is used for bitrate control and can therefore also be called a neural network model for bitrate control. The following will combine steps R1-R4 and... Figure 5a The training process of the bitrate prediction network and quality factor prediction network in the foregoing embodiments is described in detail below:

[0084] Step R1: Acquire multiple sample videos. This application embodiment does not limit the type or content of the sample videos, nor does it limit the method of acquiring them. Sample videos can be acquired from the internet or relevant video libraries, or they can be acquired or captured in real-time using video capture or shooting tools as needed.

[0085] Step R2: Based on the specified visual quality, select a baseline bitrate from the different bitrates used for encoding multiple sample videos according to the VBR bitrate control algorithm. Specifically, this step can be implemented based on the following steps R21-R22:

[0086] Step R21: Encode multiple sample videos using different bitrates according to a variable bitrate control algorithm to obtain multiple first sample encoded bitstreams. Different bitrates can be used as bitrate constraints. The variable bitrate control algorithm controls the bitrate during the encoder's encoding of multiple sample videos, so that each sample video can be encoded by the encoder into a first sample encoded bitstream.

[0087] like Figure 5a As shown, the sample video sequence includes sample video X1, sample video X2... sample video X... i Given a total of i sample videos, the encoder encodes each of the i sample videos to obtain i first sample encoded bitstreams, where the sample video sequence is {X1, X2, ..., X...}. i The corresponding bitrates are {b1, b2, ..., b}, respectively. j …}

[0088] Step R22: Based on the decoding results of multiple first sample encoded bitstreams, select the bitrate from different bitrates whose decoding results meet the specified visual quality requirements, and use it as the reference bitrate.

[0089] like Figure 5a As shown, a decoder can be used to decode multiple first-sample encoded bitstreams, resulting in multiple decoded first-sample encoded bitstreams. Because different bitrates are used during encoding, the visual quality of the resulting multiple first-sample encoded bitstreams is also different. For example... Figure 5aAs shown, the visual quality of the multiple decoding results obtained by decoding multiple first sample encoded bitstreams is as follows: Based on this, visual quality v can be selected from the decoding results of multiple first sample encoded bitstreams according to Formula 1. * The difference between them meets the set requirements for the first target decoding result, and the bitrate corresponding to the first target decoding result is used as the reference bitrate, that is, from {b1, b2…b j A baseline bitrate is selected from ...}. In Formula 1, the visual quality is related to the specified visual quality v. * The difference between the values ​​meets the set requirements, meaning the difference is less than the set target difference ∈, but it is not limited to this. It can also be the difference between the visual quality and the specified visual quality v. * The difference between them should be minimal, or within a set range, etc.

[0090]

[0091] In Formula 1, ∈ represents the preset target difference, which can be set to any value according to actual design requirements, such as 1, 2 or 3, etc. This embodiment does not impose any restrictions.

[0092] After determining the baseline bitrate relatively accurately based on steps R21-R22, this baseline bitrate is the ideal bitrate corresponding to the sample video while meeting the specified visual quality, and it is also the label bitrate that the model needs to refer to during training. Further, step R3 can be performed to determine the baseline quality factor more accurately.

[0093] Step R3: Based on the specified visual quality, select a baseline quality factor from the different quality factors used in encoding multiple sample videos according to a quality-constant algorithm. Specifically, this can be achieved based on the following steps R31-R32:

[0094] Step R31: Encode multiple sample videos using different quality factors according to a constant quality algorithm to obtain multiple second sample encoded bitstreams. For example, Figure 5a As shown, different quality factors can be used, and a constant quality algorithm can be employed to control the bitrate during the encoder's encoding of multiple sample videos. Thus, each sample video can be encoded by the encoder into a second sample encoded bitstream.

[0095] like Figure 5a As shown, the encoder can use different quality factors for the sample video sequence {X1, X2…X... i Encoding the i sample videos in the sequence {X1, X2, ..., X} yields the corresponding i second sample encoded bitstreams. i The corresponding quality factors are {c1, c2, ..., c}, respectively. j …}

[0096] Step R32: Based on the decoding results of multiple second sample encoded bitstreams, select the corresponding quality factor from different quality factors that meets the specified visual quality requirements and use it as the benchmark quality factor.

[0097] like Figure 5a As shown, a decoder can be used to decode multiple second-sample encoded bitstreams, resulting in multiple decoded second-sample encoded bitstreams. Because different quality factors are used during the encoding process, the visual quality of the resulting multiple second-sample encoded bitstreams is also different. For example... Figure 5a As shown, the visual quality of the multiple decoding results obtained by decoding multiple second sample encoded bitstreams is as follows: Based on this, visual quality v can be selected from the decoding results of multiple second sample encoded bitstreams using Formula 2. * The difference between them meets the set requirements for the second target decoding result, and the quality factor corresponding to the second target decoding result is used as the reference bitrate, that is, from {c1, c2…c… j The base bit rate is selected from ...}.

[0098]

[0099] Through steps R31-R32 above, the baseline quality factor can be determined relatively accurately. This baseline quality factor is the ideal quality factor corresponding to the sample video under the specified visual quality conditions, and it is also the label quality factor that needs to be referenced for model training. Based on this, the sample videos can be labeled. After labeling the sample videos, the following step R4 can be performed based on the baseline bitrate and baseline quality factor to train the model:

[0100] Step R4: Train the model based on multiple sample videos, baseline bitrate, and baseline quality factor to obtain the bitrate prediction network and the quality factor prediction network.

[0101] The model training process includes: constructing an initial model architecture, including a feature extraction network, a first initial network for bitrate prediction, and a second initial network for quality factor prediction; then, training the initial model architecture using multiple sample videos until the loss function of the model training meets the set conditions, thus obtaining the bitrate prediction network and the quality factor prediction network.

[0102] Specifically, for any given sample video, sample content feature information and sample encoding feature information are extracted. The sample content feature information reflects the content complexity of the sample video, and the sample encoding feature information reflects the encoding complexity of the sample video. Next, bitrate prediction is performed based on the sample content feature information and sample encoding feature information to obtain the sample bitrate used for encoding the sample video; quality factor prediction is performed based on the sample content feature information and the target bitrate to obtain the sample quality factor used for encoding the sample video.

[0103] Furthermore, such as Figure 5a As shown, after obtaining the sample bitrate during model training, a loss function between the sample bitrate and the baseline bitrate is calculated. The weight matrix of the initial network is updated until this loss function converges to a preset first error range, aiming to obtain the bitrate prediction network. The bitrate loss function can include one or more of L1 and L2 loss functions, etc., and this embodiment is not limited to any one of them. The L1 loss function is also called the absolute error loss function, and the L2 loss function is also called the mean squared error loss function.

[0104] Furthermore, such as Figure 5a As shown, after obtaining the sample quality factor during model training, a loss function between the sample quality factor and the benchmark quality factor can be calculated. The weight matrix of the second initial network is updated with the goal of the loss function converging to a preset second error range. This process continues until the loss function between the sample quality factor and the benchmark quality factor converges to the preset second error range, thus obtaining the quality factor prediction network. The quality factor loss function can include one or more of L1 loss functions, L2 loss functions, etc., and this embodiment does not impose any limitations.

[0105] In this way, the bitrate prediction network and the quality factor prediction network can be trained more accurately based on the bitrate loss function and the quality factor loss function.

[0106] In some optional embodiments, the aforementioned embodiment of "using the target bitrate as a bitrate constraint and based on the target quality factor to perform bitrate control on the encoding process of video segments with the goal of maintaining constant visual quality" can be implemented based on the following steps K1-K2:

[0107] Step K1: Based on the target bitrate, generate the maximum bitrate allowed during the bitrate control process with the goal of maintaining constant visual quality. In other words, the maximum bitrate is the bitrate constraint during the bitrate control process, and the variable bitrate during the bitrate control process cannot exceed this maximum bitrate.

[0108] The target bitrate can be directly used as the maximum bitrate, or a preset correction coefficient can be used to correct the target bitrate to obtain the maximum bitrate. The correction coefficient can be set to any value according to actual design requirements, such as 1.05, 1.1, or 1.15, etc. Taking a correction coefficient of 1.1 as an example, the maximum bitrate obtained by correcting the target bitrate is: 1.1 × target bitrate. The above is only an illustrative example, and this embodiment does not limit the specific method of generating the maximum bitrate.

[0109] Step K2: Based on the target quality factor and maximum bitrate, perform bitrate control on the video segment encoding process with the goal of maintaining a constant specified visual quality. Specifically, the maximum bitrate can be used as a bitrate constraint, and the bitrate control on the video segment encoding process can be performed based on the target quality factor with the goal of maintaining a constant specified visual quality.

[0110] In this way, electronic devices can generate the maximum bitrate in the bitrate control process more accurately, and use the maximum bitrate to constrain the bitrate control process, thereby saving encoding bandwidth while ensuring video quality.

[0111] In this embodiment, the target bitrate can be used as a default bitrate constraint. Based on the target quality factor, bitrate control is applied to the encoding process of the video segment with the goal of maintaining constant visual quality. However, this is not a limitation. Given the target bitrate and target quality factor, the target bitrate can be used alone for variable bitrate control of the video segment's encoding process, or the target quality factor can be used alone for constant quality bitrate control. Therefore, in some optional embodiments, multiple bitrate control algorithms can be pre-set in the encoder to meet the bitrate control requirements of different video segments, such as variable bitrate control algorithms, constant bitrate (CBR) algorithms, and different constant quality bitrate control algorithms.

[0112] The fixed bitrate algorithm controls the bitrate of a video segment to remain constant throughout the encoding process, regardless of changes in content complexity. This method, in contrast to the variable bitrate / variable bitrate control algorithm, works as follows: Constant Bitrate: The fixed bitrate algorithm ensures that the bitrate of the encoded output remains at the same level throughout the encoding process. This means that regardless of whether the video or audio content is simple or complex, the amount of data transmitted per second is constant. Predictable File Size: Because the bitrate is fixed, the size of the output file can be roughly estimated before encoding. Compatibility: Some low-configuration electronic devices have limited ability to handle dynamic bitrates, while the fixed bitrate algorithm is better suited for these devices.

[0113] The variable bitrate (VBR) algorithm dynamically adjusts the number of bits per second required based on the complexity of the signal content to achieve a predetermined quality target while optimizing storage space or bandwidth usage. Specifically, the VBR algorithm is implemented based on the following steps: At the start of the encoding process, the VBR encoder first analyzes the complexity of the input data (e.g., video clips). Based on the results of this complexity analysis, the VBR encoder can allocate different bitrates to different parts. For complex scenes with high content complexity, more bits can be allocated to maintain high quality; while for parts with lower content complexity, bit usage is reduced, thereby balancing the overall quality and size of the entire file.

[0114] In this context, a constant quality code control algorithm refers to a code control algorithm that aims to maintain a constant target quality. In this embodiment, constant quality code control algorithms can be categorized based on whether or not they have bitrate constraints. For example, there is a first constant quality code control algorithm with bitrate constraints and a second constant quality code control algorithm without bitrate constraints. The terms "first" and "second" are used to differentiate between constant quality code control algorithms with different bitrate constraints. The first constant quality code control algorithm refers to a constant quality code control algorithm with bitrate constraints, and more specifically, a constant quality code control algorithm with a target bitrate calculated in the first stage as a bitrate constraint. Specifically, it refers to the code control process provided in this application embodiment, which involves "using the target bitrate as a bitrate constraint and, based on the target quality factor, performing bitrate control on the encoding process of a video segment with the goal of maintaining a constant visual quality." The second constant quality code control algorithm refers to a traditional constant quality code control algorithm without bitrate constraints, such as the traditional CRF code control algorithm.

[0115] Based on this, before performing bitrate control on the video segment encoding process with the target bitrate as a bitrate constraint and the target quality factor as the target quality constant, the target bitrate control algorithm applicable to the video segment can also be determined. Different target bitrate control algorithms can correspond to different bitrate control operations.

[0116] Optionally, considering that the system configuration requirements of the various pre-configured code control algorithms are different, the system configuration file can be obtained, and the pre-configured code control algorithm can be obtained from the system configuration file as the target code control algorithm.

[0117] Optionally, users can pre-configure the bitrate control algorithm on the configuration interface provided by the encoder or operating system. For example, a list of bitrate control algorithms can be displayed to the user, from which the user can select the target bitrate control algorithm. Of course, users can also perform management operations such as adding, deleting, or replacing bitrate control algorithms in the algorithm list as needed. Optionally, users can also pre-set the appropriate video application scenario or corresponding video segment type for each bitrate control algorithm in the algorithm list. Accordingly, the target bitrate control algorithm can be automatically selected for the video segment based on the video application scenario or type to which the video segment belongs. For example, in application scenarios with many video processing tasks, considering that the scenario requires a fast video processing speed, the corresponding bitrate control algorithm can be a fixed bitrate algorithm; as another example, in application scenarios with fewer video processing tasks, the corresponding bitrate control algorithm can be a variable bitrate bitrate control algorithm.

[0118] Optionally, the target bitrate control algorithm applicable to a video segment can be determined based on a pre-defined correspondence between bitrate control algorithms and video segment types. For example, if the video segment type is a long video, the corresponding bitrate control algorithm could be a fixed bitrate algorithm; if the video segment type is a short video, the corresponding bitrate control algorithm could be a constant quality bitrate control algorithm. Similarly, if the video segment type is educational, the corresponding bitrate control algorithm could be a constant quality bitrate control algorithm; if the video segment type is advertising, the corresponding bitrate control algorithm could be a fixed bitrate algorithm. Furthermore, if the video segment type is high-complexity, the corresponding bitrate control algorithm could be a constant quality bitrate control algorithm; if the video segment type is low-complexity, the corresponding bitrate control algorithm could be a fixed bitrate algorithm.

[0119] The above is merely an illustrative example and does not limit the specific method for determining the target code control algorithm. Next, we will provide illustrative examples of code control operations corresponding to different code control algorithms, considering various scenarios:

[0120] Case 1: When the target bitrate control algorithm is the first quality constant bitrate control algorithm, the bitrate control operation is performed on the video segment encoding process with the target bitrate as the bitrate constraint and the target quality factor as the target quality constant.

[0121] The first constant quality bitrate control algorithm is a constant quality bitrate control algorithm that uses the predicted target bitrate as the bitrate constraint. In other words, before executing step 14, "using the target bitrate as the bitrate constraint and the target quality factor to perform bitrate control on the encoding process of the video segment with the goal of constant visual quality," this embodiment needs to determine what kind of algorithm the current target bitrate control algorithm is. If it is the first constant quality algorithm, then step 14 can be executed.

[0122] Scenario 2: When the target codec algorithm is the second constant quality codec algorithm, the bitrate of the video segment encoding process is controlled according to the target quality factor, with the goal of maintaining constant visual quality. This second constant quality codec algorithm is a constant quality codec algorithm without bitrate constraints. Specifically, the second constant quality codec algorithm may not contain a default maximum bitrate parameter and is not subject to bitrate constraints from a default maximum bitrate; alternatively, it may contain a default maximum bitrate parameter, and the state of no bitrate constraint can be achieved by setting the default maximum bitrate to 0. This embodiment does not impose any restrictions.

[0123] Scenario 3: When the target bitrate control algorithm is a variable bitrate control algorithm, the encoding process of the video segment is subject to variable bitrate control based on the target bitrate. In other words, in this case, when performing variable bitrate control on the encoding process of the video segment based on the target bitrate, the goal is not to maintain a constant quality factor, but rather to directly perform variable bitrate control on the encoding process of the video segment under the constraint of the target bitrate.

[0124] In this way, multiple code control modes (i.e., multiple code control algorithms) can be preset to adapt to the code control needs of different scenarios or different types of video segments.

[0125] The following will combine Figure 5b as well as Figure 5c The above process of this solution will be further explained in different scenarios.

[0126] like Figure 5b As shown in the embodiments of this application, a data storage method applied in a smart storage scenario is also provided. This method can be executed by a smart storage device in the cloud, and the method includes:

[0127] Step 51: Receive the video data to be stored, which includes at least one video segment.

[0128] Step 52: Encode at least one video segment, and use the bitrate control method in the aforementioned embodiments to control the bitrate during the encoding process, so as to obtain at least one encoded video segment.

[0129] Step 53: Store at least one encoded video segment.

[0130] This embodiment does not limit the specific implementation of intelligent storage scenarios and video data. In some exemplary embodiments, intelligent storage scenarios can be various application scenarios such as cloud photo album scenarios, cloud storage scenarios, and media data management scenarios. Among them, in the cloud photo album scenario, in addition to providing basic functions such as image file storage and management, it can also support intelligent analysis such as classification and tagging of image content and face recognition, and can also provide intelligent search services based on natural language understanding.

[0131] In different smart storage scenarios, the video data to be stored can also be of different types:

[0132] In an exemplary embodiment, in a cloud photo album or cloud storage scenario, a user terminal device can send video data to a smart storage device. The video data can be user-created video data from the user terminal device's photo album, or it can be target video data generated by the terminal device based on the user's historical videos, such as an annual summary video generated based on videos from the past year. The smart storage device can be implemented as a cloud-based smart storage device. The smart storage device can receive the video data, encode at least one video segment from the video data, and store it in the corresponding cloud photo album storage space, so that the user can retrieve the encoded video segment from the cloud photo album storage space and decode and play it as needed.

[0133] In an exemplary embodiment, in a media data management scenario, the intelligent storage device can receive video data sent from any type of data source. The data source can be a live streaming device, a video-on-demand device, a directory monitor, or a user terminal; this embodiment is not limited to these. Correspondingly, the video data can be video clips from a live streaming application sent by a live streaming device, video-on-demand videos from a live streaming application sent by a video-on-demand device, user-shot videos sent by a user terminal, or newly added video resources on the terminal device where the directory monitor is located, also without limitation in this embodiment.

[0134] The detailed implementation methods and beneficial effects of each step in this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0135] In this way, encoding and storing the video data to be stored can effectively save storage resources. Furthermore, by adopting the encoding control method provided in the embodiments of this application, encoding bandwidth can be saved, so that each frame in the final encoding result can meet the specified visual quality, thereby balancing video quality and transmission efficiency.

[0136] In addition to the data storage methods described above, this application also provides a video processing method for use in intelligent media scenarios. This method can be executed by an intelligent media device in the cloud, and may include the following steps:

[0137] Step 61: Receive video data sent from the data source. The video data includes at least one video segment.

[0138] Step 62: Encode at least one video segment, and use the bitrate control method in the aforementioned embodiments to control the bitrate during the encoding process, so as to obtain at least one encoded video segment.

[0139] Step 63: Receive a data acquisition request sent by the data requester. The data acquisition request includes the identification information of the video segment to be acquired.

[0140] Step 64: Send at least one encoded video segment corresponding to the identification information to the data requester.

[0141] This application does not limit the specific implementation of the data source; refer to... Figure 5c For example, the data source can be a live streaming client, and the video data sent from the live streaming client to the smart media device can be the live video from the live streaming client; the data source can also be a user terminal, and the video data sent from the user terminal to the smart media device can be user-generated content (UGC), i.e., video data created by the user; the data source can also be a video-on-demand client, and the video data sent from the video-on-demand client to the smart media device can be the film and television data stored on the video-on-demand client; the data source can also be a directory monitor, and the video data sent from the directory monitor to the smart media device can be: the video data added by the user in the directory detected by the directory monitor.

[0142] Reference Figure 5c For example, a smart media device can encode at least one video segment from video data, and during the encoding process, use the bitrate control methods described in the aforementioned embodiments to control the bitrate, thereby obtaining at least one encoded video segment, and then store the data accordingly. A data requester can send a data acquisition request to the smart media device according to its own needs. The smart media device can receive the data acquisition request, which may include identification information of the requested video segment. This identification information is used to identify the identity attributes of the requested video segment. Furthermore, based on the identification information, the device can determine the video segment corresponding to the identification information from at least one encoded video segment stored internally and send it to the data requester.

[0143] This application does not limit the specific implementation of the data demand side. In some optional embodiments, the data demand side can be a live streaming viewing terminal to obtain encoded live video; the data demand side can also be a video-on-demand viewing terminal to obtain encoded video-on-demand; the data demand side can also be implemented as a Content Delivery Network (CDN) to obtain video segments corresponding to the identification information and distribute them to the corresponding terminal devices. The CDN is used to cache the received video segments and redirect the video segments to the nearest node according to the user's location. In addition, the data demand side can be any audio or video terminal, such as a projector or television, etc., and this embodiment does not impose any restrictions.

[0144] The detailed implementation methods and beneficial effects of each step in this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0145] In this way, on the one hand, encoding and storing the video data to be stored can effectively save storage resources; on the other hand, the encoding control method provided in this application embodiment can save encoding bandwidth, so that each frame in the final encoding result can meet the specified visual quality, thereby balancing video quality and transmission efficiency; and on the other hand, based on the scheme architecture provided in this embodiment, different data demand ends can obtain the required encoded video data on demand.

[0146] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 11 to 14 can be device A; or the execution subject of steps 11 and 12 can be device A, and the execution subject of steps 13 to 14 can be device B; and so on.

[0147] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 11, 12, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0148] Figure 6 This is a schematic diagram of a bitrate control device provided as another exemplary embodiment of this application. (See diagram below.) Figure 6 As shown, the device includes: a feature extraction module 601, a bitrate prediction module 602, a quality prediction module 603, and a bitrate control module 604.

[0149] The feature extraction module 601 is used to extract content feature information and encoding feature information from the video segment to be encoded. The content feature information reflects the content complexity of the video segment, and the encoding feature information reflects the encoding complexity of the video segment. The bitrate prediction module 602 is used to predict the bitrate based on the content feature information and the encoding feature information to obtain a target bitrate for encoding the video segment. The quality prediction module 603 is used to predict the quality factor based on the content feature information and the target bitrate to obtain a target quality factor for encoding the video segment. The bitrate control module 604 is used to control the bitrate of the video segment with the target bitrate as a bitrate constraint and the target quality factor, aiming to maintain a constant visual quality.

[0150] Optionally, when the bitrate prediction module 602 performs bitrate prediction based on the content feature information and the encoding feature information to obtain the target bitrate for encoding the video segment, it is specifically configured to: concatenate the content feature information and the encoding feature information to obtain a first concatenated feature; input the first concatenated feature into the bitrate prediction network to perform bitrate prediction to obtain the target bitrate; wherein the bitrate prediction network is trained with the specified visual quality constant as the target.

[0151] Optionally, the bitrate prediction network includes a bitrate feature extraction layer and a bitrate prediction layer; when the bitrate prediction module 602 inputs the first concatenated feature into the bitrate prediction network for bitrate prediction to obtain the target bitrate, it is specifically used to: input the first concatenated feature into the bitrate feature extraction layer for feature extraction to obtain a first target feature; input the first target feature into the bitrate prediction layer for regression analysis of the correlation between the feature and the bitrate to obtain the target bitrate; wherein, when there are multiple bitrate feature extraction layers, the input of the subsequent bitrate feature extraction layer is the output of the previous bitrate feature extraction layer, which is used to extract higher-level features based on the previous bitrate feature extraction layer.

[0152] Optionally, the bitrate prediction network is implemented as a multilayer perceptron (MLP) network, wherein the bitrate feature extraction layer and the bitrate prediction layer respectively include fully connected layers and normalization layers in the MLP network, and the number of fully connected layers is negatively correlated with the output dimension of the fully connected layers.

[0153] Optionally, when the quality prediction module 603 performs quality factor prediction based on the content feature information and the target bitrate to obtain the target quality factor for encoding the video segment, it is specifically used to: concatenate the content feature information and the target bitrate to obtain a second concatenation feature; input the second concatenation feature into the quality factor prediction network to perform quality factor prediction to obtain the target quality factor; wherein, the quality factor prediction network is trained with the specified visual quality constant as the target.

[0154] Optionally, the quality factor prediction network includes a quality feature extraction layer and a quality factor prediction layer; when the quality prediction module 603 inputs the second concatenated feature into the quality factor prediction network to predict the quality factor and obtain the target quality factor, it is specifically used to: input the second concatenated feature into the quality feature extraction layer to extract features and obtain a second target feature; input the second target feature into the quality factor prediction layer to perform regression analysis on the correlation between the feature and the quality factor to obtain the target quality factor; wherein, when there are multiple quality feature extraction layers, the input of the subsequent quality feature extraction layer is the output of the previous quality feature extraction layer, which is used to extract higher-level features based on the previous quality feature extraction layer.

[0155] Optionally, the quality factor prediction network is implemented as a multilayer perceptron (MLP) network, wherein the quality feature extraction layer and the quality factor prediction layer respectively include fully connected layers and normalization layers in the MLP network, and the number of fully connected layers is negatively correlated with the output dimension of the fully connected layers.

[0156] Optionally, the feature extraction module 601 is further configured to: acquire multiple sample videos; select a baseline bitrate from different bitrates used for encoding the multiple sample videos according to a variable bitrate control algorithm based on the specified visual quality; select a baseline quality factor from different quality factors used for encoding the multiple sample videos according to a constant quality algorithm based on the specified visual quality; and train a model based on the multiple sample videos, the baseline bitrate, and the baseline quality factor to obtain the bitrate prediction network and the quality factor prediction network.

[0157] Optionally, when the feature extraction module 601 selects a base bitrate from different bitrates used for encoding the plurality of sample videos according to the specified visual quality, it specifically performs the following steps: encoding the plurality of sample videos using different bitrates according to the variable bitrate control algorithm to obtain a plurality of first sample encoded bitstreams; selecting a bitrate whose decoding result satisfies the specified visual quality requirement from the different bitrates based on the decoding result of the plurality of first sample encoded bitstreams, as the base bitrate; and selecting a base quality factor from different quality factors used for encoding the plurality of sample videos according to the specified visual quality, including: encoding the plurality of sample videos using different quality factors according to the quality constant algorithm to obtain a plurality of second sample encoded bitstreams; and selecting a quality factor whose decoding result satisfies the specified visual quality requirement from the different quality factors based on the decoding result of the plurality of second sample encoded bitstreams, as the base quality factor.

[0158] Optionally, when the bitrate control module 604 performs bitrate control on the encoding process of the video segment with the target bitrate as a bitrate constraint and the target quality factor as a basis, aiming to maintain a constant visual quality, it is specifically used to: generate the maximum bitrate allowed in the bitrate control process with the target bitrate as a basis; and perform bitrate control on the encoding process of the video segment with the target visual quality as a basis, based on the target quality factor and the maximum bitrate.

[0159] Optionally, when the bitrate control module 604 generates the maximum bitrate allowed to be used in the bitrate control process with the target bitrate as the goal of constant visual quality, it is specifically used to: directly use the target bitrate as the maximum bitrate.

[0160] Optionally, before performing bitrate control on the encoding process of the video segment with the target bitrate as a bitrate constraint and based on the target quality factor, the bitrate control module 604 is further configured to: determine a target bitrate control algorithm applicable to the video segment; and, if the target bitrate control algorithm is a first quality constant bitrate control algorithm, perform the operation of performing bitrate control on the encoding process of the video segment with the target bitrate as a bitrate constraint and based on the target quality factor, with the target bitrate as a bitrate constraint and based on the target quality factor, wherein the first quality constant bitrate control algorithm is a quality constant bitrate control algorithm that uses the predicted target bitrate as a bitrate constraint.

[0161] The bitrate control module 604 is further configured to: when the target bitrate control algorithm is a second constant quality bitrate control algorithm, perform bitrate control on the encoding process of the video segment with the target constant visual quality as the objective, based on the target quality factor, wherein the second constant quality bitrate control algorithm is a constant quality bitrate control algorithm without bitrate constraints; and when the target bitrate control algorithm is a variable bitrate bitrate control algorithm, perform variable bitrate control on the encoding process of the video segment based on the target bitrate.

[0162] In this embodiment, content feature information and encoding feature information can be extracted from the video segment to be encoded. The content feature information reflects the content complexity of the video segment, and the encoding feature information reflects the encoding complexity of the video segment. Based on the content feature information and encoding feature information, bitrate prediction is performed to obtain the target bitrate for video encoding. Based on the content feature information and the target bitrate, the target quality factor for video encoding is predicted. Using the target bitrate as a bitrate constraint, and based on the target quality factor, bitrate control is performed on the video segment encoding process with the goal of maintaining a constant specified visual quality. On the one hand, encoding based on a variable bitrate can save encoding bandwidth; on the other hand, based on the goal of maintaining a constant specified visual quality, each frame in the final encoded result can meet the specified visual quality, thereby balancing video quality and transmission efficiency.

[0163] The above describes the internal functions and structure of the bitrate control device, such as... Figure 7 As shown, in practice, the bit rate control device can be implemented as an electronic device, including: a memory 701, a processor 702, and a communication component 703.

[0164] Memory 701 is used to store computer programs and can be configured to store various other data to support operation on the computing platform. Examples of this data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0165] The memory 701 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0166] Processor 702, coupled to memory 701, is configured to execute a computer program in memory 701 for: extracting content feature information and encoding feature information from a video segment to be encoded, wherein the content feature information reflects the content complexity of the video segment and the encoding feature information reflects the encoding complexity of the video segment; performing bitrate prediction based on the content feature information and the encoding feature information to obtain a target bitrate for encoding the video segment; performing quality factor prediction based on the content feature information and the target bitrate to obtain a target quality factor for encoding the video segment; and, using the target bitrate as a bitrate constraint and based on the target quality factor, performing bitrate control on the encoding process of the video segment with the goal of maintaining a constant specified visual quality.

[0167] Optionally, when the processor 702 performs bitrate prediction based on the content feature information and the encoding feature information to obtain a target bitrate for encoding the video segment, it specifically performs the following steps: concatenating the content feature information and the encoding feature information to obtain a first concatenated feature; inputting the first concatenated feature into a bitrate prediction network to perform bitrate prediction to obtain the target bitrate; wherein the bitrate prediction network is trained with the goal of maintaining a constant specified visual quality.

[0168] Optionally, the bitrate prediction network includes a bitrate feature extraction layer and a bitrate prediction layer; when the processor 702 inputs the first concatenated feature into the bitrate prediction network to perform bitrate prediction to obtain the target bitrate, it is specifically used to: input the first concatenated feature into the bitrate feature extraction layer to perform feature extraction to obtain a first target feature; input the first target feature into the bitrate prediction layer to perform regression analysis on the correlation between the feature and the bitrate to obtain the target bitrate; wherein, when there are multiple bitrate feature extraction layers, the input of the subsequent bitrate feature extraction layer is the output of the previous bitrate feature extraction layer, which is used to extract higher-level features based on the previous bitrate feature extraction layer.

[0169] Optionally, the bitrate prediction network is implemented as a multilayer perceptron (MLP) network, wherein the bitrate feature extraction layer and the bitrate prediction layer respectively include fully connected layers and normalization layers in the MLP network, and the number of fully connected layers is negatively correlated with the output dimension of the fully connected layers.

[0170] Optionally, when the processor 702 performs quality factor prediction based on the content feature information and the target bitrate to obtain the target quality factor for encoding the video segment, it specifically performs the following steps: concatenating the content feature information and the target bitrate to obtain a second concatenated feature; inputting the second concatenated feature into a quality factor prediction network to perform quality factor prediction to obtain the target quality factor; wherein the quality factor prediction network is trained with the specified visual quality constant as the target.

[0171] Optionally, the quality factor prediction network includes a quality feature extraction layer and a quality factor prediction layer; when the processor 702 inputs the second concatenated feature into the quality factor prediction network to predict the quality factor and obtain the target quality factor, it is specifically used to: input the second concatenated feature into the quality feature extraction layer to extract features and obtain a second target feature; input the second target feature into the quality factor prediction layer to perform regression analysis on the correlation between the feature and the quality factor to obtain the target quality factor; wherein, when there are multiple quality feature extraction layers, the input of the subsequent quality feature extraction layer is the output of the previous quality feature extraction layer, which is used to extract higher-level features based on the previous quality feature extraction layer.

[0172] Optionally, the quality factor prediction network is implemented as a multilayer perceptron (MLP) network, wherein the quality feature extraction layer and the quality factor prediction layer respectively include fully connected layers and normalization layers in the MLP network, and the number of fully connected layers is negatively correlated with the output dimension of the fully connected layers.

[0173] Optionally, the processor 702 is further configured to: acquire multiple sample videos; select a baseline bitrate from different bitrates used for encoding the multiple sample videos according to a variable bitrate control algorithm based on the specified visual quality; select a baseline quality factor from different quality factors used for encoding the multiple sample videos according to a constant quality algorithm based on the specified visual quality; and train a model based on the multiple sample videos, the baseline bitrate, and the baseline quality factor to obtain the bitrate prediction network and the quality factor prediction network.

[0174] Optionally, when the processor 702 selects a base bitrate from different bitrates used for encoding the plurality of sample videos according to a variable bitrate control algorithm based on the specified visual quality, it specifically performs the following steps: encoding the plurality of sample videos using different bitrates according to a variable bitrate control algorithm to obtain a plurality of first sample encoded bitstreams; selecting a bitrate whose decoding result satisfies the specified visual quality requirement from the different bitrates based on the decoding result of the plurality of first sample encoded bitstreams, as the base bitrate; and selecting a base quality factor from different quality factors used for encoding the plurality of sample videos according to a constant quality algorithm based on the specified visual quality, including: encoding the plurality of sample videos using different quality factors according to a constant quality algorithm to obtain a plurality of second sample encoded bitstreams; and selecting a quality factor whose decoding result satisfies the specified visual quality requirement from the different quality factors based on the decoding result of the plurality of second sample encoded bitstreams, as the base quality factor.

[0175] Optionally, when the processor 702 performs bitrate control on the encoding process of the video segment with the target bitrate as a bitrate constraint and based on the target quality factor, aiming to maintain a constant visual quality, it is specifically used to: generate the maximum bitrate allowed in the bitrate control process with the target bitrate as the target; and perform bitrate control on the encoding process of the video segment with the target visual quality as the target, based on the target quality factor and the maximum bitrate.

[0176] Optionally, when the processor 702 generates the maximum bitrate allowed in the bitrate control process with the goal of maintaining constant visual quality based on the target bitrate, it is specifically used to: directly use the target bitrate as the maximum bitrate.

[0177] Optionally, before performing bitrate control on the encoding process of the video segment with the target bitrate as a bitrate constraint and based on the target quality factor, the processor 702 is further configured to: determine a target bitrate control algorithm applicable to the video segment; if the target bitrate control algorithm is a first quality-constant bitrate control algorithm, perform the operation of performing bitrate control on the encoding process of the video segment with the target bitrate as a bitrate constraint and based on the target quality factor, with the target bitrate as a bitrate constraint, wherein the first quality-constant bitrate control algorithm is a quality-constant bitrate control algorithm with the predicted target bitrate as a bitrate constraint. The processor 702 is further configured to: if the target bitrate control algorithm is a second quality-constant bitrate control algorithm, perform bitrate control on the encoding process of the video segment with the target quality factor, with the second quality-constant bitrate control algorithm being a quality-constant bitrate control algorithm without bitrate constraints; if the target bitrate control algorithm is a variable bitrate control algorithm, perform variable bitrate control on the encoding process of the video segment based on the target bitrate.

[0178] Furthermore, such as Figure 7 As shown, the electronic device also includes other components such as a power supply component 704, a display 705, and an audio component 706. Figure 7 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 7 The components shown. Additionally... Figure 7 The components within the dashed box are optional, not mandatory, and their specific requirements depend on the product form of the work node. In this embodiment, the work node can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device, or a server-side device such as a conventional server, cloud server, or server array. If the work node in this embodiment is implemented as a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 7 The components within the dashed box; if the working node in this embodiment is implemented as a server-side device such as a conventional server, cloud server, or server array, it may be omitted. Figure 7 The component within the dashed box.

[0179] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can perform the steps that can be executed by an electronic device in the above method embodiments.

[0180] Accordingly, this application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, performs the steps in the bitrate control method.

[0181] In this embodiment, content feature information and encoding feature information can be extracted from the video segment to be encoded. The content feature information reflects the content complexity of the video segment, and the encoding feature information reflects the encoding complexity of the video segment. Based on the content feature information and encoding feature information, bitrate prediction is performed to obtain the target bitrate for video encoding. Based on the content feature information and the target bitrate, the target quality factor for video encoding is predicted. Using the target bitrate as a bitrate constraint, and based on the target quality factor, bitrate control is performed on the video segment encoding process with the goal of maintaining a constant specified visual quality. On the one hand, encoding based on a variable bitrate can save encoding bandwidth; on the other hand, based on the goal of maintaining a constant specified visual quality, each frame in the final encoded result can meet the specified visual quality, thereby balancing video quality and transmission efficiency.

[0182] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0183] The aforementioned communication components are configured to facilitate wired or wireless communication between the device containing the communication components and other devices. The device containing the communication components can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0184] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0185] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0186] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0187] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0188] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0189] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0190] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0191] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0192] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0193] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0194] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0195] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A bitrate control method, characterized in that, include: Content feature information and encoding feature information are extracted from the video segment to be encoded. The content feature information reflects the content complexity of the video segment, and the encoding feature information reflects the encoding complexity of the video segment. Bitrate prediction is performed based on the content feature information and the encoding feature information to obtain the target bitrate for encoding the video segment; Based on the content feature information and the target bitrate, a quality factor prediction is performed to obtain the target quality factor used for encoding the video segment; Using the target bitrate as a bitrate constraint, and based on the target quality factor, the encoding process of the video segment is subjected to bitrate control with the goal of maintaining constant visual quality.

2. The method according to claim 1, characterized in that, Bitrate prediction is performed based on the content feature information and the encoding feature information to obtain the target bitrate for encoding the video segment, including: The content feature information and the encoding feature information are concatenated to obtain the first concatenated feature; The first splicing feature is input into the bitrate prediction network for bitrate prediction to obtain the target bitrate; The bitrate prediction network is trained with the goal of maintaining constant visual quality.

3. The method according to claim 2, characterized in that, The bitrate prediction network includes a bitrate feature extraction layer and a bitrate prediction layer; inputting the first concatenated feature into the bitrate prediction network for bitrate prediction to obtain the target bitrate includes: The first concatenation feature is input into the bitrate feature extraction layer for feature extraction to obtain the first target feature; The first target feature is input into the bitrate prediction layer to perform regression analysis on the correlation between the feature and the bitrate, so as to obtain the target bitrate; In the case where there are multiple bitrate feature extraction layers, the input of the subsequent bitrate feature extraction layer is the output of the previous bitrate feature extraction layer, which is used to extract higher-level features based on the previous bitrate feature extraction layer.

4. The method according to claim 3, characterized in that, The bitrate prediction network is implemented as a multilayer perceptron (MLP) network. The bitrate feature extraction layer and the bitrate prediction layer respectively include a fully connected layer and a normalization layer in the MLP network, and the number of fully connected layers is negatively correlated with the output dimension of the fully connected layer.

5. The method according to claim 2, characterized in that, Based on the content feature information and the target bitrate, a quality factor prediction is performed to obtain the target quality factor used for encoding the video segment, including: The content feature information and the target bitrate are concatenated to obtain the second concatenation feature; The second spliced ​​feature is input into the quality factor prediction network to predict the quality factor, so as to obtain the target quality factor; The quality factor prediction network is trained with the goal of maintaining constant visual quality.

6. The method according to claim 5, characterized in that, The quality factor prediction network includes a quality feature extraction layer and a quality factor prediction layer; the second concatenated feature is input into the quality factor prediction network to predict the quality factor, thereby obtaining the target quality factor, including: The second spliced ​​feature is input into the quality feature extraction layer for feature extraction to obtain the second target feature; The second target feature is input into the quality factor prediction layer to perform regression analysis on the correlation between the feature and the quality factor, so as to obtain the target quality factor; In the case where there are multiple quality feature extraction layers, the input of the subsequent quality feature extraction layer is the output of the previous quality feature extraction layer, which is used to extract higher-level features based on the previous quality feature extraction layer.

7. The method according to claim 6, characterized in that, The quality factor prediction network is implemented as a multilayer perceptron (MLP) network. The quality feature extraction layer and the quality factor prediction layer respectively include fully connected layers and normalization layers in the MLP network, and the number of fully connected layers is negatively correlated with the output dimension of the fully connected layers.

8. The method according to any one of claims 5-7, characterized in that, Also includes: Acquire multiple sample videos; Based on the specified visual quality, a base bitrate is selected from the different bitrates used to encode the plurality of sample videos according to the variable bitrate control algorithm. Based on the specified visual quality, a baseline quality factor is selected from different quality factors used to encode the plurality of sample videos according to a quality constant algorithm; The model is trained based on the multiple sample videos, the baseline bitrate, and the baseline quality factor to obtain the bitrate prediction network and the quality factor prediction network.

9. The method according to claim 8, characterized in that, Based on the specified visual quality, a base bitrate is selected from different bitrates used for encoding the plurality of sample videos according to a variable bitrate control algorithm, including: The multiple sample videos are encoded using different bitrates according to a variable bitrate control algorithm to obtain multiple first sample encoded bitstreams; Based on the decoding results of the plurality of first sample encoded bitstreams, a bitrate whose decoding results satisfy the specified visual quality requirements is selected from the different bitrates and used as the reference bitrate; Based on the specified visual quality, a baseline quality factor is selected from different quality factors used in encoding the plurality of sample videos according to a quality-constant algorithm, including: The multiple sample videos are encoded using different quality factors according to a constant quality algorithm to obtain multiple second sample encoded bitstreams; Based on the decoding results of the plurality of second sample encoded bitstreams, a quality factor whose decoding result satisfies the specified visual quality requirement is selected from the different quality factors and used as the benchmark quality factor.

10. The method according to any one of claims 1-7, characterized in that, Using the target bitrate as a bitrate constraint and based on the target quality factor, the encoding process of the video segment is subjected to bitrate control with the goal of maintaining constant visual quality, including: Based on the target bitrate, generate the maximum bitrate allowed to be used in the bitrate control process with the goal of maintaining constant visual quality; Based on the target quality factor and the maximum bitrate, the encoding process of the video segment is subject to bitrate control with the goal of maintaining the specified constant visual quality.

11. The method according to claim 10, characterized in that, Based on the target bitrate, the maximum allowed bitrate during the bitrate control process, with the goal of maintaining constant specified visual quality, is generated, including: The target bitrate is directly used as the maximum bitrate.

12. The method according to any one of claims 1-7, characterized in that, Before performing bitrate control on the encoding process of the video segment with the target bitrate as a bitrate constraint and based on the target quality factor, aiming to maintain a constant visual quality, the method further includes: Determine the target code control algorithm applicable to the video segment; When the target bitrate control algorithm is a first quality constant bitrate control algorithm, the bitrate control operation is performed on the encoding process of the video segment with the target bitrate as a bitrate constraint and the target quality factor as the target. The first quality constant bitrate control algorithm is a quality constant bitrate control algorithm with the predicted target bitrate as the bitrate constraint.

13. The method according to claim 12, characterized in that, Also includes: When the target code control algorithm is the second quality constant code control algorithm, the encoding process of the video segment is subject to code rate control with the specified visual quality constant as the target, based on the target quality factor. The second quality constant code control algorithm is a quality constant code control algorithm without code rate constraints. When the target bitrate control algorithm is a variable bitrate control algorithm, the encoding process of the video segment is subject to variable bitrate control based on the target bitrate.

14. A data storage method, characterized in that, include: Receive video data to be stored, the video data including at least one video segment; The at least one video segment is encoded, and the bitrate control method according to any one of claims 1-13 is used for bitrate control during the encoding process to obtain at least one encoded video segment; Store the at least one encoded video segment.

15. A video processing method, characterized in that, include: Receive video data sent from a data source, wherein the video data includes at least one video segment; The at least one video segment is encoded, and the bitrate control method according to any one of claims 1-13 is used for bitrate control during the encoding process to obtain at least one encoded video segment; Receive a data acquisition request sent by the data requesting end, wherein the data acquisition request includes the identification information of the requested video segment; The video segment corresponding to the identification information in the at least one encoded video segment is sent to the data requesting end.

16. An electronic device, characterized in that, include: A memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions for: performing the steps of the method according to any one of claims 1-15.

17. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the steps of the method according to any one of claims 1-15.

18. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, performs the steps of the method described in any one of claims 1-15.

Images