Encoding methods, devices, encoders, media and products

Abstract

This application discloses an encoding method, apparatus, encoder, medium, and product, belonging to the field of video encoding and decoding technology. The encoding method includes: acquiring multiple candidate encoding parameters, which indicate the encoding mode for the current frame; acquiring the encoding quality corresponding to each of the multiple candidate encoding parameters through a subjective quality prediction model, whereby the subjective quality prediction model is a machine learning model trained on sample image frames, the sample image frames being labeled with reference encoding quality, the reference encoding quality being subjectively labeled, and the reference encoding quality being used to express the data consistency before and after encoding of the sample image frames; acquiring target encoding parameters from the multiple candidate encoding parameters whose encoding quality meets the quality requirements; and outputting the bitstream data encoded for the current frame under the target encoding parameters. The subjective quality prediction model improves the efficiency of acquiring an encoding mode suitable for the current frame, and improves the accuracy and stability of the encoding process.

Description

Technical Field

[0001] This application relates to the field of video encoding and decoding technology, and in particular to an encoding method, apparatus, encoder, medium and product. Background Technology

[0002] During the video encoding process, the encoder needs to obtain encoding parameters for encoding the units to be encoded.

[0003] In related technologies, the encoder obtains the target bitrate requirement of the unit to be encoded through a fixed algorithm, and then obtains the encoding parameters of the unit to be encoded based on the target bitrate requirement. Specifically, when the texture complexity of the unit to be encoded is high, the fixed algorithm assigns a higher target bitrate requirement to the unit.

[0004] However, the aforementioned fixed algorithm has low accuracy in making decisions about encoding parameters, resulting in low encoding efficiency. Summary of the Invention

[0005] This application provides an encoding method, apparatus, encoder, medium, and product. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide an encoding method, the method comprising:

[0007] Obtain multiple candidate encoding parameters, which are used to indicate the encoding method for the current frame;

[0008] The coding quality corresponding to the multiple candidate coding parameters is obtained by a subjective quality prediction model. The subjective quality prediction model is a machine learning model trained on sample image frames. The sample image frames are labeled with reference coding quality. The reference coding quality is labeled from a subjective perspective and is used to express the data consistency of the sample image frames before and after coding.

[0009] Obtain the target coding parameter whose coding quality meets the quality requirements from the plurality of candidate coding parameters;

[0010] Output the bitstream data encoded for the current frame under the target encoding parameters.

[0011] On the other hand, embodiments of this application provide an encoding device, the device comprising:

[0012] The acquisition module is configured to acquire multiple candidate encoding parameters, which are used to indicate the encoding method for the current frame;

[0013] The acquisition module is further configured to acquire the coding quality corresponding to the plurality of candidate coding parameters respectively through a subjective quality prediction model. The subjective quality prediction model is a machine learning model trained on sample image frames. The sample image frames are labeled with reference coding quality. The reference coding quality is labeled from a subjective perspective. The reference coding quality is used to express the data consistency of the sample image frames before and after coding.

[0014] The acquisition module is further configured to acquire the target encoding parameter whose encoding quality meets the quality requirements from the plurality of candidate encoding parameters;

[0015] The output module is configured to output the bitstream data encoded for the current frame under the target encoding parameters.

[0016] On the other hand, embodiments of this application provide an encoder, the encoder including a memory and a processor, the memory for storing a computer program running on the processor; the processor for executing the steps of the encoding method as described above when running the computer program.

[0017] On the other hand, embodiments of this application provide a computer device, which includes a processor and a memory. The memory stores a computer program, which is loaded and executed by the processor to implement the steps of the above-described encoding method.

[0018] On the other hand, embodiments of this application provide a computer-readable storage medium storing a bitstream thereon, and at least one of a computer program and instructions, wherein when the at least one of the computer program and instructions is executed by a processor, it implements the steps of the encoding method as described above to generate the bitstream.

[0019] On the other hand, embodiments of this application provide a method for storing a bitstream, comprising generating a bitstream by performing the steps of the encoding method as described above; and storing the bitstream.

[0020] On the other hand, embodiments of this application provide a method for transmitting a bitstream, comprising performing the steps of the encoding method as described above to generate a bitstream; and transmitting the bitstream.

[0021] On the other hand, embodiments of this application provide a computer program product including computer instructions that are executed by a processor to implement the steps of the encoding method as described above.

[0022] In this embodiment, a subjective quality prediction model is used to obtain the coding quality corresponding to multiple candidate coding parameters for the current frame. This means that the data consistency between the original data and the reconstructed data before and after coding the current frame is evaluated. The coding quality is used to intuitively determine the target coding parameters that meet the quality requirements, thereby improving the efficiency of obtaining a coding mode that is compatible with the current frame and improving the compression efficiency of the coding process while ensuring coding quality. Furthermore, the evaluation criteria corresponding to the subjective quality prediction model are used to reduce coding decision errors. The current frame is coded based on the coding mode indicated by the target coding parameters, thereby improving the accuracy and stability of the coding process. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the encoding process at the encoding end in an exemplary embodiment of this application;

[0024] Figure 2 This is a schematic diagram illustrating the decoding process at the decoding end in an exemplary embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an encoding system provided in an exemplary embodiment of this application;

[0026] Figure 4 This is a flowchart of an exemplary embodiment of the encoding method provided in this application;

[0027] Figure 5 This is a flowchart of an encoding method provided by another exemplary embodiment of this application;

[0028] Figure 6 This is a schematic diagram of a process for obtaining texture distortion index provided in an exemplary embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the process for obtaining frame difference metrics provided in an exemplary embodiment of this application;

[0030] Figure 8 This is a flowchart illustrating the encoding method provided in an exemplary embodiment of this application;

[0031] Figure 9 This is a flowchart of an encoding method provided in yet another exemplary embodiment of this application;

[0032] Figure 10 This is a schematic diagram of the process for training a subjective quality prediction model provided in an exemplary embodiment of this application;

[0033] Figure 11 This is a structural block diagram of an encoding apparatus provided in an exemplary embodiment of this application;

[0034] Figure 12This is a structural block diagram of an encoding device provided in another exemplary embodiment of this application;

[0035] Figure 13 This is a structural block diagram of an encoder provided in an exemplary embodiment of this application;

[0036] Figure 14 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0038] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0039] Currently, most common video codec standards employ a block-based hybrid coding framework. Images in a video are divided into squares of equal size (e.g., 256×256, 128×128, 64×64, etc.) called Largest Coding Units (LCUs) or Coding Tree Units (CTUs). These LCUs or CTUs can be further subdivided into rectangular Coding Units (CUs) according to rules. Coding units may also be further divided into Prediction Units (PUs), Transform Units (TUs), etc.

[0040] The hybrid coding framework includes modules such as prediction, transformation, quantization, entropy coding, and in-loop filtering. The prediction module includes intra-prediction and inter-prediction. Inter-prediction includes motion estimation and motion compensation. Because there is a strong correlation between adjacent samples in the same image, intra-prediction can be used in video coding and decoding techniques to eliminate spatial redundancy between adjacent samples. Because there is a strong similarity between adjacent images in a video, inter-prediction can be used in video coding and decoding techniques to eliminate temporal redundancy between adjacent images, thereby improving coding efficiency.

[0041] At the encoding end, such as Figure 1 As shown, the encoder first divides the image 110 into multiple coding blocks. Then, it uses either intra-frame prediction algorithm 120 or inter-frame prediction algorithm 130 (including motion estimation and motion compensation) to generate a prediction block for the current block. The encoder then subtracts the prediction block from the original block to obtain a residual block. This residual block is then transformed and quantized to obtain quantization coefficients. Finally, entropy coding 140 encodes the quantization coefficients into the bitstream. Furthermore, the encoder reconstructs the image based on the inverse quantization / inverse transform results and the prediction block to obtain a reconstructed block. This reconstructed block is then subjected to loop filtering 150 to compensate for distortion information and provide a better reference for subsequent encoding.

[0042] For example, the loop filter 150 includes different types of loop filter tools applied sequentially in the decoding loop in a specific order, including but not limited to Luma Mapping with Chroma Scaling (LMCS), De-blocking Filter (DBF), Sample Adaptive Offset (SAO), Adaptive Loop Filter (ALF), and Cross-Component Adaptive Loop Filter (CCALF).

[0043] On the decoding end, such as Figure 2 As shown, the decoder uses either intra-frame prediction algorithm 120 or inter-frame prediction algorithm 130 to predict the prediction block of the current block. Simultaneously, it parses the bitstream and obtains quantization coefficients through entropy decoding 210. Then, it performs inverse quantization / inverse transform 220 on the quantization coefficients to obtain the residual block. The prediction block and the residual block are then added together to obtain the reconstructed block. For the reconstructed block, the decoder further performs loop filtering 150, and finally generates the decoded image 230 based on the reconstructed block after loop filtering 150.

[0044] During the video encoding process, the encoder needs to determine the encoding parameters through bitrate control. Bitrate control is the process of adjusting the encoding parameters based on the target bitrate requirement, which is used to improve compression efficiency while ensuring video quality.

[0045] In related technologies, because the image content information (such as texture details and edge contour information) is denser in the encoding units with high image complexity, the loss of such image content information is greater after compression encoding. That is, the encoding units with high image complexity are more sensitive to the distortion state after encoding. From the perspective of bitrate control, the encoding method corresponding to the encoding unit is decisively affected by the image complexity. The encoding unit includes the current frame or the current block.

[0046] To illustrate, a unit to be encoded with high image complexity generates more data while improving the accuracy of the encoding process. Therefore, a higher target bit rate requirement is assigned to the unit to be encoded with high image complexity; correspondingly, a lower target bit rate requirement is assigned to the unit to be encoded with low image complexity.

[0047] Specifically, for the current frame, the target bitrate requirement allocated to the current frame is determined based on the objective image quality requirements and the frame type of the current frame; for the current block in the current frame, the target bitrate requirement allocated to the current block is determined based on the image complexity of the current block. Optionally, the objective image quality requirements include, but are not limited to: a Peak Signal-to-Noise Ratio (PSNR) threshold, used to express the limitation on the Mean Squared Error (MSE) between the original image and the reconstructed image; or, a Structural Similarity Index (SSIM) threshold, where SSIM is used to indicate the similarity between the original image and the reconstructed image at the brightness, contrast, and structural levels, respectively, and the SSIM threshold is used to express the limitation on the above similarity.

[0048] However, the above method has high target bitrate requirements for regions with complex textures. The encoder tends to assign higher target bitrate requirements to units with higher texture complexity. Furthermore, the encoder predicts the unit to be encoded based on the texture complexity of already encoded units, and then obtains the target bitrate requirement for the unit based on the prediction result, thus obtaining the encoding parameters for the unit based on the target bitrate requirement. When there is a large difference in texture complexity between the already encoded units and the unit to be encoded, the error between the target bitrate requirement and the actual target bitrate requirement of the unit to be encoded is large. Consequently, the decision accuracy for encoding parameters is low, resulting in low encoding efficiency.

[0049] To illustrate, when the texture complexity of the encoded unit is low and the texture complexity of the unit to be encoded is high, the target bit rate requirement allocated to the unit to be encoded is low, resulting in low image quality after encoding. Furthermore, due to the excessive residual energy corresponding to the unit to be encoded, the actual bit rate generated by the unit to be encoded exceeds the allocated target bit rate requirement, leading to bit rate overflow.

[0050] In this embodiment, by adjusting the image quality standard corresponding to the target bitrate requirement, the target bitrate requirement of the unit to be encoded, especially the region with complex texture, is adjusted so that the target bitrate requirement meets the evaluation requirements of the encoded image quality from the user's perspective, thereby reducing encoding decision error. The current frame is encoded based on the encoding mode indicated by the target encoding parameters, thereby improving the accuracy and stability of the encoding process.

[0051] Please refer to Figure 3 This illustration shows a schematic diagram of an encoding system provided in one embodiment of this application. The computer system 300 includes: a terminal 320, or a terminal 320 and a server 340.

[0052] The encoding method provided in this application embodiment can be executed independently by the terminal 320, independently by the server 340, or jointly by the terminal 320 and the server 340; no limitation is made here. In some embodiments, the computer system 300 can be implemented as the system architecture of the encoding method.

[0053] The device types of terminal 320 include at least one of the following: smart TV, smartphone, laptop, desktop computer, tablet computer, smart robot, augmented reality (AR) device, virtual reality (VR) device, in-vehicle terminal, wearable device, etc.

[0054] Terminal 320 is connected to server 340 via wireless or wired network.

[0055] Those skilled in the art will understand that the number of the aforementioned devices can be more or less. For example, there may be only one device, or there may be dozens or hundreds of devices, or even more. This application does not limit the number or type of devices.

[0056] Server 340 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 340 provides functional services for implementing model training. Optionally, server 340 undertakes the main computing work, and terminal 320 undertakes the secondary computing work; or, server 340 undertakes the secondary computing work, and terminal 320 undertakes the main computing work; or, server 340 and terminal 320 collaborate on computing using a distributed computing architecture.

[0057] In one example, the encoding method is implemented collaboratively by terminal 320 and server 340. Illustratively, a pre-trained subjective quality prediction model is deployed on server 340. This model is a machine learning model trained on sample image frames, each labeled with a reference encoding quality. This reference encoding quality expresses the data consistency before and after encoding the sample image frame. Terminal 320 obtains multiple candidate encoding parameters; server 340 then uses the subjective quality prediction model to obtain the encoding quality corresponding to each candidate encoding parameter.

[0058] Optionally, server 340 inputs the quality index data into the subjective quality prediction model and outputs the coding quality corresponding to each of the multiple candidate coding parameters. Illustratively, the methods for obtaining the quality index data corresponding to the multiple candidate coding parameters include: terminal 320 sending the multiple candidate coding parameters to server 340, obtaining the quality index data corresponding to each candidate coding parameter through the coding processing model deployed on server 340, whereby the coding processing model expresses the correspondence between the candidate coding parameters and the quality index data; or, terminal 320 performing coding processing on the current frame based on the multiple candidate coding parameters, thereby obtaining the quality index data based on the reconstructed data, and sending the quality index data to server 340.

[0059] In the example above, server 340 obtains target encoding parameters that meet the quality requirements from multiple candidate encoding parameters, and sends the target encoding parameters to terminal 320, which then outputs the bitstream data encoded for the current frame under the target encoding parameters.

[0060] It is worth noting that the aforementioned terminal 320 refers to an electronic device capable of outputting and displaying video data; optionally, terminal 320 is an encoder. The aforementioned server 340 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud security, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0061] Cloud technology refers to a managed technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0062] In some embodiments, the server 340 described above can also be implemented as a node in a blockchain system.

[0063] Please refer to Figure 4 This illustrates a flowchart of an encoding method provided in an exemplary embodiment of this application. The method is applied to an encoder, or the method is implemented by a computer device (which may be configured as follows). Figure 3 The method may be executed by the terminal 320 or server 340 shown, or it may be executed jointly by the terminal and the server interacting. The method may include at least one of the following steps.

[0064] Step 410: Obtain multiple candidate encoding parameters.

[0065] Candidate coding parameters are used to indicate the coding method for the current frame. Optionally, candidate coding parameters include, but are not limited to, parameters of the type such as quantization parameters (QP), coding unit partitioning method, prediction mode parameters, and transform coefficients; illustratively, when the candidate coding parameter is a QP, the candidate coding parameter is used to indicate the method of performing quantization processing on the current frame.

[0066] The encoder provides multiple candidate encoding methods for the current frame or the current block within the current frame by acquiring multiple candidate encoding parameters. Illustratively, it provides candidate encoding parameters with similar values ​​for different units to be encoded (such as the current frame or the current block), thereby ensuring encoding accuracy and smooth image quality after encoding. Here, candidate encoding parameters with similar values ​​refer to assigning multiple candidate encoding parameters with similar values ​​to the current frame based on a preset candidate encoding parameter benchmark (such as the initial QP), with these multiple candidate encoding parameters distributed around the benchmark.

[0067] Encoding parameters are the necessary instruction set required to perform encoding operations on the current frame. On the one hand, encoding parameters are used to indicate processing details such as the precision and order of encoding processing; on the other hand, encoding parameters are transmitted as part of the output bitstream data of the encoding process. Furthermore, depending on their scope, encoding parameters include sequence / video level parameters, such as video resolution, frame rate, entropy coding mode, and global configurations used to define the encoding environment; image frame level parameters, such as frame type and initial QP; and block level parameters, such as prediction mode parameters, block QP increment, and motion vectors. The encoder uses these different levels of encoding parameters to achieve accurate control over the encoding process during encoding.

[0068] Taking QP as an example of candidate coding parameters, the methods for obtaining multiple candidate coding parameters include directly determining them using a rate control algorithm. The process of obtaining multiple candidate coding parameters through a rate control algorithm includes, but is not limited to, at least one of the following steps.

[0069] 1. Obtain the overall bitrate requirement for the video to be encoded; and obtain the image complexity corresponding to the unit to be encoded. Optionally, the method for obtaining the overall bitrate requirement includes, but is not limited to, selection by the user, or unique determination by the application scenario corresponding to the video; for example, the overall bitrate requirement for a video call is no more than 1 megabits per second (Mbps).

[0070] 2. Based on the overall bitrate requirement and the image complexity corresponding to the unit to be encoded, a target number of bits is allocated to the unit to be encoded. The target number of bits is a threshold for the amount of data generated by the unit to be encoded during the encoding process, used to indicate the target bitrate requirement of the unit to be encoded. Multiple target number of bits are allocated to the unit to be encoded, each of which satisfies the overall bitrate requirement and is adapted to the image complexity corresponding to the unit to be encoded.

[0071] To illustrate, for a unit to be encoded with high image complexity, in order to ensure the quality of the encoded image, the accuracy of the encoding process needs to be improved compared to a unit to be encoded with low image complexity. When the accuracy of the encoding process is improved, more data is generated during the encoding process. Therefore, a larger number of target bits is allocated to the unit to be encoded with high image complexity. Correspondingly, a smaller number of target bits is allocated to the unit to be encoded with low image complexity.

[0072] 3. Determine the candidate coding parameters corresponding to the unit to be encoded based on the target number of bits. Through the mapping relationship between the number of bits and the candidate coding parameters, the target number of bits is mapped to the specific coding parameters corresponding to the unit to be encoded. Specifically, the unit to be encoded is allocated multiple different target number of bits, each used to indicate multiple different candidate coding methods, thereby obtaining multiple candidate coding parameters.

[0073] The encoder's decision-making process for obtaining multiple candidate coding parameters through a bitrate control algorithm is performed before or during the coding process. Illustratively, steps 1 to 3 above involve obtaining multiple candidate coding parameters before the coding process. Obtaining multiple candidate coding parameters during the coding process includes, but is not limited to: performing coding processing such as transformation and quantization on the original image corresponding to the coding unit based on the candidate coding parameters to obtain the bitstream and reconstructed image; dynamically adjusting the candidate coding parameters of subsequent coding units based on the bitstream size and the error data between the original image and the reconstructed image, for example, by repeating the above steps to meet the overall bitrate requirements.

[0074] In some embodiments, the multiple candidate coding parameters corresponding to the current frame are related to the application scenario of the video to be encoded. Illustratively, for network video encoding, a preset benchmark of candidate coding parameters is obtained, and an optimization search algorithm is used to obtain candidate coding parameters with lower rate-distortion costs. Here, rate-distortion cost refers to the evaluation coefficient of the candidate coding scheme jointly determined by video distortion and bitrate. Correspondingly, the lower the rate-distortion cost, the more optimal the candidate coding scheme corresponding to the candidate coding parameter achieves a balance between video distortion and bitrate. Video distortion is used to measure the quality of the encoded video, and bitrate is used to indicate the bandwidth resources required for transmitting or storing the video. For engineering video encoding, candidate coding parameters are obtained based on prior knowledge. Prior knowledge refers to pre-determined preset information and empirical rules regarding the video content or coding environment, including but not limited to at least one of the following.

[0075] 1.1 Content type knowledge, for example, most areas in a surveillance video are static backgrounds, so during the encoding process, the encoder tends to assign larger candidate encoding parameters to static background areas.

[0076] 1.2 Manually labeled Regions of Interest (ROIs) are used to assign smaller candidate coding parameters to visually sensitive areas such as face regions, text regions, and key facilities of engineering interest during the coding process to ensure the quality of the encoded image.

[0077] 1.3 Encoder configuration experience, such as setting different candidate coding parameter baselines and fluctuation ranges for different application scenarios or video types. For example, different candidate coding parameters are assigned to different application scenarios such as industrial inspection, medical imaging, security monitoring, aerospace, scientific observation, and machine vision.

[0078] It is worth noting that the above-described methods for obtaining target bitrate requirements or candidate coding parameters are illustrative examples, and the embodiments of this application do not limit the methods and processes for obtaining multiple candidate coding parameters.

[0079] Step 420: Obtain the coding quality corresponding to multiple candidate coding parameters through the subjective quality prediction model.

[0080] The subjective quality prediction model is a machine learning model trained on sample image frames. These sample image frames are labeled with reference coding quality, which is determined from a subjective perspective. The reference coding quality expresses the data consistency of the sample image frame before and after encoding. The subjective perspective uses the user's perceived quality as the target for the final coding quality evaluation. The subjective quality prediction model uses the reference coding quality labeled from a subjective perspective to simulate the user's subjective visual experience, evaluate the data consistency of the image frames before and after encoding, and obtain the coding quality corresponding to multiple candidate coding parameters.

[0081] The reference coding quality annotation, from a subjective perspective, includes, but is not limited to, at least one of the following steps: 1. Manually identifying the reference coding quality of sample image frames before and after coding. 2. Performing preprocessing on the reference coding quality, such as outlier removal, data cleaning and standardization, and noise reduction, to obtain a coding quality assessment result with group interpretability, thereby improving the objectivity of the reference coding quality indicators at the analytical level. 3. Annotating the reference coding quality onto the sample image frames.

[0082] It is worth noting that the subjective quality prediction model is used to simulate the user's ability to evaluate the consistency of data before and after encoding in the current frame. The reference coding quality, annotated from a subjective perspective, represents the human eye's visual experience of image quality. The human eye's perception of the consistency of image data before and after encoding focuses on overall integrity and overall reproduction. It has high visual sensitivity to brightness, contrast, and object contours in the image data, but weaker perception of texture loss in areas with high texture complexity before and after encoding. The subjective quality prediction model is a machine learning model trained on the reference coding quality annotated from a subjective perspective on sample image frames. In other words, the subjective quality prediction model simulates the human eye's visual experience of image quality, using subjective evaluation capabilities to assess the consistency of data before and after encoding in the current frame. This yields the coding quality corresponding to multiple candidate coding parameters, thereby guiding coding decisions and improving coding efficiency.

[0083] The reference coding quality is pre-annotated from a subjective perspective. This is achieved by selecting experimenters as the data source for the reference coding quality, evaluating the consistency of sample image frames before and after coding based on their subjective opinions, and integrating this data using statistical methods to obtain the reference coding quality. This ensures that the reference coding quality more realistically reflects the stable subjective tendencies of the experimenters and represents the statistical regularity of the group. The data format of the reference coding quality is consistent with the data format of the coding quality obtained by the subjective quality prediction model. Illustratively, the annotation methods for sample image frames include, but are not limited to, at least one of the following methods.

[0084] 1. Obtain quantitative values ​​of the visual experience of data consistency before and after encoding of sample image frames through subjective questionnaires; schematically, for different data consistency dimensions, such as color consistency and texture consistency, the experimenters are required to select the corresponding indicator value within a preset range, such as the integer range of [-5, 5], as the evaluation result; the evaluation results are integrated and processed, such as taking the average or median, to obtain the reference encoding quality.

[0085] 2. Select professionals with expertise in human vision as experimental personnel, and obtain reference coding quality based on their professional knowledge and subjective experience. Optionally, the reference coding quality includes, but is not limited to, evaluation scores within a preset range, classification results in at least two evaluation levels, multi-dimensional evaluation sets (such as evaluation results for dimensions such as color, geometric structure, brightness, contrast, and texture), and relative coefficients.

[0086] A subjective quality prediction model is a pre-trained machine learning model stored in the encoder or computer device that performs the encoding method. In some embodiments, the subjective quality prediction model is used to indicate the correspondence between candidate encoding parameters and encoding quality.

[0087] The sample image frame includes the image data corresponding to the sample image frame before and after encoding, such as the reference original data and reference reconstructed data corresponding to the sample image frame. The process of obtaining the reference reconstructed data includes, but is not limited to, performing prediction processing on the reference original data under given reference encoding parameters to obtain reference prediction data; transforming and quantizing the difference between the reference original data and the reference prediction data, i.e., the reference residual data; performing inverse quantization and inverse transformation on the processed reference residual data, and adding it to the reference prediction data to obtain the reference reconstructed data.

[0088] The reference coding quality of the sample image frame annotation is used to indicate the difference between the aforementioned original reference data and the reconstructed reference data; that is, it indicates the degree to which the reconstructed reference data restores the original reference data, and in other words, it indicates the data consistency before and after the sample image frame is encoded. The reference coding quality is an evaluation result obtained from the user's perspective regarding the aforementioned differences.

[0089] In some embodiments, the implementation of reference coding quality includes, but is not limited to, at least one of the following: 1. An evaluation score within a preset range, such as a reference coding quality score of 90 out of 100; 2. Classification results in at least two evaluation levels, such as at least two evaluation levels including "extremely distorted", "relatively distorted", "relatively faithful", and "extremely faithful"; 3. A multi-dimensional evaluation set, such as, from a subjective perspective, the data consistency of sample image frames before and after coding includes at least one dimension such as color, geometric structure, brightness, contrast, and texture, and the multi-dimensional evaluation set includes the evaluation results of the above at least two dimensions to achieve a comprehensive evaluation of the reference coding quality; 4. A relative coefficient, such as the corresponding sequence number of each of the multiple sample image frames after sorting the data consistency before and after coding as an indicator value of the reference coding quality.

[0090] Optionally, the process of obtaining the subjective quality prediction model includes, but is not limited to, obtaining an initial machine learning model, which is used to indicate the algorithm architecture and loss function in the training process corresponding to the subjective quality prediction model; predicting the predicted coding quality corresponding to multiple reference coding parameters based on the machine learning model; and adjusting the model parameters of the machine learning model based on the loss value between the predicted coding quality and the reference coding quality until the loss value converges to obtain the subjective quality prediction model, wherein the loss value is used to indicate the difference between the predicted coding quality and the reference coding quality.

[0091] In some embodiments, obtaining the coding quality corresponding to multiple candidate coding parameters through a subjective quality prediction model includes, but is not limited to, inputting multiple candidate coding parameters into the subjective quality prediction model and outputting the coding quality corresponding to multiple candidate coding parameters; or, obtaining quality index data corresponding to multiple candidate coding parameters, with the quality index data serving as an intermediate indicator value to characterize the data difference of the current frame before and after coding; inputting the quality index data into the subjective quality prediction model and outputting the coding quality corresponding to multiple candidate coding parameters.

[0092] Step 430: Obtain the target coding parameter whose coding quality meets the quality requirements from multiple candidate coding parameters.

[0093] By considering quality requirements, the final target encoding parameters for encoding the current frame are selected from multiple candidate encoding parameters, thereby enabling the output of the bitstream data corresponding to the current frame under the target encoding parameters. Quality requirements are constraints on encoding quality derived by simultaneously considering the encoded video quality and the bandwidth resources required for transmitting or storing the video.

[0094] To illustrate, the encoding quality meeting the quality requirements includes, but is not limited to, at least one of the following situations.

[0095] 1. When encoding quality is expressed as an evaluation score, the encoding quality must reach a preset score threshold, or the encoding quality must fall within a preset score range. For example, if the quality requirement is a preset score threshold of 90, encoding quality ≥ 90 indicates that the encoding quality meets the quality requirement; if the quality requirement is a preset score range [80, 90], encoding quality ≤ 90 indicates that the encoding quality meets the quality requirement.

[0096] 2. When the coding quality is achieved as a classification result in at least two evaluation levels, the coding quality is within a preset quality set. For example, if the at least two evaluation levels include "extremely distorted", "relatively distorted", "relatively faithful", and "extremely faithful", and the quality requirement is a preset quality set {"relatively faithful", "extremely faithful"}, then the coding quality is achieved as the classification result corresponding to "relatively faithful" or "extremely faithful", respectively, indicating that the coding quality meets the quality requirements.

[0097] 3. When the coding quality achieves a relative coefficient, the coding quality reaches a preset coefficient threshold. For example, when the coding quality is achieved by sorting the data consistency before and after encoding multiple sample image frames, the coefficient threshold is a positive integer K. A coding quality ≤ K indicates that the coding quality is among the top K in the sequence and meets the quality requirements. Optionally, when K=1, the first candidate coding parameter is obtained in descending order of coding quality and used as the target coding parameter that meets the quality requirements.

[0098] Optionally, when the coding quality is implemented as a multi-dimensional evaluation set, the indication results of each dimension of the coding quality need to meet the corresponding quality requirements for each dimension. For example, the color dimension of coding quality is indicated by a percentage-based evaluation score, and the geometric structure is indicated by an evaluation result including two evaluation levels: "distortion" and "fidelity". Accordingly, for the color dimension, the quality requirement includes a color evaluation score of 90, and for the geometric structure dimension, the quality requirement includes a geometric structure evaluation result of "fidelity". If the quality requirements of the above two dimensions are met respectively, it means that the coding quality meets the quality requirements.

[0099] It is worth noting that the above-mentioned correspondence between the implementation methods of encoding quality and quality requirements, as well as the implementation methods of quality requirements, are all illustrative examples. The specific implementation methods of obtaining target encoding parameters according to quality requirements in the embodiments of this application are not limited. For example, when the encoding quality is implemented as a score, the encoding quality can be determined to meet the encoding requirements by selecting the top K encoding quality in the sort.

[0100] By obtaining target encoding parameters that meet the quality requirements based on the subjective user perspective, the encoding processing method for the current frame is dynamically adjusted. Optionally, the quality requirements are determined by the application scenario corresponding to the video to be encoded, or are obtained by user-defined settings. By setting the quality requirements, the video playback method of different bitrates or quality versions can be adapted to meet the needs of different network conditions and terminal devices. Furthermore, it can avoid encoding problems such as sudden drop in encoding quality, bitrate overflow, or bitrate underflow when the video content changes drastically, thereby allocating compression bits more reasonably, improving resource utilization, and enhancing the stability of video transmission or storage.

[0101] To illustrate, when the video is used in a live streaming scenario, the quality requirements are lower, and the encoding process focuses on reducing transmission latency and ensuring a stable bitrate. When the video is used in a video-on-demand scenario or as a video archive, the quality requirements are higher, and the encoding process focuses on ensuring stable video quality and maximizing the user's visual experience while maintaining storage efficiency.

[0102] In some embodiments, obtaining the target coding parameter from multiple candidate coding parameters by coding quality includes: obtaining at least one undetermined coding parameter from multiple candidate coding parameters, wherein the coding quality corresponding to the undetermined coding parameter meets the quality requirements; obtaining the bitstream size corresponding to each of the at least one undetermined coding parameters, wherein the bitstream size is the amount of data in the bitstream data encoded for the current frame under the undetermined coding parameter; arranging the at least one undetermined coding parameter in ascending order of bitstream size to obtain the first undetermined coding parameter, which is used as the target coding parameter.

[0103] Optionally, the above methods for obtaining the bitstream size corresponding to at least one undetermined encoding parameter include, but are not limited to: encoding the current frame based on at least one undetermined encoding parameter to obtain the bitstream of the current frame corresponding to at least one undetermined encoding parameter, thereby obtaining the bitstream size corresponding to at least one undetermined encoding parameter; or, obtaining the bitstream size corresponding to at least one undetermined encoding parameter through a pre-trained machine learning model; or, simulating the complete encoding process through rate-distortion cost calculation to obtain the bitstream size corresponding to at least one undetermined encoding parameter.

[0104] The target encoding parameter is the encoding parameter with the smallest bitstream size among at least one undetermined encoding parameter. In other words, the encoding quality corresponding to the target encoding parameter is the same as or different from the best encoding quality among the at least one undetermined encoding parameter. When the encoding quality corresponding to the target encoding parameter is not optimal, the encoding method indicated by the target encoding parameter focuses on more efficiently improving the visual quality of the current frame. Illustratively, when bandwidth or storage resources are limited during the encoding process, the priority is to ensure that the subjective quality of the encoded current frame meets the user's required quality requirements. Then, by using the encoding method indicated by the target encoding parameter, details that are not sensitive to the user's perspective, such as areas with high texture complexity, are discarded, while ensuring that key parts affecting the user's visual experience are preserved. This reduces encoding decision errors and improves the accuracy and stability of the encoding process.

[0105] By jointly considering the quality of the encoded video and the bandwidth resources required for transmitting or storing the video, the method avoids pursuing high quality or low bitrate alone during the acquisition of target encoding parameters, thereby improving resource utilization in the encoding process. Under the condition of meeting the user's subjective visual requirements for video quality, the method of selecting the encoding method with a smaller bitrate, that is, less bandwidth resources required, improves the accuracy of encoding decisions and encoding efficiency.

[0106] In some embodiments, the method of obtaining at least one undetermined coding parameter from multiple candidate codes based on quality requirements includes, but is not limited to, at least one of the following: 1. Candidate coding parameters whose coding quality reaches a preset first threshold are used as undetermined coding parameters; 2. Candidate coding parameters whose coding quality is within a preset quality set are used as undetermined coding parameters, wherein the quality set includes at least one coding quality that meets the quality requirements, or the quality set is a value range corresponding to the quality requirements; 3. The first K candidate coding parameters are obtained in descending order of coding quality as at least one undetermined coding parameter, where K is a positive integer.

[0107] Schematic, the way the above quality requirements are implemented is related to the way the coding quality is implemented; for example, when the coding quality is implemented as the classification result of "particularly distorted", "relatively distorted", "relatively true" and "particularly true" in at least two evaluation levels, the quality requirements are implemented as a preset quality set, which includes at least one coding quality of "relatively true" or "particularly true" that meets the quality requirements; or, when the coding quality is implemented as a percentage evaluation score, the quality requirements are implemented as a preset quality set, which is the value range [80, 90] corresponding to the quality requirements.

[0108] By expanding the implementation methods of quality requirements, the subjective quality requirements of video in diverse application scenarios can be met. Setting different quality requirements can obtain encoding parameters that meet the quality requirements based on user needs or different application scenarios corresponding to the video, thereby adjusting the focus of the encoding method corresponding to the current frame and improving the accuracy of encoding decisions.

[0109] Step 440: Output the bitstream data encoded for the current frame under the target encoding parameters.

[0110] The bitstream data serves as the sole input to the decoder, including decision information from the encoder during the encoding process. The decoder at the decoder parses and executes the syntax elements in the bitstream data to ultimately reconstruct the decoded image for display.

[0111] The process of obtaining the bitstream data encoded for the current frame under the target coding parameters includes: dividing the current frame into multiple coding blocks under the guidance of the coding method indicated by the target coding parameters; then generating a prediction block for the current block using intra-frame prediction or inter-frame prediction (including motion estimation and motion compensation) algorithms; subtracting the prediction block from the original block to obtain a residual block; transforming and quantizing the residual block to obtain quantization coefficients; and finally encoding the quantization coefficients and target coding parameters into the bitstream using entropy coding to obtain the bitstream data. Optionally, the bitstream data includes, but is not limited to, coding parameters such as the prediction mode, the motion vector of the current block, the division method of the coding unit, and filtering parameters.

[0112] In some embodiments, the output of the bitstream data encoded for the current frame under the target encoding parameters includes at least one of the following cases.

[0113] Scenario 1: Before determining the target encoding parameters for the current frame, the bitstream data encoded under the target encoding parameters has already been obtained through complete encoding processing. In this case, the bitstream data is output directly.

[0114] Schematic, in the process of obtaining the coding quality corresponding to multiple candidate coding parameters through a subjective quality prediction model, the current frame can be encoded based on each of the multiple candidate coding parameters to obtain the bitstream data corresponding to each of the multiple candidate coding parameters, including the bitstream data encoded for the current frame under the target coding parameters. Alternatively, in the process of obtaining the target coding parameter whose coding quality meets the quality requirements from multiple candidate coding parameters, at least one undetermined coding parameter can be obtained from the multiple candidate coding parameters based on the quality requirements; in an optional case, the current frame can be encoded based on at least one undetermined coding parameter to obtain the bitstream data of the current frame corresponding to each of the at least one undetermined coding parameters, including the bitstream data encoded for the current frame under the target coding parameters.

[0115] Scenario 2: When determining the target encoding parameters for the current frame, the current frame has not yet undergone complete encoding processing based on the target encoding parameters, meaning the bitstream data corresponding to the current frame has not been obtained. In this case, the current frame is encoded based on the target encoding parameters to obtain the bitstream data encoded under the target encoding parameters, and then output.

[0116] In summary, this embodiment uses a subjective quality prediction model to obtain the coding quality corresponding to multiple candidate coding parameters for the current frame. This provides an evaluation of the data consistency between the original data and the reconstructed data before and after coding the current frame. The coding quality directly determines the target coding parameters that meet the quality requirements, improving the efficiency of obtaining a coding mode suitable for the current frame and increasing the compression efficiency of the coding process while ensuring coding quality. Furthermore, the evaluation criteria corresponding to the subjective quality prediction model reduce coding decision errors, and the current frame is encoded based on the coding mode indicated by the target coding parameters, improving the accuracy and stability of the coding process.

[0117] In an optional embodiment, a subjective quality prediction model is used to express the correspondence between quality indicator data and coding quality, whereby the quality indicator data characterizes the difference between the current frame before and after encoding from an objective data perspective. Figure 5 This is a flowchart of an encoding method provided by another exemplary embodiment of this application, the method being applied to an encoder, or the method being implemented by a computer device (which may be configured as follows). Figure 3 The method is executed by the terminal 320 or server 340 shown, or it is executed jointly by the terminal and server interacting. Figure 5 As shown, step 420 above can also be implemented as steps 422 to 424 below.

[0118] Step 422: Obtain the quality index data corresponding to the multiple candidate coding parameters.

[0119] Quality index data is used to characterize the differences in data before and after encoding of the current frame. Quality index data is obtained from the original data and reconstructed data corresponding to the current frame and is used to objectively indicate the coding quality.

[0120] In some embodiments, the method of obtaining quality index data corresponding to multiple candidate coding parameters includes at least one of the following processes.

[0121] 1. Encode the current frame based on multiple candidate coding parameters to obtain the reconstructed data of the current frame corresponding to each candidate coding parameter; obtain quality index data based on the reconstructed data.

[0122] Optionally, the aforementioned quality indicator data includes at least one indicator data of different dimensions. By combining the indicator data of these different dimensions, the quality indicator data can comprehensively indicate the data differences of the current frame before and after encoding. The different dimensions of the indicator data correspond to different methods of obtaining data differences, that is, different focuses in measuring the image quality of the current frame before and after encoding; for example, the first indicator data might indicate data differences based on the difference in texture complexity before and after encoding the current frame, or the second indicator data might indicate data differences based on the pixel difference before and after encoding the current frame, etc.

[0123] In some embodiments, obtaining quality index data based on reconstruction data includes: for each candidate coding parameter, obtaining the reconstruction block corresponding to at least one coding block in the current frame; for each candidate coding parameter, obtaining the difference data between at least one coding block and the reconstruction block corresponding to the coding block, to obtain the difference data corresponding to at least one coding block; and obtaining quality index data based on the difference data corresponding to at least one coding block.

[0124] Optionally, the method for obtaining quality indicator data based on the difference data corresponding to at least one coding block includes: dividing the difference data corresponding to at least one coding block into at least one cluster using a clustering algorithm, such as K-means clustering, to ensure high similarity of difference data points within the same cluster; calculating the cluster center for each cluster using a clustering algorithm, which serves as the representative value of the difference data within that cluster. Specifically, the cluster center, i.e., the quality indicator data, is obtained by setting the number of clusters to 1; or, the quality indicator data is obtained by merging the cluster centers corresponding to at least one cluster according to specific rules, such as selecting the cluster center containing the most difference data points as the quality indicator data, or by weighted averaging the cluster centers corresponding to each cluster.

[0125] By obtaining quality index data corresponding to the current frame based on the difference data corresponding to at least one coding block in the current frame, and obtaining quality index data that can represent the current frame according to the local content characteristics of each coding block, the algorithm process for obtaining quality index data is simplified, the accuracy of representing the difference between the data before and after encoding of the current frame is improved, as well as the accuracy of representing the quality of the encoded image, thereby improving the accuracy of encoding decisions.

[0126] In some embodiments, obtaining quality index data corresponding to multiple candidate coding parameters includes at least one of the following:

[0127] The texture distortion index is obtained as the first dimension index data. The texture distortion index is used to characterize the accuracy loss of the current frame during the quantization process.

[0128] The texture complexity metric is obtained as the second dimension metric data. The texture complexity metric is used to characterize the loss of image detail before and after encoding of the current frame.

[0129] The frame difference index is obtained as the third dimension index data. The frame difference index is used to characterize the loss of pixel information before and after the current frame is encoded.

[0130] It is worth noting that the three dimensions of the quality indicator data mentioned above are merely illustrative examples. This application does not limit the specific dimensions of the indicator data included in the quality indicator data. The terms "first," "second," and "third" are only used to distinguish the indicator data of different dimensions in the quality indicator data. For example, without departing from the scope of this application, the first dimension indicator data can also be called the second dimension indicator data, and similarly, the second dimension indicator data can also be called the third dimension indicator data, etc. The quality indicator data includes indicator data of at least one dimension, indicating the differences in data before and after encoding from different dimensions. The encoding quality is obtained based on the quality indicator data through a subjective quality prediction model. This realizes the prediction of subjective encoding quality based on objectively calculable indicator features during the encoding process through a machine learning model corresponding to the subjective quality prediction model, thereby improving the accuracy and efficiency of encoding decisions.

[0131] The methods for obtaining quality indicator data differ depending on the type of candidate code. Taking QP as an example, in some embodiments, the quality indicator data includes at least one of the following indicators.

[0132] 1.1 Texture Distortion Index: The texture distortion index characterizes the accuracy loss of the current frame during the quantization process. Before and after quantization in the encoding process, the current frame experiences numerical loss due to the quantization of its data. The texture distortion index represents this numerical loss. Correspondingly, the texture distortion index is obtained by statistically processing the error data of the current frame before and after quantization. Each CTU acquires the processing result obtained by transforming the residual data during the encoding process, i.e., the frequency domain result. The frequency domain result is then quantized and dequantized to obtain the reconstructed frequency domain result. Based on the error between the frequency domain result and the reconstructed frequency domain result, the texture distortion index is obtained. The transform process decomposes the residual data using a set of orthogonal basis functions to obtain coefficients representing different frequency components, i.e., the frequency domain result; in some embodiments, the frequency domain result is referred to as the transform coefficients. The frequency domain result projects the residual data from the pixel value representation in the spatial domain to the corresponding coefficient representation in the frequency domain. By concentrating the main image information corresponding to the CTU, such as smooth region information and overall contour information, onto a few low-frequency coefficients, the high-frequency coefficients are made close to zero, thus facilitating subsequent compression coding processes such as quantization. Optionally, the transform processing Discrete Cosine Transform (DCT), including orthogonal basis functions, can be implemented as cosine waves in the DCT.

[0133] Schematic illustration: In the encoding process, quantization is performed after transform processing. Quantization processes the result of transform processing on the current frame, i.e., the frequency domain result. Quantization rounds the frequency domain result by the quantizer according to the quantization step size. The purpose of quantization is to compress the data volume by dividing the frequency domain result by the quantization step size and rounding it, discarding high-frequency details and redundant information that are insensitive to the human eye, making many coefficients in the frequency domain result zero. This improves the compression efficiency of subsequent processing in the encoding process (such as entropy coding) and reduces the bit rate while keeping distortion under control. However, quantization results in a loss of precision in the frequency domain result corresponding to the current frame. When recovering the result through dequantization, the exact value of the original frequency domain result cannot be restored, resulting in irreversible errors. The error generated during the quantization process is the first-dimensional data difference of the current frame before and after encoding, represented by the texture distortion index, where the texture distortion index is the first-dimensional index data.

[0134] The process of obtaining the texture distortion index based on the error between the frequency domain result and the reconstructed frequency domain result includes: for at least one CU in the CTU, calculating the variance of the frequency domain result and the reconstructed frequency domain result corresponding to the CU, and the covariance between the frequency domain result and the reconstructed frequency domain result, and obtaining the MSE as the error energy corresponding to the CU; accumulating the error energy corresponding to each CU in the CTU, and normalizing the sum of the error energy, for example, normalizing the sum of the error energy to [0,100], to obtain the normalized result, which is used as the texture distortion index. Schematic, the error energy D corresponding to the CU can be expressed as the following formula (1).

[0135] (1)

[0136] Where x is the frequency domain result corresponding to CU, and q is the reconstructed frequency domain result corresponding to CU. It is the variance of the frequency domain result. It is the variance of the reconstructed frequency domain result. It is the covariance between the frequency domain result and the reconstructed frequency domain result.

[0137] For illustrative purposes, please refer to the following: Figure 6 , Figure 6 This is a schematic diagram of a process for obtaining texture distortion metrics provided in an exemplary embodiment of this application, as shown below. Figure 6 As shown, during the encoding process of each CTU, the frequency domain result 610 and the reconstructed frequency domain result 620 corresponding to at least one CU in the CTU are obtained respectively. H (High) and L (Low) represent the high and low frequencies of the frequency components in the frequency domain result 610, respectively. LL is the low-frequency horizontal and low-frequency vertical component, used to indicate the main information and contour information in the residual data; LH is the low-frequency horizontal and high-frequency vertical component, used to indicate the edges and texture details in the horizontal direction; HL is the high-frequency horizontal and low-frequency vertical component, used to indicate the edges and texture details in the vertical direction; HH is the high-frequency horizontal and high-frequency vertical component, used to indicate the edges and texture details in the diagonal direction. Accordingly, the reconstructed frequency domain result 620 includes four frequency components: L'L', L'H', H'L', and H'H'.

[0138] like Figure 6 As shown, based on the variances of the frequency domain results 610 and the reconstructed frequency domain results 620 corresponding to the CU, and the covariance between the frequency domain results 610 and the reconstructed frequency domain results 620, the MSE between the frequency domain results 610 and the reconstructed frequency domain results 620 is obtained as the error energy 630 corresponding to the CU; the error energy 630 corresponding to each CU in the CTU is accumulated, and the sum of the error energy 630 is normalized to obtain the texture distortion index 640 corresponding to the CTU.

[0139] 1.2 Texture Complexity Metric: The texture complexity metric is used to characterize the loss of image detail in the current frame before and after encoding. Before and after quantization processing in the encoding process, the current frame experiences image detail loss due to the quantization of its data. The texture complexity metric characterizes this loss of detail. Compared to the texture distortion metric, which focuses on measuring the accuracy loss of the frequency domain results (i.e., transform coefficients) and directly reflects the degree of information loss during quantization, the texture complexity metric focuses on measuring the visual detail loss in the spatial domain, reflecting the degree of information loss in subjectively visually sensitive areas such as edges. It is more closely related to subjective visual experience and encoding quality.

[0140] Accordingly, statistical processing is performed on the image detail loss data before and after quantization of the data corresponding to the current frame to obtain the texture complexity index in the quality index data. Based on the original data and reconstructed data corresponding to each CTU, original texture data and reconstructed texture data are obtained respectively, where the texture data is used to indicate the intensity of pixel changes in the image data; and the texture complexity index is obtained based on the error between the original texture data and the reconstructed texture data. For example, the Sobel operator can be used to obtain the original texture data and the reconstructed texture data.

[0141] The difference in the second dimension of the current frame before and after encoding, as represented by the texture complexity index, mainly arises from the quantization process in the encoding workflow. Because the quantizer rounds the result of the transform processing by a step size, high-frequency details are discarded. Specifically, the loss of high-frequency details leads to the loss of information in areas of drastic texture change, such as edges, noisy regions (e.g., sensor noise), and local motion boundaries (the boundary between moving objects and the background), resulting in image detail loss in the reconstructed image in the spatial domain, including blurred edges, lost texture, or artifacts. These areas of drastic texture change are subjectively visually sensitive, hence the aforementioned subjectively visually sensitive areas. This loss of image detail is reflected in the gradient magnitude maps before and after encoding, which refer to the original texture data and reconstructed texture data corresponding to the original data and reconstructed data, respectively.

[0142] Schematic representation: The Sobel operator includes horizontal and vertical operators. The horizontal operators are used separately. and vertical operator The original data is convolved to obtain horizontal texture data. and vertical texture data ; using level operators respectively and vertical operator The reconstructed data is convolved to obtain horizontally reconstructed texture data. and vertical reconstruction texture data The Sobel operator calculates the first derivatives of the image data in the horizontal and vertical directions to obtain the gradient values ​​at each pixel location. The larger the gradient value, the more drastic the pixel value change at that location, i.e., the stronger the texture, which may correspond to texture information such as edges and details. The smaller the gradient value, i.e., close to 0, the smoother the texture at that pixel location.

[0143] In some embodiments, the texture complexity index is represented by the Additive Distortion Measurement (ADM). Schematic, for each pixel in the CTU, the ADM can be expressed as the following formula (2).

[0144] (2)

[0145] in, It is a constant used to avoid division by zero errors during calculation by using a smaller positive number, thereby improving calculation stability. It is the absolute value of the texture sum of the original data. is the absolute value of the texture sum of the reconstructed data, based on the ratio of the texture sum of the original data to that of the reconstructed data. It represents the error between the original texture data and the reconstructed texture data. The closer this ratio is to 1, the higher the consistency of the data before and after encoding. 'a' is the preset maximum value of the ADM corresponding to this pixel; for example, a = 1.2.

[0146] Given the ADM corresponding to each pixel in the CTU, normalize the ADM corresponding to each pixel to obtain the texture complexity index corresponding to the CTU. Indicatively, the texture complexity index can be expressed as the following formula (3).

[0147] (3)

[0148] in, It is the number of pixels in this CTU. It is the maximum value in the ADM corresponding to at least one pixel in the CTU. It is the minimum value in the ADM corresponding to at least one pixel in the CTU. Through normalization, the texture complexity index is constrained to the range of [0,1], eliminating the influence of absolute values ​​on the texture complexity index, so that the texture complexity index can be integrated or compared between different CTUs.

[0149] 1.3 Frame Difference Metric: The frame difference metric characterizes the loss of pixel information in the current frame before and after encoding. The current frame contains data errors before and after encoding. Specifically, during encoding processes such as quantization, lossy compression is performed on the pixel data of the current frame to compress the data volume, resulting in the aforementioned data errors. Lossy compression includes transform processing and quantization processing. The frame difference metric characterizes the pixel-level data errors generated during the entire encoding process.

[0150] In the encoding process, transformation processing includes, but is not limited to, transforming the original image signal corresponding to the current frame to the frequency domain to obtain transform coefficients, i.e., the frequency domain result, so that the information of the current frame is concentrated on the low-frequency coefficients; and performing quantization processing on the current frame, including dividing and rounding the frequency domain result to discard tiny, human-eye-insensitive high-frequency details and precision redundancy, reducing the amount of data corresponding to the current frame. After quantization, inverse quantization, inverse transform, and loop filtering of the transformation coefficients corresponding to the current frame, the reconstructed frame corresponding to the current frame is obtained. Therefore, there is a data error between the original frame and the reconstructed frame of the current frame. That is, the reconstructed frame is the current frame after encoding processing, and the original frame is the current frame before encoding processing. The reconstruction of the original frame by the reconstructed frame is not accurate, and there is a loss of pixel information. The data error between the original frame and the reconstructed frame mentioned above is the third-dimensional data difference of the current frame before and after encoding, represented by the frame difference index, where the frame difference index is the third-dimensional index data.

[0151] In some embodiments, the current frame includes at least one current block, and the frame difference index of the current block is the data error between the original block and the reconstructed block of the current block, representing the data error generated by the current block during the encoding process. For example, in the compression encoding process, in order not to consume excessive bits, the quantization step size is large when quantizing the current frame (or the current block), resulting in a large data error between the original frame and the reconstructed frame (or the original block and the reconstructed block), such as a large error between pixel values. Accordingly, the data consistency before and after encoding of the current frame (or the current block) is low.

[0152] In some embodiments, after statistical processing of the data errors corresponding to at least one current block in the current frame, the frame difference index in the quality index data is obtained. The frame difference index is represented by the frame difference method, whereby the frame difference is the difference between the pixel values ​​corresponding to the reconstructed data (including reconstructed blocks or reconstructed frames) and the original data (including original blocks or original frames).

[0153] To illustrate, the frame difference data corresponding to the current block is obtained by acquiring the frame difference of each pixel in the current block; the coded blocks in the current frame are traversed, and the frame difference value corresponding to the current frame is obtained as the frame difference index based on the frame difference data corresponding to each coded block.

[0154] The process of obtaining the frame difference data corresponding to the current block based on the frame difference of each pixel includes: arranging the frame differences of at least one pixel in the current block in descending order, and obtaining the frame differences of the first N pixels as the frame difference data corresponding to the current block, where N is a positive integer. Using N larger frame differences helps prevent flat regions in the current block from weakening or eliminating the motion. Optionally, the frame difference of each pixel can be represented as... ,in, It is the pixel value corresponding to that pixel position in the reconstructed data. It is the pixel value corresponding to that pixel position in the original data.

[0155] To illustrate, obtaining the frame difference value corresponding to the current frame based on the frame difference data corresponding to each coding block includes: arranging the frame difference data corresponding to the previous coding block and the frame difference data corresponding to the current block in descending order of frame difference, obtaining the first N frame differences as the updated frame difference data of the current block, until traversing to the last coding block in the current frame, obtaining the updated frame difference data corresponding to the last coding block, averaging the frame difference data to obtain the frame difference value corresponding to the current frame, which is used as the frame difference index.

[0156] For illustrative purposes, please refer to the following: Figure 7 , Figure 7 This is a schematic diagram of the process for obtaining frame difference metrics provided in an exemplary embodiment of this application, as shown below. Figure 7 As shown, the frame difference of at least one pixel in the current block 711 is obtained. The frame differences of at least one pixel in the current block 711 are arranged in descending order. The frame differences of the first N pixels in the current block 711 are stored in the difference list 712, where N is a positive integer, optionally N=16. The frame difference of at least one pixel in the next coding block 721 of the current block is obtained. The frame differences of the first N pixels in the next coding block 721 of the current block are sorted with the frame differences of the N pixels in the difference list 712 in descending order. The first N frame differences are retained to update the difference list 712, resulting in the updated difference list 722.

[0157] like Figure 7 As shown, it is understood that the above-mentioned current block 711 and the next coding block 721 of the current block are only illustrative examples. When the current block 711 is the first coding block in the current frame, the frame difference of N pixels in the difference list 712 is the first N frame difference in the current block 711. When the current block 711 is not the first coding block in the current frame, the first N frame difference in the current block 711 is used to update the difference list 712. The frame difference of N pixels in the difference list 712 includes the first N larger frame differences between the first N frame differences in the current block 711 and the first N larger frame differences between the first N frame differences in the previous coding block of the current block 711.

[0158] like Figure 7As shown, the above frame difference calculation process is repeated until the next coding block 721 of the current block is the last coding block in the current frame. The N frame differences in the updated difference list 722 are averaged to obtain the frame difference value corresponding to the current frame, which is used as the frame difference index 730.

[0159] 2. Obtain the quality index data corresponding to multiple candidate coding parameters through the coding processing model. The coding processing model is a pre-trained machine learning model used to express the correspondence between candidate coding parameters and quality index data.

[0160] Optionally, the encoding processing model is a pre-trained machine learning model and is stored in the encoder or the computer device that executes the encoding method. The encoding processing model simulates the ability to encode image frames under candidate encoding parameters, taking multiple candidate encoding parameters and the current frame as input, and outputting quality index data corresponding to the multiple candidate encoding parameters during the encoding process.

[0161] It is worth noting that the above-described methods for obtaining quality indicator data are merely illustrative examples, and this application does not limit the specific implementation methods for obtaining corresponding quality indicator data based on multiple candidate coding parameters.

[0162] By expanding the methods for acquiring quality indicator data, applicable approaches are provided in different coding processes. Based on multiple candidate coding parameters, corresponding quality indicator data are obtained respectively, improving the accuracy of quality indicator data and the accuracy of coding quality based on quality indicator data, thereby improving the accuracy and stability of coding decisions.

[0163] It is worth noting that at least one dimension of the quality index data mentioned above, such as the first dimension index data, the second dimension index data, and the third dimension index data, can be directly obtained by the encoder during the intermediate process of encoding processing of the current frame, or directly output by the encoding processing model. This has low computational complexity and is suitable for real-time encoding.

[0164] Step 424: Input the quality index data into the subjective quality prediction model and output the coding quality corresponding to multiple candidate coding parameters.

[0165] In some embodiments, the quality indicator data includes an n-dimensional vector, where the i-th element is the indicator data of the i-th dimension, and n is a positive integer, 0 < i ≤ n. The method for obtaining the coding quality corresponding to multiple candidate coding parameters based on the quality indicator data includes: inputting the n-dimensional vector into a subjective quality prediction model, and outputting the coding quality corresponding to each of the multiple candidate coding parameters. The subjective quality prediction model is used to express the correspondence between the candidate coding parameters and the n-dimensional vector.

[0166] To illustrate, when the quality metrics data include texture distortion, texture complexity, and frame difference metrics, the first element of the n-dimensional vector (n=3) is the texture distortion metric corresponding to the current frame, the second element is the texture complexity metric corresponding to the current frame, and the third element is the frame difference metric corresponding to the current frame. The quality metrics data (i.e., the above 3-dimensional vector) corresponding to multiple candidate codes are input into the subjective quality prediction model, and the output is the coding quality corresponding to the multiple candidate coding parameters.

[0167] By expressing the degree of difference between the data before and after encoding through the indicator values ​​corresponding to different quality index dimensions, the subjective evaluation of image quality is transformed from discrete qualitative labels into numerical values ​​in a continuous space, which improves the accuracy of describing the consistency of data before and after encoding, thereby improving the accuracy of obtaining encoding quality based on candidate encoding parameters, improving the efficiency of encoding decisions and the stability of the encoding process.

[0168] For illustrative purposes, please refer to the following: Figure 8 , Figure 8 This is a flowchart illustrating the encoding method provided in an exemplary embodiment of this application, as shown below. Figure 8 As shown, multiple candidate coding parameters are obtained, including a first candidate coding parameter 811, a second candidate coding parameter 812, and a third candidate coding parameter 813. Quality indicator data 830 corresponding to each of the multiple candidate coding parameters are obtained through a video coding process 820. The video coding process 820 includes, but is not limited to, performing complete coding processing on the current frame based on each of the multiple candidate coding parameters, or simulating the coding processing process corresponding to the multiple candidate coding parameters using a pre-trained coding processing model. Optionally, the quality indicator data 830 includes, but is not limited to, at least one of the following indicators: texture distortion indicator, texture complexity indicator, and frame difference indicator.

[0169] like Figure 8 As shown, after obtaining quality index data 830 corresponding to multiple candidate codes, the subjective quality prediction model 840 obtains the coding quality corresponding to the multiple candidate coding parameters. Optionally, the coding quality can be implemented as a quality score, including the first coding quality 851 corresponding to the first candidate parameter 811, the second coding quality 852 corresponding to the second candidate parameter 812, and the third coding quality 853 corresponding to the third candidate parameter 813. The target coding parameter 860 that meets the quality requirements is obtained from the multiple candidate codes; the bitstream data 870 encoded for the current frame under the target coding parameter 860 is output.

[0170] like Figure 8As shown, optionally, the output of the bitstream data 870 encoded for the current frame under the target encoding parameters 860 includes at least one of the following methods: Before obtaining the target encoding parameters 860, the bitstream data 870 has been obtained through complete encoding processing. For example, in the video encoding process 820, the current frame has been completely encoded based on multiple candidate encoding parameters to obtain the bitstream data of the current frame corresponding to the multiple candidate encoding parameters, including the bitstream data 870 of the current frame corresponding to the target encoding parameters 860. In this case, the bitstream data 870 is directly output; or, when the target encoding parameters 860 corresponding to the current frame are determined, the bitstream data 870 encoded for the current frame under the target encoding parameters 860 has not yet been obtained. In this case, the current frame is completely encoded based on the target encoding parameters 860 through the video encoding process 820, and the bitstream data 870 is output.

[0171] It is worth noting that models such as Video Multimethod Assessment Fusion (VMAF), which are used to indicate the quality of video before and after encoding, require feature extraction from the image data before and after encoding. Compared with models such as VMAF, subjective quality prediction models, given input quality index data, output the encoding quality corresponding to multiple candidate encoding parameters respectively. By predicting the results through end-to-end forward propagation, the cumbersome feature extraction steps are bypassed, significantly reducing the amount of computation and improving the overall computational efficiency.

[0172] In summary, this embodiment uses a subjective quality prediction model to obtain the coding quality corresponding to multiple candidate coding parameters for the current frame. This provides an evaluation of the data consistency between the original data and the reconstructed data before and after coding the current frame. The coding quality directly determines the target coding parameters that meet the quality requirements, improving the efficiency of obtaining a coding mode suitable for the current frame and increasing the compression efficiency of the coding process while ensuring coding quality. Furthermore, the evaluation criteria corresponding to the subjective quality prediction model reduce coding decision errors, and the current frame is encoded based on the coding mode indicated by the target coding parameters, improving the accuracy and stability of the coding process.

[0173] The method provided in this embodiment indicates the objective differences between the current frame and the previous and current frames before and after encoding processing under candidate encoding parameters through quality index data. Based on the quality index data, a subjective quality prediction model representing the correspondence between quality index data and encoding quality is used to obtain the encoding quality corresponding to multiple candidate encoding parameters. This improves the accuracy of obtaining encoding quality, thereby improving the accuracy of obtaining the target encoding parameter among multiple candidate encoding parameters based on encoding quality, improving the accuracy of encoding decision-making and the encoding process, and improving the stability of encoding processing.

[0174] Figure 9This is a flowchart of an encoding method provided in another exemplary embodiment of this application. The method is applied to an encoder, or the method is implemented by a computer device (which may be configured as follows). Figure 3 The method is executed by the terminal 320 or server 340 shown, or it is executed jointly by the terminal and server interacting. Figure 9 As shown, steps 910 to 920 are included before step 424 above.

[0175] Step 910: Obtain the reference quality index data corresponding to the sample image frame, and obtain the reference coding quality of the sample image frame annotation.

[0176] There is a correspondence between the reference quality index data and the reference coding quality based on the sample image frames.

[0177] In some embodiments, when obtaining sample image frames, the sample image frames are encoded using a complete encoding process under reference candidate encoding parameters. This includes obtaining reconstructed data of the sample image frames corresponding to the reference candidate encoding parameters, obtaining reference quality index data based on the reconstructed data, and, based on the bitstream data obtained after encoding the sample image frames, decoding the decoded image corresponding to the sample image frames to obtain the decoded image corresponding to the sample image frames. The data consistency between the decoded image and the sample image frames is evaluated from the user's perspective, resulting in a reference encoding quality label on the sample image frames, or establishing a correspondence between the reference encoding quality corresponding to the sample image frames and the reference quality index data corresponding to the sample image frames.

[0178] Optionally, methods for evaluating the data consistency between the decoded image and the sample image frame from the user's perspective include, but are not limited to: selecting a large number of experimenters, randomly assigning them to independent groups to avoid communication between different groups, and collecting the evaluation results of the experimenters on the unlabeled decoded image and the sample image frame to obtain a reference coding quality. For decoded images and sample image frames with significantly differing evaluation results, the final evaluation result is determined through expert arbitration or majority voting, and this result serves as the reference coding quality.

[0179] In some embodiments, multiple reference candidate coding parameters corresponding to a sample image frame are obtained, and the correspondence between the reference coding quality labeled on the sample image frame and the reference candidate coding parameters is obtained. It is understood that there is a correspondence between the reference quality index data obtained by encoding the sample image frame under the reference candidate coding parameters and the corresponding reference candidate coding parameters. Therefore, the same sample image frame, when encoded under multiple different reference candidate coding methods, yields multiple different reference quality index data. Correspondingly, the sample image frame has multiple different decoded images, and thus multiple different reference coding qualities. Alternatively, for different sample image frames, different reference quality index data are obtained based on the reference candidate coding parameters, and the reference coding quality corresponding to the reference quality index data is obtained.

[0180] Step 920: Train a subjective quality prediction model based on the reference quality index data and the reference coding quality corresponding to the reference quality index data.

[0181] The subjective quality prediction model is a machine learning model trained on sample image frames. Each sample image frame is labeled with a reference encoded quality, which is expressed from a subjective perspective and represents the consistency of the data before and after encoding. Optionally, the subjective quality prediction model is a deep learning model.

[0182] In some embodiments, when obtaining reference quality index data corresponding to sample image frames and obtaining reference coding quality of sample image frame annotations, the above training data (including reference quality index data and reference coding quality corresponding to reference quality data) is divided into training set, validation set and test set, and the above training data is preprocessed to ensure that the training data meets the input requirements corresponding to the subjective quality prediction model, wherein the preprocessing includes, but is not limited to, standardization processing, normalization processing, etc.

[0183] In some embodiments, the process of training a subjective quality prediction model based on reference quality index data and the reference coding quality corresponding to the reference quality index data includes, but is not limited to: obtaining a subjective quality prediction model to be trained; obtaining the predicted coding quality based on the reference quality index data and the subjective quality prediction model to be trained; obtaining a loss value based on the predicted coding quality and the reference coding quality, wherein the loss value is used to indicate the difference between the predicted coding quality and the reference coding quality; and training the subjective quality prediction model to be trained based on the loss value to obtain the subjective quality prediction model.

[0184] Optionally, the subjective quality prediction model to be trained is a deep learning model, which can be an untrained deep learning model architecture, or a subjective quality prediction model obtained within a historical time period.

[0185] In some embodiments, a loss value corresponding to a preset loss function is obtained based on the predicted coding quality and the reference coding quality. For example, the loss function includes, but is not limited to, the MSE function and the Mean Absolute Error (MAE) function.

[0186] In some embodiments, the process of training a subjective quality prediction model based on a loss value to obtain the subjective quality prediction model includes: adjusting the model parameters in the subjective quality prediction model to be trained based on the difference between the predicted coding quality and the reference coding quality, i.e., the loss value; for example, adjusting the specified parameters in the subjective quality prediction model to be trained according to the loss value between the predicted coding quality and the reference coding quality using the gradient corresponding to the loss value; continuing to make predictions based on the reference quality index data through the subjective quality prediction model to be trained; and determining the loss value between the predicted coding quality and the reference coding quality again; if the loss value does not meet the convergence requirement, continuing to train the subjective quality prediction model to be trained through iterative training until the difference between the predicted coding quality and the reference coding quality of the sample image frames analyzed by the subjective quality prediction model to be trained is less than the difference threshold, i.e., the training result of the subjective quality prediction model has reached the convergence condition.

[0187] The difference threshold is preset, for example, it is used to indicate the convergence condition for multiple iterations in the iterative training process of the subjective quality prediction model through training parameters.

[0188] The process of acquiring reference quality index data corresponding to sample image frames includes encoding the sample image frames based on preset reference coding parameters to obtain reference reconstructed data for the sample image frames; and acquiring reference quality index data based on this reference reconstructed data. The reference quality index data is used to objectively characterize the data differences of the sample image frames before and after encoding. The reference coding quality labeled on the sample image frames is a subjective evaluation result obtained from the user's perspective regarding the data consistency between the sample image frames before and after encoding. By obtaining the loss value through the difference between the predicted coding quality and the reference coding quality, a continuous and differentiable quantitative indicator is used to clarify the training direction in the process of training the subjective quality prediction model based on the predicted coding quality and the reference coding quality. This enables the subjective quality prediction model to have the ability to predict coding quality, and to predict subjective coding quality based on objective quality index data. This provides a basis for the subjective quality prediction model to learn the correspondence between the reference quality index data and the reference coding quality, thereby improving the training efficiency of the subjective quality prediction model.

[0189] In some embodiments, the process of training a subjective quality prediction model based on the loss value to obtain the subjective quality prediction model includes, but is not limited to: obtaining the gradient value corresponding to the subjective quality prediction model to be trained through the backpropagation algorithm based on the loss value; and adjusting the model parameters of the subjective quality prediction model to be trained based on the gradient value to obtain the subjective quality prediction model.

[0190] Schematic, the subjective quality prediction model to be trained includes an input layer, an intermediate layer, and an output layer. Having obtained the loss value between the reference coding quality and the predicted coding quality, the gradient of the loss value with respect to each model parameter is obtained based on the chain rule, and the gradients corresponding to each model parameter are backpropagated from the output layer to the input layer. Optionally, during the process of obtaining the predicted coding quality corresponding to the reference quality index data through the subjective quality prediction model to be trained, if feature dimensionality reduction processing is performed on the input feature vector corresponding to the reference quality index data in the intermediate layer, then during backpropagation, the gradient values ​​are recovered through upsampling or other methods to ensure the accuracy of backpropagation.

[0191] The backpropagation algorithm described above automatically calculates the gradient of the preset loss function relative to the model parameters of the subjective quality prediction model to be trained. This allows for the determination of the necessary adjustments to be made in each iteration, along with the urgency of these adjustments, thereby adjusting the model parameters. This improves the training efficiency of the subjective quality prediction model's parameters. Furthermore, it ensures that the loss value is reduced in each iteration, and the loss function helps the predicted coding quality approximate the reference coding quality. This allows the subjective quality prediction model to learn the correspondence between the reference quality index data and the reference coding quality, improving the training accuracy and the accuracy of coding quality predictions based on the subjective quality prediction model.

[0192] Optionally, the model parameters of the subjective quality prediction model to be trained are adjusted based on the gradient value through multiple rounds of iteration. The multiple rounds of iteration are controlled by training parameters, which include, but are not limited to, at least one of the following parameters.

[0193] 1. Learning rate: Used to control the step size of parameter updates in each round. The size of the learning rate affects the efficiency of training. For example, if the learning rate is too large, the parameter iteration trajectory will be unstable, making it difficult to reach the convergence condition and ending the iteration. If the learning rate is too small, it will lead to slow convergence or cause the iteration process to stop prematurely in the suboptimal solution region.

[0194] 2. Batch size: This indicates the size of the subset of sample data input to the subjective quality prediction model at one time during each round of iterative training, i.e., the number of sample data contained in the batch.

[0195] Batch size affects the stability of the iterative training process. For example, a larger batch size results in a higher accuracy of the first gradient. Furthermore, batch size affects the processor's memory requirements. A larger batch size requires more storage space during training. Additionally, batch size affects the convergence speed during iterative training. A larger batch size allows for more data to be processed in parallel in each round, resulting in a faster iteration speed per round. However, this also means that more iterations are needed to reach the convergence condition, leading to a slower convergence speed.

[0196] 3. Training epochs refer to the number of iterations during training. Correspondingly, an optional implementation of the convergence condition is a training epoch threshold, which indicates the upper limit of the number of iterations during training. Optionally, this training epoch threshold is determined based on the complexity of the reference quality index data corresponding to the sample image frames.

[0197] The number of training epochs affects the computational complexity and model training quality during the training process. To illustrate, too many training epochs can lead to overfitting of the subjective quality prediction model, resulting in lower accuracy of the encoding quality prediction based on the subjective quality prediction model. Too many training epochs can also lead to lower generalization performance of the subjective quality prediction model, resulting in lower accuracy of the encoding quality prediction based on the subjective quality prediction model.

[0198] It is worth noting that the above-mentioned method of setting training parameters is only an illustrative example, and the specific process of adjusting the parameters of the subjective quality prediction model based on gradient values ​​through multiple rounds of iteration is not limited in the embodiments of this application.

[0199] For illustrative purposes, please refer to the following: Figure 10 , Figure 10 This is a schematic diagram of the process for training a subjective quality prediction model provided in an exemplary embodiment of this application, such as... Figure 10 As shown, after obtaining the reference quality index data 1010 corresponding to the sample image frame and obtaining the subjective quality prediction model 1020 to be trained, the first reference index data 1011, the second reference index data 1012, and the third reference index data 1013 included in the reference quality index data 1010 are input into the subjective quality prediction model 1020 to be trained. Optionally, the reference quality index data 1010 is input into the subjective quality prediction model 1020 to be trained in the form of an n-dimensional vector, wherein the first dimension of the n-dimensional vector is the first reference index data 1011, the second dimension is the second reference index data 1012, and the third dimension is the third reference index data 1013.

[0200] like Figure 10As shown, the input layer of the subjective quality prediction model 1020 to be trained performs preprocessing on the reference quality index data 1010, and the intermediate layer of the subjective quality prediction model 1020 to be trained performs feature processing on the input feature vector corresponding to the reference quality index data 1010, including but not limited to feature dimensionality reduction through downsampling, such as reducing the dimension of the input feature vector corresponding to the reference quality index data 1010 from d dimensions to d' dimensions, extracting abstract features from the input feature vector, discarding redundant spatial details, and retaining high-level semantic information, thereby compressing the amount of data corresponding to the reference quality index data 1010 and reducing the computational burden of the subsequent model.

[0201] like Figure 10 As shown, after feature processing is completed, the output layer of the subjective quality prediction model 1020 to be trained generates a predicted coding quality 1030. The loss function is used to obtain the loss value between the reference coding quality and the predicted coding quality 1030 corresponding to the sample image frame. Based on the loss value, the model parameters in the subjective quality prediction model 1020 to be trained are adjusted. For example, the model parameters are updated using the gradient descent algorithm based on the gradient of the loss value with respect to each model parameter. The updated subjective quality prediction model 1020 continues to predict the corresponding predicted coding quality 1030, and the above model parameter adjustment process is repeated until the difference between the predicted coding quality obtained by the subjective quality prediction model 1020 to be trained and the reference coding quality is less than the difference threshold. That is, the training result of the subjective quality prediction model 1020 to be trained converges, resulting in the subjective quality prediction model 840.

[0202] In summary, this embodiment uses a subjective quality prediction model to obtain the coding quality corresponding to multiple candidate coding parameters for the current frame. This provides an evaluation of the data consistency between the original data and the reconstructed data before and after coding the current frame. The coding quality directly determines the target coding parameters that meet the quality requirements, improving the efficiency of obtaining a coding mode suitable for the current frame and increasing the compression efficiency of the coding process while ensuring coding quality. Furthermore, the evaluation criteria corresponding to the subjective quality prediction model reduce coding decision errors, and the current frame is encoded based on the coding mode indicated by the target coding parameters, improving the accuracy and stability of the coding process.

[0203] The method provided in this embodiment trains a subjective quality prediction model by referencing quality index data and the corresponding reference coding quality. The correspondence between the reference quality index data and the reference coding quality provides a basis for updating the subjective quality prediction model. Based on the reference coding quality, the difference between the predicted coding quality obtained by the subjective quality prediction model and the reference coding quality can be obtained, thereby objectively quantifying the performance of the subjective quality prediction model and improving the model training efficiency and accuracy.

[0204] Please refer to Figure 11 This diagram illustrates a structural block diagram of an encoding apparatus provided in an exemplary embodiment of this application. The encoding apparatus includes at least one of the following modules.

[0205] The acquisition module 1110 is configured to acquire multiple candidate encoding parameters, which are used to indicate the encoding method for the current frame;

[0206] The acquisition module 1110 is also configured to acquire the coding quality corresponding to multiple candidate coding parameters through a subjective quality prediction model. The subjective quality prediction model is a machine learning model trained on sample image frames. The sample image frames are labeled with reference coding quality. The reference coding quality is labeled from a subjective perspective and is used to express the data consistency before and after the sample image frames are encoded.

[0207] The acquisition module 1110 is also configured to acquire target coding parameters whose coding quality meets the quality requirements from multiple candidate coding parameters;

[0208] Output module 1120 is configured to output the bitstream data encoded for the current frame under the target encoding parameters.

[0209] In an optional embodiment, the acquisition module 1110 is further configured to acquire quality index data corresponding to multiple candidate coding parameters, wherein the quality index data is used to characterize the data difference of the current frame before and after encoding.

[0210] The acquisition module 1110 is also configured to input quality index data into the subjective quality prediction model and output the coding quality corresponding to multiple candidate coding parameters.

[0211] In an optional embodiment, the acquisition module 1110 is configured to acquire quality index data corresponding to multiple candidate coding parameters, including at least one of the following:

[0212] The acquisition module 1110 is also configured to encode the current frame based on multiple candidate coding parameters to obtain the reconstructed data of the current frame corresponding to the multiple candidate coding parameters; and to acquire quality index data based on the reconstructed data.

[0213] The acquisition module 1110 is also configured to acquire quality index data corresponding to multiple candidate coding parameters through an encoding processing model. The encoding processing model is a pre-trained machine learning model used to express the correspondence between candidate coding parameters and quality index data.

[0214] In an optional embodiment, the acquisition module 1110 is further configured to acquire, for each candidate coding parameter, the reconstructed block corresponding to at least one coding block in the current frame;

[0215] The acquisition module 1110 is also configured to acquire, for each candidate coding parameter, at least one coding block and the corresponding reconstructed block of the coding block to obtain the difference data corresponding to at least one coding block respectively;

[0216] The acquisition module 1110 is also configured to acquire quality index data based on the difference data corresponding to at least one coding block.

[0217] In an optional embodiment, the quality indicator data includes an n-dimensional vector, where the i-th element is the indicator data of the i-th dimension, n is a positive integer, and 0 < i ≤ n; the acquisition module 1110 acquires the quality indicator data corresponding to multiple candidate encoding parameters, including at least one of the following:

[0218] The acquisition module 1110 is also configured to acquire the texture distortion index as the first dimension index data. The texture distortion index is used to characterize the accuracy loss of the current frame during the quantization process.

[0219] The acquisition module 1110 is also configured to acquire the texture complexity index as the second dimension index data. The texture complexity index is used to characterize the loss of image details before and after encoding of the current frame.

[0220] The acquisition module 1110 is also configured to acquire the frame difference index as the third dimension index data. The frame difference index is used to characterize the loss of pixel information before and after the current frame is encoded.

[0221] The acquisition module 1110 is also configured to input an n-dimensional vector into the subjective quality prediction model and output the coding quality corresponding to multiple candidate coding parameters.

[0222] In an optional embodiment, the acquisition module 1110 is further configured to acquire at least one undetermined encoding parameter from a plurality of candidate encoding parameters, wherein the encoding quality corresponding to the undetermined encoding parameter meets the quality requirements.

[0223] The acquisition module 1110 is also configured to acquire the bitstream size corresponding to at least one pending encoding parameter, wherein the bitstream size is the amount of data of the bitstream data encoded for the current frame under the pending encoding parameters;

[0224] The acquisition module 1110 is also configured to arrange at least one undetermined encoding parameter in ascending order of bitstream size to obtain the first undetermined encoding parameter as the target encoding parameter.

[0225] In an optional embodiment, the acquisition module 1110 is configured to acquire at least one undetermined encoding parameter from a plurality of candidate encoding parameters, including at least one of the following:

[0226] The acquisition module 1110 is also configured to acquire candidate encoding parameters whose encoding quality reaches a preset first threshold, as undetermined encoding parameters;

[0227] The acquisition module 1110 is further configured to acquire candidate encoding parameters whose encoding quality is within a preset quality set as undetermined encoding parameters, wherein the quality set includes at least one encoding quality that meets the quality requirements, or the quality set is a range of values ​​corresponding to the quality requirements.

[0228] The acquisition module 1110 is also configured to acquire the top K candidate encoding parameters in descending order of encoding quality, as at least one undetermined encoding parameter, where K is a positive integer.

[0229] In an optional embodiment, Figure 12 This is a structural block diagram of an encoding device provided in another exemplary embodiment of this application, such as... Figure 12 As shown, before the acquisition module 1110 acquires the coding quality corresponding to multiple candidate coding parameters through the subjective quality prediction model, the coding device further includes a model training module 1130, which is configured to acquire reference quality index data corresponding to the sample image frame, and acquire the reference coding quality of the sample image frame annotation, and there is a corresponding relationship between the reference quality index data and the reference coding quality.

[0230] The model training module 1130 is also configured to train a subjective quality prediction model based on the reference quality index data and the reference coding quality corresponding to the reference quality index data.

[0231] In an optional embodiment, the model training module 1130 is further configured to acquire a subjective quality prediction model to be trained.

[0232] The model training module 1130 is also configured to obtain the predicted coding quality based on the reference quality index data and the subjective quality prediction model to be trained.

[0233] The model training module 1130 is also configured to obtain a loss value based on the predicted coding quality and the reference coding quality, the loss value being used to indicate the difference between the predicted coding quality and the reference coding quality;

[0234] The model training module 1130 is also configured to train the subjective quality prediction model to be trained based on the loss value, thereby obtaining the subjective quality prediction model.

[0235] In an optional embodiment, the model training module 1130 is further configured to obtain the gradient value corresponding to the subjective quality prediction model to be trained through a backpropagation algorithm based on the loss value.

[0236] The model training module 1130 is also configured to adjust the model parameters of the subjective quality prediction model to be trained based on the gradient value, so as to obtain the subjective quality prediction model.

[0237] In summary, this embodiment uses a subjective quality prediction model to obtain the coding quality corresponding to multiple candidate coding parameters for the current frame. This provides an evaluation of the data consistency between the original data and the reconstructed data before and after coding the current frame. The coding quality directly determines the target coding parameters that meet the quality requirements, improving the efficiency of obtaining a coding mode suitable for the current frame and increasing the compression efficiency of the coding process while ensuring coding quality. Furthermore, the evaluation criteria corresponding to the subjective quality prediction model reduce coding decision errors, and the current frame is encoded based on the coding mode indicated by the target coding parameters, improving the accuracy and stability of the coding process.

[0238] It should be noted that the encoding device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the encoding device provided in the above embodiments belongs to the same concept as the encoding method embodiments, and its specific implementation process can be found in the method embodiments, which will not be repeated here.

[0239] Please refer to Figure 13 This diagram illustrates a structural block diagram of an encoder provided in an exemplary embodiment of this application. The encoder may include one or more components such as a processor 1301 and a memory 1302. The components are coupled together via a bus system. It is understood that the bus system is used to enable communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0240] The memory 1302 is used to store a computer program that can run on the processor 1301; the processor 1301 is used to perform at least one of the following steps when running the computer program.

[0241] Obtain multiple candidate encoding parameters, which are used to indicate the encoding method for the current frame;

[0242] The subjective quality prediction model obtains the coding quality corresponding to multiple candidate coding parameters. The subjective quality prediction model is a machine learning model trained on sample image frames. The sample image frames are labeled with reference coding quality, which is labeled from a subjective perspective. The reference coding quality is used to express the data consistency before and after the sample image frame is encoded.

[0243] Obtain the target coding parameters that meet the quality requirements from multiple candidate coding parameters;

[0244] Outputs the bitstream data encoded for the current frame under the target encoding parameters.

[0245] It is understood that the memory 1302 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1302 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0246] The processor 1301 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above encoding method can be completed by the integrated logic circuitry in the hardware of the processor 1301 or by software instructions. The processor 1301 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1302. Processor 1301 reads information from memory 1302 and, in conjunction with its hardware, completes the steps of the above method. It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more ASICs, DSPs, digital signal processing devices (DSP Devices, DSPDs), programmable logic devices (PLDs), FPGAs, general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof. For software implementation, the technology described in this application can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described in this application. Software code can be stored in memory and executed by the processor. The memory can be implemented in the processor or external to the processor. Optionally, as another embodiment, processor 1301 is also configured to execute the encoding method described in any of the foregoing embodiments when running the computer program.

[0247] Please refer to Figure 14 This diagram illustrates a structural block diagram of a computer device 1400 provided in one embodiment of this application. The computer device 1400 may be... Figure 3 The terminal 320 or server 340 in the computer system shown is used to implement the encoding method provided in the above embodiments. Specifically, please refer to the following description.

[0248] The computer device 1400 may be a portable mobile terminal, also referred to as a mobile terminal in this embodiment. Examples include: smartphones, tablets, Moving Picture Experts Group Audio Layer III (MP3) players, and Moving Picture Experts Group Audio Layer IV (MP4) players. The computer device 1400 may also be referred to as a user device, portable terminal, or other names.

[0249] Typically, computer device 1400 includes a processor 1401 and a memory 1402.

[0250] Processor 1401 may include one or more processing cores, such as a 4-core processor or a 14-core processor. Processor 1401 may be implemented using at least one hardware form selected from DSP, FPGA, and Programmable Logic Array (PLA). Processor 1401 may also include a main processor and a coprocessor. The main processor, also known as a Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1401 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1401 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0251] The memory 1402 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 1402 may also include high-speed random access storage devices and non-volatile storage devices, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1402 are used to store at least one instruction, which is executed by the processor 1402 to implement the encoding methods provided in the various method embodiments of this application.

[0252] In some embodiments, the computer device 1400 may also optionally include: a peripheral device interface 1403 and at least one peripheral device.

[0253] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a bitstream thereon, and one or more computer programs and instructions, which, when executed by a processor, implement the steps of the encoding method as described in the above embodiments to generate a bitstream.

[0254] On the other hand, embodiments of this application provide a method for storing a bitstream, which involves generating a bitstream by performing the encoding method described in the above embodiments; and storing the bitstream.

[0255] On the other hand, embodiments of this application provide a method for transmitting a bitstream, comprising generating a bitstream by performing the encoding method as described in the above embodiments; and transmitting the bitstream.

[0256] On the other hand, embodiments of this application provide a computer program product, which includes computer instructions that are executed by a processor to implement the encoding method as described in the above embodiments.

[0257] On the other hand, embodiments of this application provide a computer device including the processor described above. Optionally, the processor is a GPU. The computer device can be at least one of a portable computer, a desktop computer, a server, a server cluster, an AI computing cluster, and a cloud computing cluster. The AI ​​computing cluster can also be simply referred to as an intelligent computing cluster or a smart computing cluster.

[0258] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0259] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An encoding method, characterized in that, The method includes: Obtain multiple candidate encoding parameters, which are used to indicate the encoding method for the current frame; Obtain the quality index data corresponding to the multiple candidate coding parameters, and the quality index data is used to characterize the data difference of the current frame before and after encoding; The coding quality corresponding to the quality index data is obtained by a subjective quality prediction model, and the coding quality corresponding to the multiple candidate coding parameters is obtained respectively. The subjective quality prediction model is a machine learning model trained by sample image frames. The sample image frames are labeled with reference coding quality. The reference coding quality is labeled from a subjective perspective and is used to express the data consistency before and after the encoding of the sample image frames. Obtain the target coding parameter whose coding quality meets the quality requirements from the plurality of candidate coding parameters; Output the bitstream data encoded for the current frame under the target encoding parameters.

2. The method according to claim 1, characterized in that, The step of obtaining the quality index data corresponding to the plurality of candidate coding parameters includes at least one of the following: The current frame is encoded based on the multiple candidate coding parameters to obtain the reconstructed data of the current frame corresponding to the multiple candidate coding parameters; the quality index data is obtained based on the reconstructed data. The quality index data corresponding to the multiple candidate coding parameters are obtained through a coding processing model, which is a pre-trained machine learning model used to express the correspondence between the candidate coding parameters and the quality index data.

3. The method according to claim 2, characterized in that, The step of encoding the current frame based on the plurality of candidate coding parameters to obtain the reconstructed data of the current frame corresponding to the plurality of candidate coding parameters includes: For each candidate coding parameter, obtain the reconstructed block corresponding to at least one coding block in the current frame; The process of obtaining the quality indicator data based on the reconstructed data includes: For each candidate coding parameter, at least one difference data between the coding block and the corresponding reconstruction block is obtained to obtain the difference data corresponding to the at least one coding block. The quality index data is obtained based on the difference data corresponding to the at least one coding block.

4. The method according to claim 1, characterized in that, The quality indicator data includes an n-dimensional vector, where the i-th element is the indicator data of the i-th dimension, and n is a positive integer, 0 < i ≤ n; The step of obtaining the quality index data corresponding to the plurality of candidate coding parameters includes at least one of the following: The texture distortion index is obtained as the first dimension index data, and the texture distortion index is used to characterize the accuracy loss of the current frame during the quantization process; The texture complexity index is obtained as the second dimension index data, and the texture complexity index is used to characterize the loss of image details before and after the current frame is encoded; The frame difference index is obtained as the third dimension index data. The frame difference index is used to characterize the pixel information loss of the current frame before and after encoding. The step of obtaining the coding quality corresponding to the quality index data through a subjective quality prediction model, and obtaining the coding quality corresponding to the multiple candidate coding parameters respectively, includes: The n-dimensional vector is input into the subjective quality prediction model, and the output is the coding quality corresponding to the multiple candidate coding parameters.

5. The method according to claim 1, characterized in that, The step of obtaining the target coding parameters that meet the quality requirements from the plurality of candidate coding parameters includes: At least one undetermined coding parameter is obtained from the plurality of candidate coding parameters, wherein the coding quality corresponding to the undetermined coding parameter meets the quality requirements; Obtain the bitstream size corresponding to at least one of the undetermined encoding parameters, wherein the bitstream size is the amount of data in the bitstream data encoded for the current frame under the undetermined encoding parameters; At least one of the undetermined encoding parameters is arranged in ascending order of bitstream size to obtain the first undetermined encoding parameter, which is then used as the target encoding parameter.

6. The method according to claim 5, characterized in that, The step of obtaining at least one undetermined encoding parameter from the plurality of candidate encoding parameters includes at least one of the following: The candidate encoding parameters whose encoding quality reaches a preset first threshold are obtained and used as the undetermined encoding parameters; The candidate encoding parameters whose encoding quality is within a preset quality set are obtained as the undetermined encoding parameters, wherein the quality set includes at least one encoding quality that meets the quality requirements, or the quality set is a value range corresponding to the quality requirements; According to the order of the encoding quality from largest to smallest, the first K candidate encoding parameters are obtained as at least one undetermined encoding parameter, where K is a positive integer.

7. The method according to claim 1, characterized in that, Before obtaining the coding quality corresponding to the quality index data through the subjective quality prediction model and obtaining the coding quality corresponding to the multiple candidate coding parameters, the method further includes: Obtain reference quality index data corresponding to the sample image frame, and obtain the reference coding quality labeled on the sample image frame, wherein there is a corresponding relationship between the reference quality index data and the reference coding quality; The subjective quality prediction model is trained based on the reference quality index data and the reference coding quality corresponding to the reference quality index data.

8. The method according to claim 7, characterized in that, The step of training the subjective quality prediction model based on the reference quality index data and the reference coding quality corresponding to the reference quality index data includes: Obtain the subjective quality prediction model to be trained; Based on the reference quality index data, the predicted coding quality is obtained through the subjective quality prediction model to be trained; Based on the predicted coding quality and the reference coding quality, a loss value is obtained, which is used to indicate the difference between the predicted coding quality and the reference coding quality; The subjective quality prediction model to be trained is obtained by training the model based on the loss value.

9. The method according to claim 8, characterized in that, The process of training the subjective quality prediction model based on the loss value to obtain the subjective quality prediction model includes: Based on the loss value, the gradient value corresponding to the subjective quality prediction model to be trained is obtained through the backpropagation algorithm; Based on the gradient value, the model parameters of the subjective quality prediction model to be trained are adjusted to obtain the subjective quality prediction model.

10. An encoding device, characterized in that, The device includes: The acquisition module is configured to: acquire multiple candidate encoding parameters, which indicate the encoding method for the current frame; acquire quality index data corresponding to each of the multiple candidate encoding parameters, which characterize the data difference of the current frame before and after encoding; acquire the encoding quality corresponding to the quality index data through a subjective quality prediction model, thereby obtaining the encoding quality corresponding to each of the multiple candidate encoding parameters, wherein the subjective quality prediction model is a machine learning model trained on sample image frames, the sample image frames are labeled with reference encoding quality, which is labeled from a subjective perspective and is used to express the data consistency of the sample image frame before and after encoding; and acquire the target encoding parameters whose encoding quality meets the quality requirements from the multiple candidate encoding parameters. The output module is configured to output the bitstream data encoded for the current frame under the target encoding parameters.

11. An encoder, characterized in that, The encoder includes a memory and a processor, the memory being used to store a computer program running on the processor; the processor being used to execute the steps of the encoding method as described in any one of claims 1 to 9 when running the computer program.

12. A computer-readable storage medium, characterized in that, It stores a bitstream thereon, and at least one of a computer program and instructions, which, when executed by a processor, implement the steps of the encoding method as described in any one of claims 1 to 9 to generate the bitstream.

13. A computer program product, characterized in that, The computer program product includes computer instructions that are executed by a processor to implement the steps of the encoding method as described in any one of claims 1 to 9.

14. A method for transmitting a code stream, characterized in that, The method includes: generating a bitstream by performing the steps of the encoding method according to any one of claims 1 to 9; and transmitting the bitstream.

Images