Image encoding and image compression method, apparatus, storage medium, and program product

Abstract

This application provides an image encoding and compression method, apparatus, storage medium, and program product. The method relates to the field of image processing. Before image encoding, it extracts the content features of the original image, obtains at least one distorted image of the original image, and extracts the content features and distortion behavior features of each distorted image. Using a prediction model, based on the quality target, the content features of the original image, and the content features and distortion behavior features of each distorted image, it predicts the quantization step size corresponding to the quality target. The original image is then encoded according to the corresponding quantization step size. By fitting the relationship between the quantization behavior quantization step size and the quality index through the prediction model, the corresponding quantization step size can be inversely derived from the quality target of the encoding. Furthermore, the image encoding quality can be precisely controlled according to the corresponding quantization step size, improving the subjective quality of the image encoding result and ensuring that the image encoding result meets the given quality target.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to an image encoding and image compression method, apparatus, storage medium, and program product. Background Technology

[0002] In many scenarios of image and video services, it is necessary to encode images and require the encoded image results to meet given quality requirements.

[0003] Current image coding methods rely on fixed empirical quality and quantization parameters, which cannot control the image coding quality, resulting in poor subjective quality of the image coding results and failure to meet the given quality requirements. Summary of the Invention

[0004] This application provides an image encoding and image compression method, apparatus, storage medium, and program product to solve the problem of poor subjective quality of image encoding results that do not meet quality requirements.

[0005] In a first aspect, this application provides an image encoding method, including:

[0006] Obtain the original image to be encoded and the quality target, and extract the content features of the original image;

[0007] Obtain at least one distorted image of the original image, and extract the content features and distortion behavior features of each distorted image;

[0008] Using a prediction model, based on the quality target, the content features of the original image, the content features and distortion behavior features of each of the distorted images, the quantization step size corresponding to the quality target is predicted.

[0009] The original image is encoded according to the quantization step size to obtain an encoding result that meets the quality target.

[0010] Secondly, this application provides an image compression method, comprising:

[0011] Obtain the original image to be compressed and the quality target, and extract the content features of the original image;

[0012] Acquire at least one distorted image of the original image, and determine the content features and distortion behavior features of each distorted image;

[0013] Using a prediction model, based on the quality target, the content features of the original image, the content features and distortion behavior features of each of the distorted images, the quantization step size corresponding to the quality target is predicted.

[0014] The original image is encoded according to the quantization step size to obtain a compressed image that meets the quality target.

[0015] Thirdly, this application provides a computing device, including: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein the computer programs / instructions, when executed by the processor, implement the methods provided in any of the foregoing aspects.

[0016] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method provided in any of the foregoing aspects.

[0017] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods provided in any of the foregoing aspects.

[0018] The image encoding and compression method, apparatus, storage medium, and program products provided in this application involve extracting the content features of the original image to be encoded before image encoding, obtaining at least one distorted image of the original image, and extracting the content features and distortion behavior features of each distorted image. A prediction model is used to predict the quantization step size corresponding to the quality target based on the quality target, the content features of the original image, the content features of each distorted image, and the distortion behavior features. The original image is then encoded according to the quantization step size corresponding to the quality target. The relationship between the quantization behavior quantization step size and quality indicators (such as VMAF) can be fitted through the prediction model. The corresponding quantization step size is then derived from the quality target of the encoding, and the image encoding quality is precisely controlled according to the corresponding quantization step size, thereby improving the subjective quality of the image encoding result and ensuring that the image encoding result meets the given quality target. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 This is an architectural diagram of an exemplary image encoding / compression system to which this application applies;

[0021] Figure 2 A flowchart illustrating an exemplary embodiment of this application;

[0022] Figure 3 A schematic diagram of the architecture of a prediction model provided for an exemplary embodiment of this application;

[0023] Figure 4A flowchart illustrating the training of a prediction model provided in an exemplary embodiment of this application;

[0024] Figure 5 A flowchart illustrating the image encoding process provided in an exemplary embodiment of this application;

[0025] Figure 6 A flowchart of an image compression method provided as an exemplary embodiment of this application;

[0026] Figure 7 This is a structural block diagram of a computing device according to an embodiment of this application.

[0027] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0030] First, let me explain the terms used in this application:

[0031] Quality control: During the image encoding / compression process, the image quality is controlled after lossy compression by adaptively optimizing the encoder parameters, that is, accurately achieving the preset quality target.

[0032] VMAF (Video Multimethod Assessment Fusion) is a video quality assessment tool used to predict how human viewers perceive video quality. The design philosophy of VMAF is to integrate multiple quality assessment metrics, including the Detail Loss Metric (DLM), Visual Information Fidelity (VIF), and inter-frame motion intensity. These metrics are learned from a large dataset using a Support Vector Machine (SVM) and then fused together to provide a single, comprehensive video quality score. This score ranges from 0 to 100, with higher scores indicating better video quality and a closer approximation to human perception. When calculating the VMAF of a video, the VMAF of a single frame is first calculated using the DLM and VIF of that frame, and then the overall VMAF is synthesized using inter-frame motion intensity information. Therefore, VMAF can be used for subjective quality assessment of images.

[0033] Image complexity, also known as image content complexity, refers to the complexity of an image in the spatial and frequency domains. It directly affects the efficiency of the encoder in processing images and the corresponding coding quality. Spatial domain complexity refers to the complexity of image texture, while frequency domain complexity refers to the distribution and intensity of different components of the image after transformation to the frequency domain.

[0034] Pre-analysis: In image and video coding, most encoder parameters are fixed based on statistical information and may not be optimal for the current coding sample. Especially when achieving a specific coding goal, it is necessary to consider the characteristics of the current sample. Pre-analysis coding refers to performing necessary coding steps or sample analysis operations and collecting statistical information before actual coding to support the selection of optimal parameters during actual coding.

[0035] Two-pass coding: In image and video coding, the coding result may deviate from the predetermined target. This result can serve as a reference to guide the improvement of coding parameters. The second coding iteration, using the reference result for optimization, is called two-pass coding. An example is two-pass bitrate control in video coding.

[0036] Quantization Step (Qstep): This parameter controls the quantization intensity of the encoder. The core of achieving high compression rates with an encoder is lossy compression, which in turn involves quantizing image or video data. A higher quantization intensity results in a lower bitrate after compression, but also higher compression distortion. Conversely, a lower quantization intensity leads to higher bitrate.

[0037] Quantization Parameter (QP): QP is an external parameter that can be input into the encoder. The encoder converts it to Qstep according to a predefined conversion formula, thereby changing the quantization intensity. QP is an integer parameter, typically ranging from 0 to 51 or 0 to 63. Generally, a larger QP value indicates a longer quantization step size, a lower compression rate, and higher compression distortion.

[0038] Quantization Scaling (Qscaling): QP is an integer parameter with relatively coarse control granularity. Qscaling is also an integer parameter with a wider range of values, which allows for fine adjustment of the quantization step size after conversion, enabling fine-tuning of quantization behavior.

[0039] Multi-Layer Perceptron (MLP): A fundamental neural network architecture in deep learning, belonging to the category of Feedforward Neural Networks (FNNs). An MLP consists of multiple layers, including an input layer, one or more hidden layers, and an output layer. Each layer comprises multiple neurons (or nodes), and information is passed between layers via weighted connections. Each neuron typically applies a non-linear activation function to the input information.

[0040] Traditional video encoding / compression processes primarily focus on improving objective compression performance, typically encoding based on fixed empirical quality and quantization parameters. This lack of control over image encoding quality results in poor subjective quality of the encoded images, failing to meet given quality requirements.

[0041] This application provides an image encoding method that uses a predictive model to fit the relationship between a quantization step size Qstep and a subjective quality metric (such as VMAF). Before image encoding, the original image is pre-analyzed to extract its content features and obtain at least one distorted image corresponding to the original image. The content features and distortion behavior features of each distorted image are then extracted. Using the predictive model, based on a given quality target, the content features of the original image to be encoded, and the content features and distortion behavior features of each distorted image, the quantization step size Qstep corresponding to the quality target is predicted. The original image is then encoded based on this quantization step size Qstep. The corresponding quantization step size Qstep can be inversely derived from the quality target of the encoding, and the image encoding quality can be precisely controlled based on the quantization step size Qstep, thereby improving the subjective quality of the image encoding result and ensuring that the image encoding result meets the given quality target.

[0042] Figure 1An architecture diagram of an exemplary image encoding / compression system applicable to this application is shown below. Figure 1 As shown, the system architecture may specifically include servers and end-side devices.

[0043] Specifically, the server can be a locally located computing device or a server / server cluster located in the cloud. The server has communication links with each end-device, enabling communication between the server and each end-device. The server stores prediction models and is responsible for encoding / compressing the images.

[0044] The edge device can specifically be a device used to perform image-related tasks. It can be a hardware device with network communication, computing, and information display capabilities, including but not limited to smartphones, tablets, desktop computers, IoT devices, and servers. During the execution of image-related tasks, the edge device needs to encode or compress the image. The edge device sends the original image to be encoded / compressed, along with a given quality target, to the server.

[0045] The server receiving device receives the original image to be encoded / compressed and the quality target, extracts the content features of the original image, obtains at least one distorted image corresponding to the original image, and extracts the content features and distortion behavior features of each distorted image. Using a prediction model, based on the quality target, the content features of the original image, and the content features and distortion behavior features of each distorted image, it predicts the quantization step size corresponding to the quality target. The original image is then encoded according to the quantization step size to obtain an encoding / compression result that meets the quality target.

[0046] Furthermore, the server returns the image encoding / compression result to the end device. The end device receives the image encoding / compression result returned by the server and performs subsequent image processing based on the image encoding / compression result.

[0047] It should be noted that in practical applications, the quality target of image encoding / compression can be specified by the customer through the end-side device, or it can be a quality target uniformly configured by the server within the image encoding / compression system. This embodiment does not make any specific limitations here.

[0048] The method in this embodiment can be applied to the encoding and compression of image data such as pictures and videos. For example, in a smart storage scenario, images / videos to be stored can be compressed. When encoding / compressing a video, each image frame contained in the video can be encoded / compressed separately. This embodiment can be specifically applied to the encoding or compression of images in various scenarios, and no specific application scenario is limited here.

[0049] For example, during the cloud service image transcoding process, it is necessary to control the quality of the transcoded image, especially its subjective quality, to ensure the user's subjective experience when using the image subsequently. The image transcoding process involves image encoding. When encoding an image, the image encoding method provided in this application is used to obtain the image to be encoded and the quality target of the encoded image, and to extract the content features of the original image; at least one distorted image corresponding to the original image is obtained, and the content features and distortion behavior features of each distorted image are extracted; a prediction model is used to predict the quantization step size corresponding to the quality target based on the quality target, the content features of the original image, and the content features and distortion behavior features of each distorted image; the original image is encoded according to the quantization step size to obtain an image encoding result that meets the quality target. The server stores the image encoding result that meets the quality target.

[0050] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0051] Figure 2 This is a flowchart illustrating an exemplary embodiment of the image encoding method provided in this application. The executing entity in this embodiment is the server in the aforementioned system architecture, used to encode images such as image frames in pictures and videos based on determined quality targets, obtaining encoding results that meet the quality targets. Figure 2 As shown, the specific steps of this method are as follows:

[0052] Step S201: Obtain the original image to be encoded and the quality target.

[0053] The original image to be encoded can be an image frame from a picture or video, and can be image data of various formats, without specific limitations here.

[0054] A quality objective refers to the target quality that the encoded image needs to achieve; it represents the client's or system's requirements for the quality of the encoded image (i.e., the image encoding result). The quality objective can be a quality threshold for subjective quality metrics such as VMAF or other quality indicators. The specific threshold can be set according to actual application needs and is not limited here. If the quality of the encoded image is higher than the quality threshold, it indicates that the encoded image meets the quality objective. The quality objective can be a uniform quality objective set by the image encoding server or customized by the client according to their individual needs.

[0055] For example, a client can send an image encoding request to the server via a client-side device. The image encoding request includes the original image to be encoded and the quality target given by the client. The server receives the original image to be encoded and the quality target given by the client from the client-side device. The client-side device can be a mobile terminal or a server running other services, such as an image compression and storage server or a video compression server.

[0056] Step S202: Extract the content features of the original image.

[0057] In this step, frequency domain features and / or spatial domain features of the original image are extracted as content features of the original image.

[0058] Optionally, when extracting the spatial features of the original image, the gray-level co-occurrence matrix (GLCM) of the original image can be calculated, and the texture features of the GLCM can be used as the spatial features of the original image.

[0059] For example, co-occurrence matrices (GLCMs) with co-occurrence distances of 1, 3, and 5 are calculated for the original image. Spatial texture features of the GLCMs, such as energy, entropy, homogeneity, correlation, and contrast, are then statistically analyzed and used as spatial features of the original image. The co-occurrence distance and direction of the calculated GLCMs can be set according to actual application requirements and are not specifically limited here.

[0060] In addition, when extracting spatial features of the original image, other arbitrary methods for extracting spatial texture features of the image, such as feature extractors or image transformations, can also be used, without specific limitations here.

[0061] Optionally, when extracting the frequency domain features of the original image, an image frequency domain feature extraction method based on Discrete Cosine Transform (DCT) can be used. Specifically, the original image is divided into multiple pixel blocks of a specified single line (e.g., 8×8), which serve as the basic unit of the DCT transformation. A two-dimensional DCT transformation is performed on each pixel block to convert the spatial domain information of the image into frequency domain information. The coefficients in the frequency domain represent the frequency components of the image. After the DCT transformation, most of the image's energy is concentrated on the low-frequency coefficients, which contain the main information of the image. The high-frequency coefficients carry relatively less information and usually contain details that are difficult for the human eye to perceive. Furthermore, based on the DCT values ​​of each pixel block, the average energy within each pixel block is calculated as the frequency domain feature of the image.

[0062] In addition, when extracting the frequency domain features of the original image, image frequency domain feature extraction methods based on wavelet transform, power spectral density (PSD) and other methods can also be used. This embodiment does not make specific limitations here.

[0063] Step S203: Obtain at least one distorted image corresponding to the original image, and extract the content features and distortion behavior features of each distorted image.

[0064] In this step, before actually encoding the original image, the original image is pre-analyzed and encoded using at least one preset quantization parameter QP to obtain at least one distorted image corresponding to the original image. The content features and distortion behavior features of each distorted image are then obtained to provide data for optimizing the encoder's encoding parameters.

[0065] Specifically, based on at least one preset specified QP, the original image is encoded using an encoder configured with the specified QP to obtain the distorted image corresponding to the specified QP. The number and specific value of the specified QP can be set according to actual application needs and experience, and are not specifically limited here.

[0066] For example, three specified QPs are preset: QP1, QP2, and QP3. QP1 is input to the encoder to set the encoder's QP to QP1. The original image is then encoded using the encoder with QP1 set, yielding the distorted image corresponding to QP1, i.e., the distorted image encoded using the QP1 encoding parameter. Similarly, the distorted images corresponding to QP2 and QP3 can be obtained, resulting in distorted images corresponding to the three specified QPs.

[0067] The number of QPs can be configured and adjusted according to actual application needs, and no specific limit is set here.

[0068] Furthermore, the frequency domain features and / or spatial domain features of the distorted image corresponding to each specified QP are extracted to obtain the content features of the distorted image corresponding to each specified QP. The method for extracting the content features of the distorted image is the same as the method for extracting the content features of the original image in step S202 above, and will not be repeated here.

[0069] Furthermore, the distortion behavior features of each distorted image are extracted.

[0070] For example, the Structure Similarity Index Measure (SSIM) between the distorted image and the original image corresponding to each specified QP is calculated as a distortion behavior feature of the distorted image corresponding to each specified QP. SSIM is an index used to measure the degree of structural similarity between two images. It is obtained by calculating the similarity of brightness, contrast, and structure between the two images, and then weighting and summing these three similarities to obtain the final similarity index. SSIM considers information from three aspects: brightness, contrast, and structure, and can more accurately reflect the human visual system's perception of image quality. The SSIM of one image and the original image can be implemented using any existing algorithm for calculating the SSIM between two images; no specific limitation is made here.

[0071] In addition, when extracting the distortion behavior features of the distorted image, methods such as calculating the mean squared error (MSE) and peak signal-to-noise ratio (PSNR) can be used to calculate the distortion description information / distortion behavior information between the distorted image and the original image, which can be used as the distortion behavior features of the distorted image. This embodiment does not make specific limitations here.

[0072] Step S204: Using a prediction model, predict the quantization step size corresponding to the quality target based on the quality target, the content features of the original image, the content features of each distorted image, and the distortion behavior features.

[0073] The prediction model is a pre-trained neural network model that accurately fits the relationship between the quality index and the quantization step size Qstep.

[0074] In this embodiment, by statistically analyzing the relationship between encoder quantization behavior and image subjective quality (such as VMAF) on a large number of data samples, a linear relationship was found: (100-VMAF) = a×Qstep+b, which fits the relationship between quantization behavior Qstep and subjective quality VMAF for more than 95% of the samples. By training the prediction model on a large number of data samples, the prediction model can accurately fit the relationship between quality indicators (such as VMAF) and quantization step size Qstep.

[0075] In this step, the quality target, the content features of the original image, the content features of each distorted image, and the distortion behavior features are input into the prediction model for inference and prediction, and the quantization step size Qstep corresponding to the quality target can be obtained.

[0076] For example, taking the quality target as the target VMAF as an example. In this step, the target VMAF, the content features of the original image, the content features and distortion behavior features of the distorted image corresponding to each specified QP are input into the prediction model. The prediction model performs the following three stages of inference prediction to obtain the quantization step size Qstep corresponding to the target VMAF:

[0077] Phase 1: Based on the content features of the original image, the content features and distortion behavior features of the distorted image corresponding to each specified QP, predict the detail loss index (DLM) and visual information fidelity (VIF) corresponding to each specified QP.

[0078] The second stage: Based on the content features of the original image and the DLM and VIF corresponding to each specified QP, predict the VMAF value corresponding to each specified QP.

[0079] The third stage: Based on the content features of the original image, the target VMAF, and the VMAF values ​​corresponding to each specified QP, predict the quantization step size Qstep corresponding to the target VMAF.

[0080] Optionally, the prediction model may include three prediction networks, each responsible for performing the predictions in the three stages described above. The first prediction network predicts the DLM and VIF corresponding to each specified QP based on the content features of the original image, the content features of the distorted image corresponding to each specified QP, and the distortion behavior features. The second prediction network predicts the VMAF value corresponding to each specified QP based on the content features of the original image, and the DLM and VIF corresponding to each specified QP. The third prediction network predicts the quantization step size corresponding to the quality target based on the content features of the original image, the target VMAF, and the VMAF value corresponding to each specified QP. The first, second, and third prediction networks can all be implemented using a multilayer perceptron (MLP) or other neural networks with similar capabilities; no specific limitations are imposed here.

[0081] In another alternative embodiment, the prediction model can use a thought chain-based deep learning model, which instructs the deep learning model to perform inference predictions according to the following steps: First, based on the content features of the input original image, the content features and distortion behavior features of the distorted image corresponding to each specified QP, predict the DLM and VIF of the distorted image corresponding to each specified QP, referred to as the DLM and VIF corresponding to each specified QP. Then, based on the content features of the original image, the DLM and VIF corresponding to each specified QP, predict the VMAF value of the distorted image corresponding to each specified QP, referred to as the VMAF value corresponding to each specified QP. Finally, based on the content features of the original image, the target VMAF, and the VMAF value corresponding to each specified QP, predict the quantization step size Qstep corresponding to the target VMAF.

[0082] By performing inference predictions in the three stages mentioned above using a predictive model, and obtaining the quantization step size Qstep corresponding to the target VMAF through multi-step inference prediction, the relationship between the quality index (such as VMAF) and the quantization step size Qstep can be more accurately fitted, thereby improving the prediction accuracy of the quantization step size Qstep corresponding to the target VMAF.

[0083] In other embodiments, the prediction model can use a deep learning model that does not use a thought chain. After training on a large number of data samples, the prediction model can accurately fit the relationship between the quality index and the quantization step Qstep. It can also obtain the quantization step Qstep corresponding to the quality target through one inference prediction based on the target VMAF, the content features of the original image, the content features and distortion behavior features of the distorted image corresponding to each specified QP.

[0084] Step S205: Encode the original image according to the quantization step size corresponding to the quality target to obtain the encoding result that meets the quality target.

[0085] After predicting the quantization step size Qstep corresponding to the quality target, the original image is encoded according to the quantization step size to obtain the encoding result that meets the quality target.

[0086] In this step, the encoder's encoding parameters are optimized based on the quantization step size (Qstep), and the optimized encoder is used to encode the original image to obtain an encoding result that meets the quality target. By optimizing the encoder's encoding parameters according to the quantization step size (Qstep) corresponding to the quality target, the quantization behavior of the encoder can be precisely controlled, improving the encoder's encoding quality and ensuring that the encoding result meets the aforementioned quality target.

[0087] Specifically, based on the quantization step size corresponding to the quality target and the correspondence between the quantization step size and the encoding parameters, the encoding parameter value corresponding to the quantization step size is determined, thus obtaining the encoding parameter value corresponding to the quality target; based on the encoding parameter value corresponding to the quality target, the encoding parameters of the encoder are optimized to obtain the optimized encoder.

[0088] Optionally, when optimizing the encoder's encoding parameters based on the encoding parameter values ​​corresponding to the quality target, the current encoder's encoding parameters can be set to the encoding parameter values ​​corresponding to the quality target to obtain the optimized encoder.

[0089] In this embodiment, the encoder used for image encoding can be any type of image encoder. In practical applications, some encoders include quantization parameters QP, and the correspondence between quantization parameters QP and quantization step size Qstep is fixed and known. Other encoders include quantization parameters QP and quantization scaling Qscaling, and the correspondence between quantization parameters QP and quantization step size Qstep is also fixed and known.

[0090] For example, in a scenario where the encoder's encoding parameters include the quantization parameter QP, the QP value corresponding to the quantization step size corresponding to the quality target is determined based on the predicted quantization step size corresponding to the quality target and the correspondence between the quantization step size and QP. This yields the QP value corresponding to the quality target. The QP value corresponding to the quality target is then input into the encoder to set the encoder's QP to the value corresponding to the quality target, resulting in an optimized encoder.

[0091] For example, in a scenario where the encoder's encoding parameters include quantization parameter QP and quantization scaling Qscaling, the QP and Qscaling values ​​corresponding to the quantization step size corresponding to the quality target are determined based on the predicted quantization step size corresponding to the quality target and the correspondence between the quantization step size and QP and Qscaling. The QP and Qscaling values ​​corresponding to the quality target are then input into the encoder to set the encoder's QP and Qscaling to the values ​​corresponding to the quality target, resulting in an optimized encoder.

[0092] In one optional embodiment, the encoder's encoding parameters are optimized based on the encoding parameter values ​​corresponding to the quality target to obtain the optimized encoder. This can be achieved in the following way:

[0093] The current encoder's encoding parameters are set to values ​​corresponding to the quality target, resulting in a modified encoder. The original image is then encoded using this modified encoder to obtain the encoded image. The VMAF value of the encoded image is calculated. This yields the VMAF value of the encoded result of the original image based on the encoding parameter values ​​corresponding to the quality target. Further, a fourth prediction network predicts the correction amount for the encoding parameter values ​​based on the content features of the original image, the quality target, the VMAF values ​​of each distorted image, the encoding parameter values ​​corresponding to the quality target, and the VMAF value of the encoded image. The correction amount is then added to the encoding parameter values ​​to obtain the corrected encoding parameter values. The fourth prediction network can be implemented using an MLP or other neural networks with similar capabilities; no specific limitations are specified here.

[0094] This process is called Two-pass. Two-pass sets the encoding parameters of the current encoder to the corrected encoding parameter values, resulting in an optimized encoder. This makes the VMAF value of the encoding result of the original image using the optimized encoder closer to the target VMAF.

[0095] For example, when the difference (absolute value of the difference) between the VMAF value of the obtained encoded image and the target VMAF is greater than a preset threshold (such as 1, 2, etc.), the fourth prediction network predicts the correction amount of the QP value based on the content features of the original image, the target VMAF, the VMAF values ​​of each distorted image, the QP value corresponding to the quality target, and the VMAF value of the encoded image. The correction amount is added to the QP value corresponding to the quality target to obtain the corrected QP value. The QP of the current encoder is set to the corrected QP value to obtain the optimized encoder, which allows for the correction of the encoder's QP.

[0096] In scenarios where the encoder's encoding parameters include QP and Qscaling, the QP value corresponding to the quality target can be corrected, while the Qscaling value corresponding to the quality target remains unchanged. Alternatively, the QP value and Qscaling value corresponding to the quality target can be corrected separately. The parameters of the prediction network used for correcting the QP value and Qscaling value separately can be different and obtained by training on different datasets.

[0097] In this embodiment, considering that VMAF has a more flexible trend and range of values ​​compared to other objective quality indicators (such as PSNR), and is more sensitive to changes in encoding parameters, when the actual VMAF value of the encoded image differs significantly from the target VMAF, the predicted encoding parameter values ​​corresponding to the quality target are corrected by the fourth prediction network. This makes the corrected encoding parameter values ​​more accurate, improves the robustness of encoding quality control, and thus enables more precise VMAF control of the encoder.

[0098] In this embodiment, before image encoding, the original image is pre-analyzed to extract content features of the original image to be encoded, and at least one distorted image of the original image is obtained. The content features and distortion behavior features of each distorted image are extracted. A prediction model is used to predict the quantization step size corresponding to the quality target based on the quality target, the content features of the original image, the content features of each distorted image, and the distortion behavior features. The original image is encoded according to the quantization step size Qstep corresponding to the quality target. The relationship between the quantization behavior Qstep and subjective quality indicators (such as VMAF) can be fitted by the prediction model. The corresponding quantization step size Qstep is obtained by back-calculation based on the quality target of the encoding. The image encoding quality is precisely controlled according to the quantization step size Qstep, thereby improving the subjective quality of the image encoding result and ensuring that the image encoding result meets the given quality target.

[0099] Figure 3 This is a schematic diagram of the architecture of a prediction model provided for an exemplary embodiment of this application. Figure 3 The image shows an example architecture of a prediction model, such as Figure 3 As shown, the prediction model includes a first prediction network, a second prediction network, and a third prediction network.

[0100] The first prediction network is used to predict the DLM and VIF corresponding to each specified QP based on the content features of the original image, the content features of the distorted image corresponding to each specified QP, and the distorted behavior features.

[0101] Specifically, the first prediction network includes a first prediction module and a second prediction module. The first prediction module is used to predict the DLM corresponding to each specified QP based on the content features of the original image, the content features of the distorted image corresponding to each specified QP, and the distortion behavior features.

[0102] The second prediction module is used to predict the VIF corresponding to each specified QP based on the content features of the original image, the content features of the distorted image corresponding to each specified QP, and the distortion behavior features.

[0103] The first prediction module and the second prediction module can be implemented using a multilayer perceptron (MLP) or other neural networks with similar capabilities. The first prediction module and the second prediction module can be isomorphic, but they do not share parameters.

[0104] The second prediction network is used to predict the VMAF value corresponding to each specified QP based on the content features of the original image and the DLM and VIF corresponding to each specified QP. The second prediction network can be implemented using a multilayer perceptron (MLP) or other neural networks with similar capabilities.

[0105] The third prediction network is used to predict the quantization step size corresponding to the target VMAF based on the content features of the original image, the target VMAF, and the VMAF values ​​corresponding to each specified QP. The third prediction network can be implemented using a multilayer perceptron (MLP) or other neural networks with similar capabilities.

[0106] In this embodiment, when the first prediction module and the second prediction module, the second prediction network, and the third prediction network in the first prediction network are all implemented using MLP, these MLPs are isomorphic but do not share parameters.

[0107] based on Figure 3 The prediction model shown, in step S206 above, uses the prediction model to predict the quantization step size corresponding to the quality target based on the quality target, the content features of the original image, the content features of each distorted image, and the distortion behavior features. Specifically, this can be achieved in the following way:

[0108] The content features of the original image, the content features of the distorted image corresponding to each specified QP, and the distortion behavior features are input into the first prediction network. The first prediction network predicts the DLM and VIF corresponding to each specified QP based on the content features of the original image, the content features of the distorted image corresponding to each specified QP, and the distortion behavior features.

[0109] The content features of the original image, the DLM and VIF corresponding to each specified QP predicted by the first prediction network, are input into the second prediction network. The second prediction network then predicts the VMAF value corresponding to each specified QP based on the content features of the original image and the DLM and VIF corresponding to each specified QP.

[0110] The content features of the original image, the target VMAF, and the VMAF values ​​corresponding to each specified QP predicted by the second prediction network are input into the third prediction network. The third prediction network then predicts the quantization step size corresponding to the quality target based on the content features of the original image, the target VMAF, and the VMAF values ​​corresponding to each specified QP.

[0111] Figure 4 A flowchart illustrating the training of a prediction model provided for an exemplary embodiment of this application. Figure 4 As shown, the training process for the prediction model is as follows:

[0112] Step S401: Construct the training dataset.

[0113] In this embodiment, in order to meet the training needs of the prediction model, the training data in the constructed training dataset includes: content features of the original image sample, content features and distortion behavior features of the distortion image sample corresponding to at least one specified QP, target VMAF sample, DLM ground truth and VIF ground truth corresponding to each specified QP, VMAF ground truth corresponding to each specified QP, and quantization step size ground truth corresponding to the target VMAF sample.

[0114] Here, the original image sample refers to the sample of the original image before encoding.

[0115] The distorted image sample corresponding to any specified QP refers to the distorted image obtained by encoding the original image sample using an encoder with the specified QP set.

[0116] The target VMAF sample refers to the target VMAF encoded from the original image sample.

[0117] The DLM ground truth value for each specified QP refers to the DLM ground truth value of the distorted image corresponding to that specified QP. The VIF ground truth value for each specified QP refers to the VIF ground truth value of the distorted image corresponding to that specified QP. The VMAF ground truth value for each specified QP refers to the VMAF ground truth value of the distorted image corresponding to that specified QP.

[0118] The true value of the quantization step size corresponding to the target VMAF sample refers to the true value of the encoder's quantization step size Qstep when the encoding result of the original image sample satisfies the target VMAF sample. It can be calculated based on the encoder's encoding parameters (including QP or including QP and Qscaling) when the encoding result of the original image sample satisfies the target VMAF sample.

[0119] In this step, a large number of original image samples and target VMAF samples are collected in the image encoding scene. Based on at least one preset specified QP, the original image samples are encoded using an encoder with the specified QP set, thereby obtaining at least one distorted image sample corresponding to the specified QP.

[0120] Extract the content features of the original image samples, as well as the content features and distortion behavior features of the distorted image samples corresponding to each specified QP. For the specific implementation principle, please refer to the relevant content of steps S202 and S203 in the aforementioned embodiment, which will not be repeated here.

[0121] Based on the original image sample and the distorted image sample corresponding to each specified QP, calculate the DLM value, VIF value and VMAF value of the distorted image corresponding to each specified QP, and obtain the DLM ground truth value, VIF ground truth value and VMAF ground truth value corresponding to each specified QP.

[0122] The ground truth value of the quantization step size corresponding to the target VMAF sample can be obtained through manual annotation.

[0123] Based on the content features of the original image samples obtained above, the content features and distortion behavior features of the distortion image samples corresponding to at least one specified QP, the target VMAF samples, the DLM ground truth and VIF ground truth corresponding to each specified QP, the VMAF ground truth corresponding to each specified QP, and the quantization step size ground truth corresponding to the target VMAF samples, a training dataset is constructed.

[0124] Furthermore, the prediction model is trained in the following three stages using the training dataset.

[0125] In the first stage, steps S402-S403 are used to train the first prediction network in the prediction model. DLM and VIF are the metrics used to calculate VMAF. Through the training in the first stage, the first prediction network can fit the values ​​and trends of the two metrics, DLM and VIF, providing a data foundation for the subsequent prediction in the second stage.

[0126] Step S402: Input the content features of the original image sample, the content features and distortion behavior features of the distorted image sample corresponding to each specified QP into the first prediction network for prediction, and obtain the DLM prediction value and VIF prediction value corresponding to each specified QP.

[0127] For example, the first prediction network may include a first prediction module and a second prediction module, which are two relatively independent MLPs (or other neural networks with similar capabilities).

[0128] In the first stage of training, the content features of the original image samples, and the content features and distortion behavior features of the distorted image samples corresponding to each specified QP are input into the first prediction module and the second prediction module, respectively. The first prediction module predicts the DLM prediction value for each specified QP based on the content features of the original image samples, and the content features and distortion behavior features of the distorted image samples corresponding to each specified QP. The second prediction module predicts the VIF prediction value for each specified QP based on the content features of the original image samples, and the content features and distortion behavior features of the distorted image samples corresponding to each specified QP.

[0129] Step S403: Adjust the parameters of the first prediction network according to the DLM prediction value and VIF prediction value corresponding to each specified QP, as well as the DLM ground truth value and VIF ground truth value corresponding to each specified QP.

[0130] In this step, the first cross-entropy loss function value is calculated based on the DLM prediction value and the DLM ground truth value corresponding to each specified QP, thus obtaining the DML prediction loss. Based on the DML prediction loss, the parameters of the first prediction module are updated through backpropagation.

[0131] Based on the VIF predicted values ​​and true VIF values ​​corresponding to each specified QP, the second cross-entropy loss function value is calculated to obtain the VIF prediction loss. The parameters of the second prediction module are then updated via backpropagation based on the VIF prediction loss.

[0132] It should be noted that the training strategy used in the first stage of training the first prediction network (including the first prediction module and the second prediction module), including the learning rate and optimization algorithm, can be designed and adjusted according to the actual application requirements, and no specific limitations are made here.

[0133] In the second stage, through steps S404-S405, the second prediction network in the prediction model is trained, so that the second prediction network can learn the process of calculating VMAF based on DLM and VIF through support vector machine SVM.

[0134] Step S404: Input the content features of the original image sample, the DLM prediction value and VIF prediction value corresponding to each specified QP into the second prediction network for prediction, and obtain the VMAF prediction value corresponding to each specified QP.

[0135] In the second stage of training, based on the DLM and VIF prediction values ​​corresponding to each specified QP output by the first prediction network in the first stage of training, as well as the content features of the original image sample, the VMAF corresponding to each specified QP is predicted by the second prediction network to obtain the VMAF prediction value corresponding to each specified QP.

[0136] Step S405: Adjust the parameters of the second prediction network according to the VMAF predicted value and the VMAF true value corresponding to each specified QP.

[0137] Specifically, based on the VMAF predicted values ​​and the VMAF ground truth values ​​corresponding to each specified QP, the third cross-entropy loss function value is calculated to obtain the VMAF prediction loss. Based on the VMAF prediction loss, the parameters of the second prediction network are updated through backpropagation to obtain the trained second prediction network.

[0138] It should be noted that the training strategy used in the second stage for training the second prediction network, including the learning rate and optimization algorithm, can be designed and adjusted according to the actual application requirements, and no specific limitations are made here.

[0139] In the third stage, through steps S406-S407, the third prediction network in the prediction model is trained, so that the third prediction network, under at least one specified QP and the corresponding VMAF of the given sample, implicitly fits the relationship between VMAF and Qstep with reference to the content features of the original image sample, thereby obtaining the ability to predict the Qstep corresponding to the target VMAF sample.

[0140] Step S406: Input the content features of the original image, the target VMAF sample, and the VMAF prediction value corresponding to each specified QP into the third prediction network for prediction to obtain the quantization step size prediction value.

[0141] In the third stage of training, based on the VMAF prediction values ​​corresponding to each specified QP output by the second prediction network in the second stage of training, the quantization step size Qstep corresponding to the target VMAF sample is predicted by combining the content features of the original image and the target VMAF sample, thus obtaining the quantization step size prediction value.

[0142] Step S407: Adjust the parameters of the third prediction network based on the predicted quantization step size and the true quantization step size.

[0143] Specifically, based on the predicted and true quantization step sizes, the fourth cross-entropy loss function is calculated to obtain the quantization step size prediction loss. Based on this loss, the parameters of the third prediction network are updated via backpropagation to obtain the trained third prediction network.

[0144] It should be noted that the training strategy used in the third stage of training the third prediction network, including the learning rate and optimization algorithm, can be designed and adjusted according to the actual application requirements, and no specific limitations are made here.

[0145] In this embodiment, the first prediction module and second prediction module, the second prediction network, and the third prediction network of the first prediction network in the prediction model are implemented using MLPs. The training of the prediction model is carried out in three stages, with the ground truth values ​​of DLM, VIF, VMAF, and Qstep used to supervise the regression predictions of each MLP, respectively. During inference, the model is directly and completely forward-propagated.

[0146] The scheme in this embodiment, by training the prediction model in three stages, can ensure the performance of each prediction network in the training prediction model. By using the output of the first prediction network in the first stage of training as the input of the second prediction network in the second stage of training, and using the output of the second prediction network in the second stage of training as the input of the third prediction network in the third stage of training, the correlation and cooperation between the first and second prediction networks, and between the second and third prediction networks, can be improved, thereby improving the overall performance of the prediction network.

[0147] In an optional embodiment, a fourth prediction network is used to predict the correction amount of the encoded parameter value corresponding to the quality target. Exemplarily, the fourth prediction network can be a classification prediction network, where different categories correspond to different correction amounts for the encoded parameters. For example, QP is an integer parameter, and several optional QP correction amounts can be set as follows: -3, -2, -1, 1, 2, 3, with different QP correction amounts representing different categories. The fourth prediction network performs classification prediction on the QP correction amount, and the corresponding QP correction amount can be determined based on the classification prediction result.

[0148] The fourth prediction network can be implemented using an MLP or other neural networks with similar capabilities; no specific limitations are specified here. The training data used for training the fourth prediction network includes: content features of the original image samples, the target VMAF, the VMAF values ​​of the distorted images corresponding to each specified QP, the encoding parameter values ​​corresponding to the target VMAF, the VMAF values ​​of the encoded images obtained by encoding the original image samples based on the encoding parameter values ​​corresponding to the target VMAF, and the labeled encoding parameter corrections. The labeled encoding parameter corrections can be obtained through manual annotation.

[0149] The training process of the fourth prediction network includes:

[0150] The content features of the original image sample, the target VMAF, the VMAF values ​​of the distorted image corresponding to each specified QP, the encoding parameter values ​​corresponding to the target VMAF, and the VMAF values ​​of the encoded image obtained by encoding the original image sample based on the encoding parameter values ​​corresponding to the target VMAF are input into the fourth prediction network for inference and prediction, and the correction amount of the encoding parameter values ​​is output. Based on the correction amount of the encoding parameter values ​​and the labeled encoding parameter correction amount, the parameters of the fourth prediction network are adjusted to obtain the trained fourth prediction network.

[0151] For example, the fifth cross-entropy loss function value is calculated based on the correction amount of the encoding parameter values ​​and the correction amount of the labeled encoding parameters to obtain the corrected prediction loss. Based on the corrected prediction loss, the parameters of the fourth prediction network are adjusted through backpropagation.

[0152] It should be noted that the training strategy used to train the fourth prediction network, including the learning rate and optimization algorithm, can be designed and adjusted according to the actual application requirements, and no specific limitations are made here.

[0153] In this embodiment, training the fourth prediction network can improve its performance, thereby improving the accuracy of the prediction coding parameter correction.

[0154] Figure 5This is a flowchart illustrating an exemplary embodiment of the image encoding process provided in this application. The solution of this application, through statistical analysis of the relationship between encoder quantization behavior and subjective image quality (such as VMAF) on a large number of data samples, discovers a linear relationship: (100-VMAF) = a×Qstep+b. This linear relationship fits the relationship between quantization behavior Qstep and subjective quality VMAF for more than 95% of the samples.

[0155] By training a prediction model on a large number of data samples, the model can accurately fit the relationship between quality metrics (such as VMAF) and quantization step size Qstep. The prediction model can then be used to predict the corresponding quantization step size Qstep based on the target VMAF.

[0156] like Figure 5 As shown, the overall process of image coding based on prediction models includes two parts: one-pass and two-pass.

[0157] One-pass section:

[0158] Feature extraction is performed on the original image to obtain its content features F1. Using at least one specified QP (the figure uses QP values ​​22, 27, and 32 as examples), the original image is encoded using encoders with three specified QP values, obtaining distorted images corresponding to each specified QP. Feature extraction is performed on the distorted images corresponding to each specified QP to obtain their content features. For example, F22, F27, and F32 in the figure represent the content features of the distorted images corresponding to QP values ​​of 22, 27, and 32, respectively. Distortion behavior features of the distorted images corresponding to each specified QP are then extracted.

[0159] Furthermore, we will make predictions in the following three stages:

[0160] Phase 1: Input the content features F1 of the original image, the content features (F22, F27 and F32) of the distorted image corresponding to each specified QP, and the distortion behavior features of the distorted image corresponding to each specified QP into the first prediction network of the prediction model for prediction, and obtain the DLM and VIF corresponding to each specified QP.

[0161] Second stage: Input the content features F1 of the original image, the DLM and VIF corresponding to each specified QP into the second prediction network of the prediction model to make predictions and obtain the VMAF value corresponding to each specified QP.

[0162] Three stages: The content features of the original image are analyzed using F1 sorting, and the target VMAF (as shown in the figure) is then processed. tThe VMAF value corresponding to each specified QP is input into the third prediction network of the prediction model for prediction, and the Qstep corresponding to the target VMAF is obtained.

[0163] Furthermore, based on the Qstep corresponding to the target VMAF, the QP value and Qscaling value corresponding to the target VMAF are determined.

[0164] Set the encoder's QP and Qscaling to the QP and Qscaling values ​​corresponding to the target VMAF to obtain the optimized encoder. Use the optimized encoder to encode the original image to obtain the encoded image.

[0165] Calculate the actual VMAF value of the encoded image (as shown in the figure). a The actual VMAF value of the encoded image (VMAF) a ) and target VMAF (VMAF) t If the absolute value of the difference between the two values ​​is greater than 1, the encoder optimized in the first round is used as the encoder to be corrected, and a two-pass process is performed. This is based on the actual VMAF value of the encoded image (VMAF). a ) and target VMAF (VMAF) t If the absolute value of the difference between the original image and the original image is not greater than 1, the encoded image is used as the encoding result of the original image.

[0166] Two-pass section:

[0167] The actual VMAF value of the encoded image (VMAF) a ), target VMAF (VMAF) t The QP value corresponding to the target VMAF, the content features F1 of the original image, and the DLM and VIF corresponding to each specified QP are input into the fourth prediction network for prediction to obtain the correction amount of the QP value corresponding to the target VMAF (as shown in the figure). bias ).

[0168] Add the correction amount QP to the QP value corresponding to the target VMAF. bias The corrected QP value is obtained (QP as shown in the figure). refined ).

[0169] Furthermore, the QP of the encoder to be corrected is set to the corrected QP value to obtain the second optimized encoder. The original image is then encoded using the second optimized encoder to obtain the final encoding result.

[0170] The solution in this application discovers the linear relationship between Qstep and VMAF. Based on this, a prediction model is used to fit the linear relationship between Qstep and VMAF, enabling the prediction of the corresponding Qstep based on the target VMAF. The encoding parameters corresponding to Qstep (including QP or including QP and Qscaling) are used to control the image encoding process, accurately adjusting the encoder's quantization behavior and achieving control over the encoded VMAF. Furthermore, considering that VMAF, as a fusion feature, needs to reference multiple basic indicators (i.e., DLM and VIF for the image), a three-stage progressive method for predicting the Qstep corresponding to the target VMAF is proposed. First, DLM and VIF are predicted. Then, based on the predicted DLM and VIF, subsequent VMAF and Qstep predictions are performed, effectively improving the control accuracy of predicting the Qstep corresponding to the target VMAF.

[0171] Figure 6 A flowchart illustrating an exemplary embodiment of this application is provided. Figure 6 As shown, the specific steps of this method are as follows:

[0172] Step S601: Obtain the original image to be compressed and the quality target.

[0173] The original image to be compressed can be a picture or a frame from a video, and can be image data of various formats, without any specific limitations here.

[0174] A quality objective refers to the target quality that the compressed image needs to achieve; it represents the client's or system's requirements for the quality of the compressed image (i.e., the image compression result). The quality objective can be a quality threshold for subjective quality metrics such as VMAF or other quality indicators. The specific threshold can be set according to actual application needs and is not limited here. If the quality of the compressed image is higher than the quality threshold, it indicates that the compressed image meets the quality objective. The quality objective can be a uniform quality objective set by the image compression server or it can be customized by the client according to their individual needs.

[0175] For example, in an image compression and storage scenario, in response to receiving an image compression and storage instruction, the service obtains the image to be stored and the image compression quality target, and uses the image to be stored as the original image to be compressed.

[0176] Step S602: Extract the content features of the original image.

[0177] The implementation principle of this step is the same as that of the aforementioned step S202. For details, please refer to the relevant content of the aforementioned embodiments, which will not be repeated here.

[0178] Step S603: Obtain at least one distorted image of the original image, and determine the content features and distortion behavior features of each distorted image.

[0179] The implementation principle of this step is the same as that of the aforementioned step S203. For details, please refer to the relevant content of the aforementioned embodiments, which will not be repeated here.

[0180] Step S604: Using a prediction model, predict the quantization step size corresponding to the quality target based on the quality target, the content features of the original image, the content features of each distorted image, and the distortion behavior features.

[0181] The implementation principle of this step is the same as that of the aforementioned step S204. For details, please refer to the relevant content of the aforementioned embodiments, which will not be repeated here.

[0182] Step S605: Encode the original image according to the quantization step size to obtain a compressed image that meets the quality target.

[0183] The implementation principle of this step is the same as that of the aforementioned step S205. For details, please refer to the relevant content of the aforementioned embodiments, which will not be repeated here.

[0184] Step S606: Encode the original image using the optimized encoder to obtain a compressed image that meets the quality target.

[0185] In this embodiment, image compression is achieved by encoding the original image using an optimized encoder.

[0186] After obtaining a compressed image that meets the quality target, the server can store the compressed image that meets the quality target, thus realizing the image compression and storage function.

[0187] In this embodiment, the image encoding scheme is applied to the image compression scenario. For the specific implementation principle and technical effects, please refer to the content of the foregoing embodiments, which will not be repeated here.

[0188] In this embodiment, before image compression, the original image is pre-analyzed to extract its content features and obtain at least one distorted image. The content features and distortion behavior features of each distorted image are then extracted. A prediction model is used to predict the quantization step size corresponding to the quality target based on the quality objective, the content features of the original image, and the content and distortion behavior features of each distorted image. Based on the quantization step size Qstep corresponding to the quality objective, the encoder's coding parameters are optimized. The relationship between the quantization behavior Qstep and subjective quality indicators (such as VMAF) can be fitted using the prediction model. The corresponding quantization step size Qstep is then derived from the quality objective, and the encoder's coding parameters are precisely controlled based on the quantization step size Qstep. This allows for precise control of the image compression quality, improving the subjective quality of the image compression result and ensuring that the image compression result meets the given quality objective.

[0189] Figure 7 This is a structural block diagram of a computing device according to an embodiment of this application. Figure 7 As shown, the computing device may include one or more (only one is shown in the figure) processors 701 and memory 702. The memory 702 stores computer programs / instructions, and the processor 701 executes the computer programs / instructions. When the computer programs / instructions are executed by the processor 701, they implement the technical solutions provided in any of the aforementioned method embodiments. Their specific functions and the technical effects they can achieve are similar and will not be repeated here.

[0190] The aforementioned computing device can be understood as an integrated smart terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), all-in-one model machines, mobile phones, tablet computers, or other portable smart terminals, and the computing device may have the model in the above embodiments of this application pre-installed.

[0191] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided to manage and monitor the applications.

[0192] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0193] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method of any of the foregoing embodiments. The specific functions and technical effects to be achieved are not described here.

[0194] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the foregoing embodiments. The computer program is stored in a readable storage medium, and at least one processor of a computing device can read the computer program from the readable storage medium. The execution of the computer program by the at least one processor causes the computing device to perform the technical solutions provided in any of the above method embodiments. Specific functions and achievable technical effects are not elaborated here.

[0195] This application provides a chip, including a processing module and a communication interface. The processing module is capable of executing the technical solutions of the aforementioned method embodiments. Optionally, the chip further includes a storage module (e.g., a memory), which stores instructions. The processing module executes the instructions stored in the storage module, and the execution of the instructions stored in the storage module causes the processing module to execute the technical solutions provided in any of the aforementioned method embodiments.

[0196] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0197] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules in at least one processor.

[0198] The memory may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0199] The aforementioned storage device can be object storage service (OSS).

[0200] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0201] The aforementioned communication components are configured to facilitate wired or wireless communication between the device containing the communication components and other devices. The device containing the communication components can access wireless networks based on communication standards, such as mobile hotspots (WiFi), second-generation (2G), third-generation (3G), fourth-generation (4G) / Long Term Evolution (LTE), fifth-generation (5G), or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID), infrared, Ultra Wide Band (UWB), Bluetooth, and other technologies.

[0202] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0203] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0204] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components within an electronic device or host device.

[0205] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0206] The order of the embodiments described above is merely for illustrative purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. "Multiple" means two or more, unless otherwise explicitly specified.

[0207] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0208] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0209] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image encoding method, characterized in that, include: Obtain the original image to be encoded and the quality target, and extract the content features of the original image; Obtain at least one distorted image of the original image, and extract the content features and distortion behavior features of each distorted image; Using a prediction model, based on the quality target, the content features of the original image, the content features and distortion behavior features of each of the distorted images, the quantization step size corresponding to the quality target is predicted. The original image is encoded according to the quantization step size to obtain an encoding result that meets the quality target.

2. The method according to claim 1, characterized in that, The step of encoding the original image according to the quantization step size to obtain an encoding result that meets the quality target includes: Based on the quantization step size, optimize the encoder's encoding parameters; The original image is encoded using an optimized encoder to obtain an encoding result that meets the quality target.

3. The method according to claim 1, characterized in that, The extraction of content features from the original image includes: Extract the frequency domain features and / or spatial domain features of the original image to obtain the content features of the original image.

4. The method according to claim 1, characterized in that, The step of acquiring at least one distorted image of the original image and determining the distortion behavior features and content features of each distorted image includes: Based on at least one preset specified quantization parameter QP, the original image is encoded using an encoder with the specified QP set to obtain a distorted image corresponding to the specified QP. Extract the frequency domain features and / or spatial domain features of the distorted image corresponding to each specified QP to obtain the content features of the distorted image corresponding to each specified QP; Calculate the structural similarity index between the distorted image and the original image corresponding to each specified QP, and use it as the distortion behavior feature of the distorted image corresponding to each specified QP.

5. The method according to claim 1, characterized in that, The quality objective is a target VMAF, the distorted image includes at least one distorted image corresponding to a specified QP, and the prediction model includes a first prediction network, a second prediction network, and a third prediction network. The method of using a prediction model to predict the quantization step size corresponding to the quality target, based on the quality target, the content features of the original image, the content features and distortion behavior features of each of the distorted images, includes: The first prediction network predicts the loss of detail (DLM) and visual information fidelity (VIF) corresponding to each specified QP based on the content features of the original image, the content features of the distorted image corresponding to each specified QP, and the distortion behavior features. The second prediction network predicts the VMAF value corresponding to each specified QP based on the content features of the original image and the DLM and VIF corresponding to each specified QP. The third prediction network predicts the quantization step size corresponding to the target VMAF based on the content features of the original image, the target VMAF, and the VMAF value corresponding to each specified QP.

6. The method according to claim 5, characterized in that, The first prediction network includes a first prediction module and a second prediction module. The step of predicting the loss of detail (DLM) and visual information fidelity (VIF) corresponding to each specified QP through the first prediction network, based on the content features of the original image, the content features and distortion behavior features of the distorted image corresponding to each specified QP, includes: The first prediction module predicts the loss of detail (DLM) corresponding to each specified QP based on the content features of the original image, the content features of the distorted image corresponding to each specified QP, and the distortion behavior features. The second prediction module predicts the visual information fidelity (VIF) corresponding to each specified QP based on the content features of the original image, the content features of the distorted image corresponding to each specified QP, and the distortion behavior features.

7. The method according to claim 5, characterized in that, Also includes: Construct a training dataset, which includes the content features of the original image samples, the content features and distortion behavior features of the distortion image samples corresponding to the at least one specified QP, the target VMAF sample, the DLM ground truth and VIF ground truth corresponding to each specified QP, the VMAF ground truth corresponding to each specified QP, and the quantization step size ground truth corresponding to the target VMAF sample. The prediction model is trained using the training dataset.

8. The method according to claim 7, characterized in that, The step of training the prediction model using the training dataset includes: The content features of the original image sample, the content features of the distorted image sample corresponding to each specified QP, and the distortion behavior features are input into the first prediction network for prediction, so as to obtain the DLM prediction value and VIF prediction value corresponding to each specified QP. The parameters of the first prediction network are adjusted based on the DLM prediction value and VIF prediction value corresponding to each specified QP, as well as the DLM ground truth value and VIF ground truth value corresponding to each specified QP.

9. The method according to claim 8, characterized in that, Also includes: The content features of the original image sample, the DLM prediction value and the VIF prediction value corresponding to each specified QP are input into the second prediction network for prediction to obtain the VMAF prediction value corresponding to each specified QP. The parameters of the second prediction network are adjusted based on the VMAF predicted values ​​and the VMAF true values ​​corresponding to each specified QP.

10. The method according to claim 9, characterized in that, Also includes: The content features of the original image, the target VMAF sample, and the VMAF prediction values ​​corresponding to each specified QP are input into the third prediction network for prediction to obtain the quantization step size prediction value. The parameters of the third prediction network are adjusted based on the predicted quantization step size and the true quantization step size.

11. The method according to any one of claims 1-10, characterized in that, The step of optimizing the encoder's encoding parameters based on the quantization step size includes: Based on the quantization step size and the correspondence between the quantization step size and the encoding parameters, the encoding parameter value corresponding to the quantization step size is determined. Based on the encoding parameter values, the encoding parameters of the encoder are optimized to obtain an optimized encoder.

12. The method according to claim 11, characterized in that, The step of determining the encoding parameter value corresponding to the quantization step size based on the quantization step size and the correspondence between the quantization step size and the encoding parameters includes: Based on the quantization step size and the correspondence between the quantization step size and QP, determine the QP value corresponding to the quantization step size; or, Based on the quantization step size and the correspondence between the quantization step size, QP, and quantization scaling, the QP value and quantization scaling corresponding to the quantization step size are determined.

13. The method according to claim 11, characterized in that, The step of optimizing the encoder parameters based on the encoding parameter values ​​to obtain an optimized encoder includes: The encoder's encoding parameters are set to the encoding parameter values ​​to obtain the encoder to be corrected; The original image is encoded using the encoder to be corrected to obtain an encoded image; Calculate the VMAF value of the encoded image; The fourth prediction network predicts the correction amount of the coding parameter value based on the content features of the original image, the quality target, the VMAF value of each distorted image, the coding parameter value, and the VMAF value of the coded image. Add the correction amount to the encoding parameter value to obtain the corrected encoding parameter value; The encoder's encoding parameters are set to the corrected encoding parameter values ​​to obtain the optimized encoder.

14. An image compression method, characterized in that, include: Obtain the original image to be compressed and the quality target, and extract the content features of the original image; Acquire at least one distorted image of the original image, and determine the content features and distortion behavior features of each distorted image; Using a prediction model, based on the quality target, the content features of the original image, the content features and distortion behavior features of each of the distorted images, the quantization step size corresponding to the quality target is predicted. The original image is encoded according to the quantization step size to obtain a compressed image that meets the quality target.

15. The method according to claim 14, characterized in that, The process of obtaining the original image to be compressed and the quality target includes: In response to an image compression and storage command, the image to be stored and the image compression quality target are obtained; The image to be stored is used as the original image to be compressed; After obtaining the compressed image that meets the quality target, the process further includes: Store compressed images that meet the stated quality targets.

16. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein when the computer programs / instructions are executed by the processor, they implement the method according to any one of claims 1-15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-15.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-15.

Images