Encoding method, decoding method, system, electronic device and storage medium

Abstract

Embodiments of the present application provide an encoding method, a decoding method, a system, an electronic device and a storage medium. The encoding method comprises: extracting image frequency features of a source image in at least two frequency domains; selecting features in the image frequency features that match a quality level of the source image in terms of importance as to-be-encoded features, the to-be-encoded features being necessary feature representations required when encoding based on the quality level; and encoding the to-be-encoded features according to a code rate that matches the image frequency features, to obtain a code stream corresponding to the image frequency features, the code rate being proportional to a frequency component corresponding to the image frequency features. The technical solution of the embodiments of the present application can adaptively select to-be-encoded features according to the quality of a source image for different frequency components of the source image, and encode using different code rates for different frequency components, which is not only versatile and flexible.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an encoding method, a decoding method, a system, an electronic device, and a storage medium. Background Technology

[0002] Artificial intelligence (AI) encoding and decoding models encode and decode images through neural networks, and AI encoding and decoding models can be obtained through pre-training.

[0003] Generally, an AI codec model only supports encoding and decoding for a fixed bitrate. If it is applied to encoding and decoding scenarios with multiple bitrates, a common approach is to deploy an AI codec model and multiple model weights as an AI codec system. The AI ​​codec system changes the encoding and decoding bitrate of the AI ​​codec model by adjusting the weights used in the AI ​​codec model.

[0004] It is evident that conventional AI encoding and decoding systems have poor versatility and flexibility. Furthermore, deploying multiple model weights requires more storage resources and increases the complexity and computational cost of the AI ​​encoding and decoding system. Summary of the Invention

[0005] To overcome the problems existing in related technologies, embodiments of this application provide an encoding method, a decoding method, a system, an electronic device, and a storage medium.

[0006] According to a first aspect of the embodiments of this application, an encoding method is provided, applied to an AI encoding model, the method comprising:

[0007] Extract image frequency features from at least two frequency domains of the source image;

[0008] The features selected from the image frequency features whose importance matches the quality level of the source image are used as the features to be encoded. The features to be encoded refer to the necessary feature representations required for encoding based on the quality level.

[0009] The feature to be encoded is encoded according to a bitrate that matches the image frequency feature to obtain a bitstream corresponding to the image frequency feature, wherein the bitrate is proportional to the frequency component corresponding to the image frequency feature.

[0010] According to a second aspect of the embodiments of this application, a decoding method is provided, applied to an AI decoding model, the method comprising:

[0011] At least two bitstreams are acquired, and the at least two bitstreams correspond one-to-one with at least two image frequency features in the frequency domain; each bitstream is obtained by encoding a feature whose importance in the corresponding image frequency feature matches the quality level of the source image, and the feature to be encoded refers to the necessary feature representation required for encoding based on the quality level;

[0012] For any bitstream, the bitstream is decoded according to a bitrate that matches the bitstream to obtain decoding features. The bitrate that matches the bitstream refers to the bitrate that matches the image frequency features corresponding to the bitstream. The bitrate is proportional to the frequency components corresponding to the image frequency features.

[0013] After obtaining at least two decoding features, the at least two decoding features are fused to obtain the reconstructed image of the source image, wherein the at least two decoding features correspond one-to-one with the at least two bitstreams.

[0014] According to a third aspect of the embodiments of this application, an encoding and decoding system is provided, the system comprising: an artificial intelligence (AI) encoding model and an AI decoding model, wherein...

[0015] The AI ​​encoding model is used to extract image frequency features from at least two frequency domains of the source image; select features among the image frequency features whose importance matches the quality level of the source image as features to be encoded, wherein the features to be encoded refer to the necessary feature representations required for encoding based on the quality level; encode the features to be encoded according to a bitrate matching the image frequency features to obtain the bitstream corresponding to the image frequency features, wherein the bitrate is proportional to the frequency components corresponding to the image frequency features.

[0016] The AI ​​decoding model is used to decode any bitstream according to a bitrate that matches the bitstream to obtain decoding features. The bitrate that matches the bitstream refers to the bitrate that matches the image frequency features corresponding to the bitstream. The at least two decoding features obtained by the at least two decoding modules are fused to obtain the reconstructed image of the source image. The at least two decoding features correspond one-to-one with the at least two bitstreams.

[0017] According to a fourth aspect of the embodiments of this application, an encoding apparatus is provided, the apparatus comprising:

[0018] The extraction unit is used to extract image frequency features in at least two frequency domains from the source image;

[0019] The feature selection unit is used to select features among the image frequency features whose importance matches the quality level of the source image as features to be encoded. The features to be encoded refer to the necessary feature representations required for encoding based on the quality level.

[0020] An encoding unit is used to encode the feature to be encoded according to a bit rate that matches the image frequency feature, so as to obtain a bit stream corresponding to the image frequency feature, wherein the bit rate is proportional to the frequency component corresponding to the image frequency feature.

[0021] According to a fifth aspect of the embodiments of this application, a decoding apparatus is provided, the apparatus comprising:

[0022] An acquisition unit is used to acquire at least two bitstreams, wherein the at least two bitstreams correspond one-to-one with at least two frequency domain image frequency features; each bitstream is obtained by encoding a feature whose importance in the corresponding image frequency feature matches the quality level of the source image, and the feature to be encoded refers to the necessary feature representation required for encoding based on the quality level;

[0023] A decoding unit is used to decode any given bitstream according to a bitrate that matches the bitstream to obtain decoding features. The bitrate that matches the bitstream refers to the bitrate that matches the image frequency features corresponding to the bitstream. The bitrate is proportional to the frequency components corresponding to the image frequency features.

[0024] A fusion unit is used to fuse at least two decoded features after obtaining them to obtain a reconstructed image of the source image, wherein the at least two decoded features correspond one-to-one with the at least two bitstreams.

[0025] According to a sixth aspect of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being executed by the processor to cause the electronic device to perform the method as described in the first or second aspect.

[0026] According to a seventh aspect of the present application, a computer-readable storage medium is provided having a computer program stored thereon, the program being executed by a processor to implement the method as described in the first or second aspect.

[0027] According to an eighth aspect of the embodiments of this application, a computer program product is provided, including instructions that, when executed on a computer, cause the computer to perform the method as described in the first or second aspect.

[0028] The technical solutions provided in this application embodiment may include the following beneficial effects:

[0029] In the encoding stage of the source image, at least two frequency domain features of the source image are first extracted. This allows for differentiated processing of different frequency components of the source image, thereby effectively adjusting the bitrate. Furthermore, based on the image frequency features, features whose importance matches the quality level of the source image are selected from each image frequency feature as the features to be encoded. Specifically, for any given image frequency feature, the feature whose importance matches the quality level of the source image can refer to the necessary feature representation required for encoding based on the quality level. This reduces the amount of data encoded while maintaining high visual quality. Additionally, in this embodiment, the bitrate is proportional to the frequency domain corresponding to the image frequency feature. Therefore, for any image frequency feature to be encoded, it can be encoded at a bitrate matching the image frequency feature to obtain the corresponding bitstream. This facilitates variable bitrate encoding based on different frequency domain components of the source image, improving encoding efficiency while preserving the details of the source image. As can be seen, the technical solution of this application embodiment does not require the deployment of multiple model weights, thereby saving storage resources and reducing the complexity and computing power of the AI ​​encoding and decoding system. Moreover, the technical solution of this application embodiment can adaptively select the features to be encoded for different frequency components of the source image according to the quality of the source image, and use different bit rates for encoding for different frequency components, which is not only highly versatile and flexible.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly described below. It should be understood that those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0032] Figure 1 This application provides a typical data flow diagram for image encoding and decoding.

[0033] Figure 2 This application provides an exemplary system architecture diagram of an AI encoding / decoding system.

[0034] Figure 3 A schematic diagram of an exemplary method flow for an encoding method provided in an embodiment of this application;

[0035] Figure 4 This is an exemplary scenario diagram of image frequency division provided in the embodiments of this application;

[0036] Figure 5 An exemplary scenario diagram illustrating the relationship between channel parameters and image quality provided in this application embodiment;

[0037] Figure 6 A schematic diagram of an exemplary method flow for a decoding method provided in an embodiment of this application;

[0038] Figure 7 This is a schematic diagram of the encoding and decoding data flow in any frequency domain provided in the embodiments of this application;

[0039] Figure 8A This is an exemplary schematic diagram of the encoding device provided in the embodiments of this application;

[0040] Figure 8B This is an exemplary schematic diagram of the decoding apparatus provided in the embodiments of this application;

[0041] Figure 9 This is an exemplary structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings.

[0043] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments and is not intended to limit the technical solutions of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise.

[0044] It should also be understood that although the terms "first," "second," etc., may be used in the following embodiments to describe a class of objects, the objects are not limited to these terms. These terms are used to distinguish specific implementations of that class of objects.

[0045] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0046] The following describes the technical scenarios related to the embodiments of this application.

[0047] This application's embodiments relate to AI-based image encoding and decoding technology. For example... Figure 1 As shown, Figure 1 This diagram illustrates a typical data flow for image encoding and decoding. After acquiring the source image, it can be encoded to obtain a compressed bitstream. This bitstream is then decoded to obtain the reconstructed image of the source image. The encoding process can include quantization and entropy encoding, and can further include entropy decoding, dequantization, and reconstruction.

[0048] Those skilled in the art will understand that Figure 1 The image encoding and decoding process shown is for illustrative purposes only and does not limit the actual operation of image encoding and decoding. In real-world implementations, the image encoding and decoding process may include more advanced techniques. Figure 1 The procedures shown are not limited here.

[0049] Quantization refers to reducing the complexity of image data by converting continuous values ​​into discrete values. The quantization process can include dimensionality transformation and determining quantization parameters (such as quantization step size and quantization coefficients). Dimensionality transformation converts the image from the spatial domain to the frequency domain, obtaining its frequency domain representation y. This can be achieved using methods such as Discrete Cosine Transform (DCT) or Discrete Fourier Transform (DFT). Further, a quantization matrix can be selected or generated, defining the quantization step size QP (Quantization Parameter) for different frequency coefficients. Then, by dividing the dimensionally transformed coefficients (such as DCT coefficients) by the corresponding quantization matrix value, a quantization factor Vq is obtained. Normalizing the quantization factor approximates the original continuous coefficients as discrete values. A quantization algorithm can satisfy, for example, AdaQ(y) = Round(y / (QP*Vq)).

[0050] Entropy coding refers to encoding discrete values ​​obtained from quantization based on probability distribution to reduce redundancy in image data. The entropy coding process may include determining the frequency of each pixel value in the image data, and employing entropy coding algorithms such as Huffman coding, arithmetic coding, or run-length encoding. Based on the probability distribution of pixel values ​​in the image data, shorter codes are used for frequently occurring pixel values, while longer codes are used for less frequently occurring pixel values, resulting in an image bitstream (also called a bit stream) for further compression of the image data.

[0051] It's important to note that in AI-based image encoding and decoding systems, before entropy encoding, hyper-prior encoding and hyper-prior decoding can be used to encode the distribution parameters of the image data as auxiliary information for entropy encoding. Hyper-prior encoding extracts features from the image data and maps these features to a low-dimensional latent space to capture key semantic information of the image and the dependencies between different representations. Hyper-prior decoding reconstructs the latent space representation back into the image space. Then, based on the quality feedback of the reconstructed image, the encoding strategy of entropy encoding is adjusted to optimize the quality of the entropy-encoded image.

[0052] Entropy decoding is the inverse operation of entropy coding, used to reconstruct image data from a bitstream. It can include parsing entropy-coded symbols or codewords from the bitstream, using an entropy decoding algorithm (i.e., the inverse algorithm of entropy coding) to map the codewords back to the original symbols (such as pixel values), and reconstructing the probability distribution of the original symbols based on the context information from the entropy coding process.

[0053] Dequantization is the inverse operation of quantization. It can involve determining the quantization matrix after obtaining the quantization coefficients of the probability distribution from entropy decoding, and multiplying the quantization coefficients by the corresponding values ​​in the quantization matrix to map the quantization coefficients from the discrete levels of the quantization table back to the original continuous value range. Then, an inverse transform (such as inverse DCT) is used to restore the continuous value range to the spatial domain and reconstruct approximate pixel values ​​of the image. Finally, the inversely transformed pixel values ​​are recombine to form complete image data to obtain the reconstructed image.

[0054] although Figure 1 The encoding and decoding processes are described as a single system. However, in practical implementations, encoding functionality is typically integrated into the encoder, and decoding functionality into the decoder. The presence and (precise) division of encoder and decoder functionality may vary depending on the specific device and application. Optionally, the encoder and decoder can be located in the same electronic device; alternatively, they can be located in different electronic devices. The electronic devices described herein can include any category of handheld or stationary devices, such as laptops or notebook computers, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, cameras, in-vehicle devices, display devices, digital media players, video game consoles, video streaming devices (e.g., content service servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc.

[0055] like Figure 1The image encoding and decoding functions illustrated are implemented through an AI encoding and decoding model. The aforementioned functions, such as quantization, entropy coding, entropy decoding, dequantization, and reconstruction, can all be executed by algorithm modules. Before deployment, the model typically needs to be trained to enable these functions. A common implementation approach is to train an AI encoding and decoding model for a fixed bitrate image encoding and decoding, and this model only supports encoding and decoding at that fixed bitrate. If applied to encoding and decoding scenarios with multiple bitrates, one approach is to deploy multiple model weights, loading the corresponding weights onto the AI ​​encoding and decoding model for different bitrate tasks. It is evident that conventional AI encoding and decoding systems have poor versatility and flexibility. Deploying multiple model weights requires more storage resources, while loading model weights increases the complexity and computational cost of the AI ​​encoding and decoding system.

[0056] In view of this, before encoding the source image, this application extracts image frequency features from at least two frequency domains of the source image and determines the bit rate for different frequency components of the source image, such that the bit rate is proportional to the frequency component. During the encoding process, features whose importance matches the quality level of the source image are selected as features to be encoded, and encoding is performed according to the bit rate matching the image frequency features. In this way, there is no need to deploy multiple model weights, thereby saving storage resources and reducing the complexity and computing power of the AI ​​encoding and decoding system. Furthermore, the technical solution of this application can adaptively select features to be encoded according to different frequency components of the source image, and use different bit rates for encoding different frequency components, which is not only highly versatile but also flexible.

[0057] The system architecture involved in the embodiments of this application is described below.

[0058] See Figure 2 , Figure 2This application illustrates an AI encoding / decoding system provided in an embodiment. The system may include an AI encoding model and an AI decoding model. The AI ​​encoding model includes a frequency division module 1000, a first encoding module 2000-1, and an x-th encoding module 2000-2. The AI ​​decoding model includes a first decoding module 3000-1, an x-th decoding module 3000-2, a fusion module 4000, and a reconstruction module 5000. In this embodiment, x can be an integer greater than or equal to 2. Specifically, the frequency division module 1000, the first encoding module 2000-1, the x-th encoding module 2000-2, the first decoding module 3000-1, the x-th decoding module 3000-2, the fusion module 4000, and the reconstruction module 5000 can be implemented as hardware components, software components, or a combination of both. For example, the frequency division module 1000, the first encoding module 2000-1, the xth encoding module 2000-2, the first decoding module 3000-1, the xth decoding module 3000-2, the fusion module 4000, and the reconstruction module 5000 can be deep learning-based algorithm modules.

[0059] It should be understood that Figure 2 The AI ​​encoding / decoding system shown is for illustrative purposes only and does not limit the AI ​​encoding / decoding system described in this application. In actual implementation scenarios, the AI ​​encoding / decoding system may also include... Figure 2 The diagram shows additional modules, such as a first hyperprior coding module and a first hyperprior decoding module. This is not a limitation.

[0060] The frequency division module 1000 can be used to divide the input source image into x different frequency components, and the x different frequency components can respectively characterize the features of x different frequency domains of the source image. The source image here can include any real-world image or video, or an image or video of a real object. In some embodiments, the frequency division module 1000 can be implemented as x convolutional modules, and the weights, number of channels, and convolutional kernels of these x convolutional modules can be different. Each convolutional module can be used to extract features from one frequency domain of the source image.

[0061] In this way, by performing frequency decomposition on the source image, image information can be analyzed and compressed from a frequency perspective, which can effectively reduce redundant information, improve coding efficiency, and help retain more image details and improve quality.

[0062] Any one of the first encoding modules 2000-1 to the xth encoding module 2000-2 can be used to encode the image frequency features in the corresponding frequency domain to obtain the bitstream of the image in the corresponding frequency domain. The encoding process of any encoding module for the corresponding image frequency features is similar to... Figure 1The illustrated embodiments are similar and may include quantization and entropy coding. The coding module may include a quantization module and an entropy coding module. Optionally, in the embodiments of this application, any coding module may further include a priori coding module, a priori decoding module, and a feature selection module. Before performing entropy coding, the image frequency features can be processed by the priori coding module and the priori decoding module to obtain mask features for filtering important features. Then, the feature selection module combines the mask features to filter out features for coding.

[0063] In some embodiments, for a source image, the encoding modules used for encoding the image frequency features of high-frequency components in the first encoding module 2000-1 to the xth encoding module 2000-2 have a higher encoding code rate, while the encoding modules used for encoding the image frequency features of low-frequency components have a lower encoding code rate. This enables variable code rate encoding and helps to preserve more details of the source image.

[0064] In other embodiments, for different source images, the AI ​​encoding and decoding system of this application embodiment can adaptively calculate the encoding rate of the first encoding module 2000-1 to the xth encoding module 2000-2 based on the quality level of the source image, which is not only highly versatile but also flexible.

[0065] Any encoding module may include quantization, entropy coding, super-prior coding, super-prior decoding, and feature selection modules, which can be algorithm modules, deep learning-based image processing networks, or models. These algorithm modules, deep learning-based image processing networks, or models can be combined in different ways to achieve the functionality of the encoding module.

[0066] Any one of the decoding modules 3000-1 to the xth decoding module 3000-2 can be used to decode the bitstream in the corresponding frequency domain to obtain the image frequency features in the corresponding frequency domain. Each decoding module may include an entropy decoding module and an inverse quantization module, used to perform entropy decoding and inverse quantization on the corresponding bitstream to obtain the image frequency features in the corresponding frequency domain. The decoding process of any decoding module is similar to... Figure 1 The illustrated embodiments are similar and will not be described in detail here.

[0067] It should be noted that for any given bitstream, if the decoding bitrate matches the bitrate used to encode the bitstream, then the decoding bitrate used by the decoding module can be the same as the encoded bitstream of the corresponding encoding module. For example, the encoding bitrate of the first encoding module 2000-1 is the same as the decoding bitrate of the first decoding module 3000-1.

[0068] The first decoding module 3000-1 to the xth decoding module 3000-2 respectively decode the image decoding features in the x frequency domains. In order to reconstruct the image, the fusion module 4000 can fuse the image decoding data in the x frequency domains to obtain complete image decoding data. Then, the reconstruction module 5000 can reconstruct an image that matches the source image based on the complete image decoding data.

[0069] exist Figure 2 In the illustrated AI encoding / decoding system environment, the embodiments of this application provide, as follows: Figure 3 The encoding method shown includes the following steps:

[0070] In step 101, image frequency features in at least two frequency domains of the source image are extracted.

[0071] In step 102, features whose importance in the image frequency features matches the quality level of the source image are selected as features to be encoded.

[0072] In step 103, the feature to be encoded is encoded according to a bitrate that matches the image frequency feature to obtain a bitstream corresponding to the image frequency feature, wherein the bitrate is proportional to the frequency component corresponding to the image frequency feature.

[0073] As can be seen, by adopting the embodiments of this application, at least two frequency domain image frequency features of the source image are first extracted, thereby enabling differentiated processing of different frequency components of the source image and effectively adjusting the bitrate. Furthermore, taking the image frequency features as the main body, features whose importance matches the quality level of the source image are selected from each image frequency feature as the features to be encoded for the corresponding image frequency feature. Specifically, for any image frequency feature, the feature whose importance matches the quality level of the source image can refer to the necessary feature representation required for encoding based on the quality level. This reduces the amount of data encoded and helps maintain high visual quality. In addition, in this embodiment, the bitrate is proportional to the frequency domain corresponding to the image frequency feature. Thus, for any image frequency feature to be encoded, it can be encoded according to the bitrate matching the image frequency feature to obtain the bitstream corresponding to that image frequency feature. This facilitates variable bitrate encoding based on different frequency domain components of the source image, improving encoding efficiency while preserving the details of the source image. As can be seen, the technical solution of this application embodiment does not require the deployment of multiple model weights, thereby saving storage resources and reducing the complexity and computing power of the AI ​​encoding and decoding system. Moreover, the technical solution of this application embodiment can adaptively select the features to be encoded for different frequency components of the source image according to the quality of the source image, and use different bit rates for encoding for different frequency components, which is not only highly versatile and flexible.

[0074] In some embodiments, the AI ​​encoding / decoding system can invoke a frequency division module to extract image frequency features in at least two frequency domains from the source image. During implementation, the frequency division module can extract image features from the source image, convert these image features into frequency domain features, and then extract at least two frequency domain image frequency features from the frequency domain features.

[0075] For example, the frequency division module can use algorithms such as DCT, DFT or wavelet transform to convert the image features into frequency domain features. Then, at least two convolutional modules are used to process the frequency domain features respectively, so that each convolutional module outputs an image frequency feature with one frequency component.

[0076] It should be understood that the embodiments of this application extract image frequency features from at least two frequency domains of the source image and perform encoding compression on a unit basis for each image frequency feature, aiming to balance image quality and compression effect. In some implementation scenarios, if image frequency features from two frequency domains are extracted, although the encoding compression effect is good, it will lead to poor image quality based on the encoded image. If image frequency features from four or more frequency domains are extracted, the image quality based on the encoded image can be maintained at a better level, but due to the large amount of data, the encoding compression effect is not good. In view of this, optionally, the image frequency features from at least two frequency domains in the embodiments of this application can be implemented as three image frequency features: high-frequency component image frequency features, mid-frequency component image frequency features, and low-frequency component image frequency features. The high-frequency component image frequency features represent the texture features of the source image, the mid-frequency component image frequency features represent the contour features of the source image, and the low-frequency component image frequency features represent the color distribution features of the source image.

[0077] For example, refer to Figure 4 The illustrated image frequency division scenario includes a frequency division module that can include convolutional module 1, convolutional module 2, and convolutional module 3. At least one of the following parameters—number of convolutional layers, weights, number of channels, and number of convolutional kernels—can be different for each of these modules. The frequency division module can extract image frequency features of high-frequency components from the source image using convolutional module 1, extract image frequency features of mid-frequency components using convolutional module 2, and extract image frequency features of low-frequency components using convolutional module 3. Figure 4 As shown, the higher the frequency domain, the richer the color information contained in the image frequency features of the corresponding frequency domain components.

[0078] As can be seen, the technical solution of this application, by performing frequency decomposition on the source image, supports the analysis and compression of image information from a frequency perspective, which can effectively reduce redundant information, not only improve coding efficiency, but also help retain more image details and improve quality.

[0079] Furthermore, the AI ​​encoding / decoding system can call at least two encoding modules to encode the aforementioned at least two image frequency features, wherein each of the at least two encoding modules corresponds one-to-one with the at least two image frequency features. The algorithm modules included in each encoding module, as well as the encoding process for the corresponding image frequency features, are similar. The following uses the encoding process of one encoding module for the corresponding image frequency features as an example to illustrate the image encoding process of this application embodiment.

[0080] Before introducing the encoding process, it should be noted that in the source image acquisition stage of this embodiment, the AI ​​encoding / decoding system can obtain the quality value (Q value) of the source image. Therefore, the AI ​​encoding / decoding system can determine the encoding bitrate of the source image based on the quality value. The Q value refers to the image quality level after reconstruction, which affects the compression quality and detail of the source image. A higher Q value indicates better source image quality, larger image data size, and a lower compression ratio; conversely, a lower Q value indicates worse source image quality, smaller image data size, and a higher compression ratio. Therefore, the AI ​​encoding / decoding system in this embodiment is not only highly versatile but also flexible.

[0081] Furthermore, for any image frequency feature, the encoding module corresponding to the image frequency feature can generate a mask feature of the image frequency feature based on the image frequency feature, the source image, and the quality level. Then, the feature corresponding to the non-mask feature value in the quantized features of the image frequency feature is determined as the feature to be encoded.

[0082] Masking features are used to filter out image frequency features whose importance matches the quality level of the source image. Features whose importance matches the quality level of the source image can refer to the necessary feature representations in the image frequency features that enable the source image to achieve the corresponding quality level. Masking features can include masked feature values ​​and non-masked feature values. Masked feature values ​​are used to identify image frequency features that do not need to be encoded, while non-masked feature values ​​are used to identify image frequency features that need to be encoded.

[0083] The encoding module corresponding to the image frequency features can extract the semantic representation of the corresponding frequency domain from the image frequency features. This semantic representation can be a compressed, low-dimensional spatial feature, which can contain the key information required to reconstruct the image. Further, based on the semantic representation and the image features of the source image, an initial importance feature is generated. Each feature value in the initial importance feature represents the degree of influence of the corresponding pixel on the semantics of the source image. Then, the feature values ​​in the initial importance feature can be adjusted according to the quality level to obtain the mask feature.

[0084] In some embodiments, the encoding module may include a pre-trained importance feature learning model, which can evaluate each feature in the image frequency features based on the semantic representation and the image features of the source image, thereby determining the degree of influence of each feature element in the image frequency features on the quality of the source image, and generating initial importance features based on the degree of influence of each feature element on the quality of the source image.

[0085] For example, semantic representations derived from image frequency features can represent the key information needed to reconstruct the image corresponding to the image frequency features. The importance feature learning model can extract image features from the source image, which may include structural, texture, and content features. Based on these image features, the importance feature learning model can determine the key image features needed to reconstruct the source image. Furthermore, it can assess the impact of each feature element in the image frequency features on the quality of the source image based on the similarity between the semantic representation and the key image features, and generate initial importance features.

[0086] The initial importance feature can be a three-dimensional (3D) tensor. The three dimensions can include position (x, y) and channel (ch). x refers to the coordinate along the width of the image, y refers to the coordinate along the height of the image, and channel refers to the color channel in the image. In a color image, each pixel can contain multiple color channels, and the values ​​of these channels collectively define the color of that point. For example, color channels can be red, green, and blue (RGB) channels, with each channel having a value between 0 and 255. Any feature value includes the position parameter and channel parameter of the corresponding pixel.

[0087] Understandably, the generation of initial importance features depends on the content of the source image. Different source images usually have different important content and structure, and the initial importance features corresponding to the same frequency domain image features of different source images can be different.

[0088] Based on the description of the initial importance features, it can be seen that the parameters specifically characterizing image quality can be the channel parameters of each element. For example, please refer to [reference needed]. Figure 5 , Figure 5 This diagram illustrates an exemplary scenario of the relationship between channel parameters and image quality. The more feature elements an image contains, and the richer the channel parameters of these feature elements, the larger the Q value of the image. In this case, the higher the visual quality of the image and the richer the details of the content displayed. Conversely, the fewer feature elements an image contains, and the fewer the channel parameters of these feature elements, the smaller the Q value of the image. In this case, the lower the visual quality of the image and the less details of the content displayed.

[0089] As can be seen, the implementation method of this application supports the capture of important features in the source image based on the content of the source image. On the one hand, it can provide more detailed features for compression and encoding, which is beneficial to improving the image quality. Even for low bitrate images, the reconstructed image can maintain high quality. On the other hand, it can filter out features that do not need to be encoded, thereby improving the compression efficiency.

[0090] The initial importance feature only identifies important and unimportant regions in image frequency features from the perspective of the importance of feature elements, but it cannot characterize the importance of each feature value in important regions. The importance of each feature value can be reflected by the weights of the channel parameters in the feature value.

[0091] Accordingly, adjusting each feature value in the initial importance feature according to the quality level to obtain the mask feature can be implemented as follows: the importance feature learning model calculates the weights of the channel parameters corresponding to each feature value according to the quality level, adjusts each feature value according to the corresponding weight to obtain the importance feature, binarizes each feature value in the importance feature, and uses the binarized feature as the mask feature.

[0092] For example, the mask feature can be a 3D binary mask feature, where the feature value "1" can be a mask feature value, and the feature value "0" can be a non-mask feature value.

[0093] The same feature element has different levels of importance in images of different quality levels. For example, complex texture elements are more important in images with higher quality levels, but less important, or even unnecessary, in images with lower quality levels. Based on this, this implementation allows for adjustments to the weights of each feature value in the initial importance feature set to adapt to different image quality requirements. It can balance the detail required to match quality needs with compression efficiency, giving the embodiments of this application good adaptability and scalability.

[0094] Based on the foregoing description of the image encoding process, it can be seen that the image encoding module includes quantization and entropy encoding. Accordingly, in this embodiment, the encoding module selects the feature corresponding to the non-masked feature value in the image frequency features as the feature to be encoded. This can be achieved by selecting the feature identified by the non-masked feature value from the quantized features of the image frequency features as the feature to be encoded.

[0095] For example, the encoding module can calculate a quantization vector based on the quality level, and quantize the image frequency features based on the quantization vector to obtain the quantized features of the image frequency features. For example, the quantization feature n' can satisfy: n' = AdaQ(n) = Round(n / (Q*Vq)), where n refers to the image frequency feature, Q refers to the quality level Q value, Vq refers to the quantization factor, and Q*Vq is the quantization vector. Further, the feature identified by the non-masked feature value is selected from the quantized features and determined as the feature to be encoded. satisfy: Where M(·) is the element selection operator, m(z,q) represents the mask feature, z refers to the mask feature value in the mask feature, and q refers to the non-mask feature value in the mask feature.

[0096] In conventional quantization, the quantization factor is calculated based on the QP value, which is a pre-set fixed value; that is, the degree of quantization remains constant in conventional quantization. In contrast, the technical solution of this application uses the Q value corresponding to the source image to calculate the quantization factor. This allows for more precise control of the quantization process to adapt to different image content and compression requirements, offering good flexibility and scalability. Furthermore, by adaptively controlling the quantization degree based on the Q value, it can maintain key visual information while performing more aggressive compression on less important areas, thereby achieving better visual quality at the same bitrate.

[0097] Furthermore, using the encoding method of this application embodiment, a 3D mask feature is generated for the image frequency features, and after quantizing the image frequency features, the features to be encoded are further selected based on the 3D mask feature. Since the 3D mask feature can characterize the channel importance of different regions of the image, the encoding of this application embodiment can select the features to be encoded from a finer-grained perspective, which is beneficial for achieving variable rate compression and controlling image quality.

[0098] In some embodiments, after obtaining the quality level (i.e., Q-value) of the source image, the AI ​​encoding / decoding system can determine the rated bitrate matching the quality level of the source image. Then, according to a preset bitrate ratio corresponding to the frequency domain and the rated bitrate, it can calculate the bitrate matching each of the at least two image frequency features, where the bitrate ratio corresponding to the higher frequency domain is greater than that corresponding to the lower frequency domain. Afterwards, after the encoding module selects the feature to be encoded from the image frequency features, it can encode the feature to be encoded according to the bitrate matching that image frequency feature to obtain the bitstream corresponding to that image frequency feature. Here, encoding refers to entropy coding.

[0099] For example, the rated bitrate can be the total bitrate budget corresponding to the Q value; the larger the Q value, the larger the rated bitrate.

[0100] In some embodiments, different frequency components can be pre-assigned a fixed percentage of the bit rate. For example, the bit rate of high frequency components is 45%, the bit rate of mid frequency components is 30%, and the bit rate of low frequency components is 25%.

[0101] In other embodiments, the AI ​​codec system can use a rate allocation algorithm to dynamically allocate bitrates to different frequency components based on the Q value. The rate allocation algorithm may include, but is not limited to, Rate-Controlled Distortion Optimization (RCDO) algorithms.

[0102] In conjunction with the foregoing embodiments, the AI ​​encoding / decoding system has divided the source image into at least two frequency domains to obtain image frequency features. Using these frequency features as the primary component, entropy encoding is performed at different coding rates, thereby obtaining bitstreams from at least two frequency domains of the source image. This allows for the analysis and extraction of image components and details at different frequency domain levels, which is beneficial for improving image quality. Furthermore, encoding at a rate matching the image frequency features allows for variable bitrate encoding without increasing model complexity, resulting in strong versatility and high flexibility.

[0103] Corresponding to the foregoing encoding embodiments, this application also provides a decoding method. For example... Figure 6 As shown, the decoding method of this application embodiment may include steps 201 to 203.

[0104] Step 201: Obtain at least two bitstreams, wherein the at least two bitstreams correspond one-to-one with the image frequency features in at least two frequency domains.

[0105] Step 202: For any bitstream, decode the bitstream according to the bitrate that matches the bitstream to obtain decoding features.

[0106] The bitrate for matching the bitstream refers to the bitrate that matches the image frequency features corresponding to the bitstream, and the bitrate is proportional to the frequency components corresponding to the image frequency features.

[0107] Step 203: After obtaining at least two decoding features, the at least two decoding features are fused to obtain the reconstructed image of the source image.

[0108] Among them, at least two bitstreams are obtained by encoding at least two image frequency features, and the at least two image frequency features correspond to different frequency domains of the source image. Each bitstream is obtained by encoding a feature whose importance in the corresponding image frequency feature matches the quality level of the source image. Each image frequency feature is encoded by the encoding module corresponding to that image frequency feature.

[0109] The process of encoding any image frequency feature by the corresponding encoding module is detailed in the description of the above embodiments and will not be repeated here.

[0110] Combination Figure 2 As can be seen from the illustrated AI encoding and decoding system, Figure 6 In the illustrated implementation scenario, any one of the at least two bitstreams is received and decoded by a decoding module, meaning that at least two bitstreams correspond one-to-one with at least two decoding modules.

[0111] Based on the foregoing description of the image decoding process, it can be seen that the decoding module's decoding process for the bitstream includes entropy decoding and inverse quantization. The decoding bitrate used in the entropy decoding process can be the same as the entropy encoding bitrate corresponding to that bitstream. For any bitstream, the bitrate used for entropy decoding of that bitstream can be the bitrate obtained by the corresponding encoding module, that is, the bitrate of the image frequency feature matching corresponding to the bitstream.

[0112] Furthermore, dequantization is the inverse operation of quantization. In the embodiments of this application, the quantization factor is calculated based on the Q value corresponding to the source image. Correspondingly, the quantization factor used in the dequantization process can also be calculated based on the Q value corresponding to the source image. This will not be elaborated further in the embodiments of this application.

[0113] It should be understood that the at least two decoding features correspond one-to-one with the at least two bitstreams, and the at least two bitstreams are obtained by feature encoding of different frequency domains of the source image. Correspondingly, the at least two decoding features correspond to different frequency domains of the source image. That is, any decoding feature represents a part of the features of the source image. The decoding feature after fusing the at least two decoding features can represent the complete features of the source image. In this way, the AI ​​encoding and decoding system can reconstruct the source image based on the fused decoding features to obtain the reconstructed image of the source image.

[0114] As can be seen, before encoding the source image, this embodiment extracts image frequency features from at least two frequency domains of the source image and determines the bit rate for different frequency components of the source image, making the bit rate proportional to the frequency domain. During the encoding process, features whose importance matches the quality level of the source image are selected as features to be encoded, and encoding is performed according to the bit rate matching the image frequency features. In this way, there is no need to deploy multiple model weights, thereby saving storage resources and reducing the complexity and computing power of the AI ​​encoding and decoding system. Furthermore, the technical solution of this embodiment can adaptively select features to be encoded according to different frequency components of the source image, and use different bit rates for encoding different frequency components, which is not only highly versatile but also flexible.

[0115] The above embodiments are described from the perspectives of encoding and decoding processes. The embodiments of this application will now be described in conjunction with the composition of an AI encoding / decoding system and exemplary encoding / decoding processes.

[0116] refer to Figure 7 , Figure 7 This illustration shows a schematic diagram of the data flow for encoding and decoding in any frequency domain provided in an embodiment of this application. Figure 7 The AI ​​encoding / decoding model illustrated in the figure can be any branch model of the AI ​​encoding / decoding system in the embodiments of this application. This branch model is used to encode and decode the image frequency features in the frequency domain.

[0117] It should be noted that the branch model used in the AI ​​codec system to process features in other frequency domains can be compared with... Figure 7 Similarly, other branch models in the AI ​​encoding and decoding system will not be described again in this application embodiment.

[0118] Figure 7 The AI ​​encoding / decoding model shown may include an encoding module, a super-prior encoding module, a first entropy encoding module, a super-prior decoding module, a quantization module, a feature selection module, a second entropy encoding module, an entropy decoding module, and an inverse quantization module. Each module in this AI encoding / decoding model can be a deep learning model, such as a neural network. Upon receiving an image corresponding to this AI encoding / decoding model (e.g., an image with high-frequency components), the encoding module encodes the image into a feature representation n (i.e., image frequency features), and transmits the feature representation n to the quantization module and the super-prior encoding module respectively.

[0119] The quantization module can quantize the image frequency features using the algorithm n' = AdaQ(n) = Round(n / (Q*Vq)) to obtain the quantized feature n' of the image frequency features. Then, the quantized feature n' is transmitted to the feature selection module. The super-prior coding module, the first entropy coding module, and the super-prior decoding module can generate the 3D mask features corresponding to the image frequency features after processing the image frequency features.

[0120] For example, the super-prior encoding module is used to extract super-prior features from image frequency features. Furthermore, it can establish a distribution estimate for each super-prior feature, ensuring independence among the features. This distribution estimate characterizes the semantics of the image frequency features. This distribution estimate is then transmitted to the first entropy encoding module, which uses it to perform arithmetic encoding on the super-prior features, obtaining a binary code stream of super-prior features. This binary code stream is then transmitted to the super-prior decoding module. The super-prior decoding module performs arithmetic decoding on the binary code stream to obtain the recovered super-prior features, and applies a super-decoding neural network to the recovered super-prior features to obtain super-prior information. The advanced prior decoding module also receives the source image and Q-values, extracts image features from the source image, and these features characterize the semantics of the source image, such as structure, texture, and content. Furthermore, the advanced prior decoding module determines the key image features needed to reconstruct the source image based on these features, and evaluates the influence of each feature element in the image frequency features on the quality of the source image based on the similarity between the advanced prior information and the key image features, and generates a 3D importance feature map (i.e., the aforementioned initial importance feature). Further, the advanced prior decoding module can calculate the weights of each feature value in the 3D importance feature map based on the Q-values, so as to dynamically adjust the importance of the channel parameters of each feature value according to the Q-values.

[0121] For example, the weights of each feature value can be implemented as an importance curve, which can be a non-linear curve. The super-prior decoding module can be a convolutional neural network, in which adjustment parameters can be set to adjust the importance curve of the 3D important feature map based on the Q-value. These adjustment parameters can be obtained through pre-training.

[0122] Furthermore, the value range of each feature in the 3D important feature map after weight adjustment is between 0 and 1. The super-prior decoding module can binarize the value of each feature, and the binarized feature is used as the 3D mask feature, and the 3D mask feature is transmitted to the feature selection module.

[0123] Feature selection modules can be implemented, for example, through algorithms: Based on the 3D mask features, select the features to be encoded from the quantized features n'. and the feature to be encoded It is transmitted to the second entropy encoding module.

[0124] It should be noted that the AI ​​encoding / decoding system can determine the bitrate of image frequency features for different frequency components based on the Q-value of the source image, and configure the bitrate of the second entropy encoding module and entropy decoding module in the corresponding branch model. For example, Figure 7 The illustrated AI encoding / decoding model processes the image frequency features of high-frequency components. The code rate corresponding to the high-frequency components, calculated by the AI ​​encoding / decoding system based on the Q-value of the source image, can then be configured to... Figure 7 The second entropy encoding module and entropy decoding module are used. In this way, the second entropy encoding module can encode the features... Entropy encoding is performed according to the configured bitrate to obtain a bitstream of image frequency features of high-frequency components. This bitstream is then transmitted to the entropy decoding module, which performs entropy decoding to obtain data to be dequantized. This dequantized data is then transmitted to the dequantization module. After processing the dequantized data, the dequantization module outputs image features in the image frequency feature spatial domain. These features are then decoded into image data, which is high-frequency component image data. This image data can be used to fuse with image data from other frequency components and to reconstruct the reconstructed image corresponding to the source image.

[0125] It should be understood that Figure 7 The AI ​​codec model shown is for illustrative purposes only and does not limit the branch model in the AI ​​codec system of this application embodiment. In actual implementation scenarios, the AI ​​codec model may also include more than Figure 7 More or fewer modules can be shown, and more can be added. Figure 7 Multiple modules in the module can be merged into one module, or... Figure 7 Some modules can be split into multiple modules, etc. No restrictions are placed here.

[0126] The implementation method of this application embodiment allows the quantization module to adaptively capture important features in the source image based on the Q value, and the algorithm of pre-deploying 3D mask features in the super-prior decoding module supports improving compression efficiency based on the content of the source image while ensuring image quality.

[0127] Since the AI ​​encoding / decoding system is an algorithmic system, it should be trained to execute the above embodiments before being put into use. In some embodiments, the training method may include: inputting a sample image into a network to be trained to obtain at least two predicted frequency features output by the network, wherein the at least two predicted frequency features correspond to different frequency domains of the sample image. Further, a loss function is calculated based on the at least two predicted frequency features, the loss function including at least two loss values, each of which corresponds one-to-one with the at least two predicted frequency features, and any loss value characterizing the loss of the corresponding predicted frequency feature relative to the frequency features in the corresponding frequency domain of the sample image; if the loss function reaches a preset convergence condition, the network to be trained is determined as an encoding model, and the encoding model is used to encode the source image.

[0128] It should be noted that the semantics represented by image frequency features of different frequency components are different and each has its own emphasis. Therefore, the algorithm for calculating the loss value of each predicted frequency feature can be matched with the image semantics corresponding to the predicted frequency feature.

[0129] For example, when the network to be trained includes a high-frequency component feature extraction network, a mid-frequency component feature extraction network, and a low-frequency component feature extraction network, the at least two predicted frequency features include high-frequency component predicted frequency features, mid-frequency component predicted frequency features, and low-frequency component predicted frequency features. The high-frequency component represents edge details and texture information in the image, the mid-frequency component represents the shape and contour of objects in the image, and the low-frequency component represents the overall brightness and color distribution of the image. The perceptual loss value of the high-frequency component predicted frequency features relative to the high-frequency component frequency features of the sample image can be calculated to obtain L. high Calculate the mean squared error (MSE) loss value L of the predicted frequency features of the intermediate frequency components relative to the intermediate frequency component frequency features of the sample image. mid Calculate the color loss value of the predicted low-frequency component frequency features relative to the low-frequency component frequency features of the sample image, and obtain L. low Then, the perceptual loss value, the mean squared error loss value, and the color loss value are weighted and summed, and the result of the weighted sum is used as the loss function. The loss function L, for example, satisfies: L = W high L high +W mid L mid +W low L low Among them, W high W mid and W low These are the weighting coefficients.

[0130] As can be seen, in the encoding stage of the source image in this embodiment, at least two frequency domain image frequency features of the source image are first extracted, thereby enabling differentiated processing of different frequency components of the source image and effectively adjusting the bitrate. Furthermore, based on the image frequency features, features whose importance matches the quality level of the source image are selected from each image frequency feature as the features to be encoded for that corresponding image frequency feature. Specifically, for any image frequency feature, the feature whose importance matches the quality level of the source image can refer to the necessary feature representation required for encoding based on the quality level. This reduces the amount of data encoded and helps maintain high visual quality. In addition, in this embodiment, the bitrate is proportional to the frequency domain corresponding to the image frequency feature. Thus, for any image frequency feature to be encoded, it can be encoded according to the bitrate matching the image frequency feature to obtain the bitstream corresponding to that image frequency feature. This facilitates variable bitrate encoding based on different frequency domain components of the source image, improving encoding efficiency while preserving the details of the source image. As can be seen, the technical solution of this application embodiment does not require the deployment of multiple model weights, thereby saving storage resources and reducing the complexity and computing power of the AI ​​encoding and decoding system. Moreover, the technical solution of this application embodiment can adaptively select the features to be encoded for different frequency components of the source image according to the quality of the source image, and use different bit rates for encoding for different frequency components, which is not only highly versatile and flexible.

[0131] It should be understood that Figure 2 The components illustrated can be implemented as hardware or a combination of hardware and computer software. Whether the processing steps of any related component are executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can also implement the functions described in the above embodiments using different methods for specific applications, but such implementations should not be considered beyond the scope of this application.

[0132] For example, if the above implementation steps can be implemented by software modules, corresponding to the above encoding method, the embodiments of this application can also provide an encoding device.

[0133] like Figure 8A As shown, an encoding apparatus is provided, which may include an extraction unit 71, a feature selection unit 72, and an encoding unit 73. This encoding apparatus can be used to perform the above-described... Figures 2 to 5 Some or all of the operations of the encoding module.

[0134] For example: extraction unit 71 is used to extract image frequency features in at least two frequency domains of the source image; feature selection unit 72 is used to select features among the image frequency features whose importance matches the quality level of the source image as features to be encoded, wherein the features to be encoded refer to the necessary feature representations required when encoding based on the quality level; encoding unit 73 is used to encode the features to be encoded according to a bitrate that matches the image frequency features to obtain a bitstream corresponding to the image frequency features, wherein the bitrate is proportional to the frequency components corresponding to the image frequency features.

[0135] Optionally, the feature selection unit 72 is further configured to generate a mask feature of the image frequency feature based on the image frequency feature, the source image, and the quality level, wherein the mask feature includes a mask feature value and a non-mask feature value; and to determine the feature corresponding to the non-mask feature value in the quantized features of the image frequency feature as the feature to be encoded.

[0136] Optionally, the feature selection unit 72 is further configured to extract the semantic representation of the frequency domain corresponding to the image frequency features from the image frequency features; generate an initial importance feature based on the semantic representation and the image features of the source image, wherein any feature value in the initial importance feature represents the degree of influence of the corresponding pixel on the semantics of the source image; and adjust each feature value in the initial importance feature according to the quality level to obtain the mask feature.

[0137] Optionally, any feature value in the initial importance feature includes the position parameter and channel parameter of the corresponding pixel. The channel parameter represents the parameter of each color channel of the pixel. The feature selection unit 72 is further used to calculate the weight of the channel parameter corresponding to each feature value according to the quality level; adjust each feature value according to the corresponding weight to obtain the importance feature; binarize each feature value in the importance feature, and use the binarized feature as the mask feature.

[0138] Optionally, the feature selection unit 72 is further configured to calculate a quantization vector based on the quality level; quantize the image frequency features based on the quantization vector to obtain quantized features of the image frequency features; and select the features identified by the non-masked feature value from the quantized features to determine the features to be encoded.

[0139] Optionally, the extraction unit 71 is further configured to extract image features of the source image; convert the image features into frequency domain features; and extract image frequency features of high-frequency components, mid-frequency components, and low-frequency components from the frequency domain features, respectively; the image frequency features of high-frequency components characterize the texture features of the source image, the image frequency features of mid-frequency components characterize the contour features of the source image, and the image frequency features of low-frequency components characterize the color distribution features of the source image.

[0140] Optionally, the encoding device further includes a determining unit and a calculating unit. The determining unit is used to determine the rated bit rate matching the quality level. The calculating unit is used to calculate the bit rate matching each of the at least two image frequency features according to the preset bit rate ratio in the frequency domain and the rated bit rate, wherein the bit rate ratio corresponding to the higher frequency domain is greater than the bit rate ratio corresponding to the lower frequency domain.

[0141] Optionally, the encoding device further includes an input unit. The input unit is configured to input a sample image into the network to be trained to obtain at least two predicted frequency features output by the network, wherein the at least two predicted frequency features correspond to different frequency domains of the sample image. The calculation unit is further configured to calculate a loss function based on the at least two predicted frequency features, wherein the loss function includes at least two loss values, each corresponding one-to-one with the at least two predicted frequency features, and any loss value characterizes the loss of the corresponding predicted frequency feature relative to the frequency features in the corresponding frequency domain of the sample image. The determination unit is further configured to, when the loss function reaches a preset convergence condition, determine the network to be trained as an encoding model, wherein the encoding model is used to encode the source image.

[0142] Optionally, when the network to be trained includes a high-frequency component feature extraction network, a mid-frequency component feature extraction network, and a low-frequency component feature extraction network, the at least two predicted frequency features include high-frequency component predicted frequency features, mid-frequency component predicted frequency features, and low-frequency component predicted frequency features. The computing unit is further configured to calculate the perceptual loss value of the high-frequency component predicted frequency features relative to the high-frequency component frequency features of the sample image; calculate the mean square error loss value of the mid-frequency component predicted frequency features relative to the mid-frequency component frequency features of the sample image; calculate the color loss value of the low-frequency component predicted frequency features relative to the low-frequency component frequency features of the sample image; and perform a weighted summation of the perceptual loss value, the mean square error loss value, and the color loss value, using the weighted summation result as the loss function.

[0143] Correspondingly, such as Figure 8BAs shown, a decoding apparatus is provided, which may include an acquisition unit 81, a decoding unit 82, and a fusion unit 83. This encoding apparatus can be used to perform the above-described... Figure 6 Some or all of the operations of the decoding module.

[0144] For example: Acquisition unit 81 is used to acquire at least two bitstreams, the at least two bitstreams corresponding one-to-one with at least two image frequency features in the frequency domain; each bitstream is obtained by encoding a feature whose importance in the corresponding image frequency feature matches the quality level of the source image, the feature to be encoded refers to the necessary feature representation required when encoding based on the quality level; Decoding unit 82 is used to decode any bitstream according to a bitrate matching the bitstream to obtain decoding features, the bitrate matching the bitstream refers to the bitrate matching the image frequency feature corresponding to the bitstream, the bitrate is proportional to the frequency component corresponding to the image frequency feature; Fusion unit 83 is used to fuse the at least two decoding features after obtaining them to obtain a reconstructed image of the source image, the at least two decoding features corresponding one-to-one with the at least two bitstreams.

[0145] Understandable, Figure 8A and Figure 8B The division of the various units is merely a logical functional division; in actual implementation, the functions of these units can be integrated into the hardware entity of the electronic device. (Reference) Figure 9 , Figure 9 An electronic device is provided, comprising a processor 811, a transceiver 812, and a memory 813, which are connected and communicate via a communication bus 814. The processor 811 may integrate... Figure 2 The transceiver 812 is used to acquire source images, and the memory 813 includes the functions of each module. Figure 2 The diagram illustrates the data and program instructions of each algorithm module. When a program instruction is invoked, it causes the processor 811 to execute the aforementioned... Figures 3 to 6 Some or all of the operations in the process.

[0146] For details on the implementation process, please refer to [link / reference]. Figures 3 to 6 Descriptions of electronic devices are omitted here.

[0147] This application also provides a computer-readable storage medium storing data read and write instructions, which, when run on a computer, cause the computer to perform some or all of the steps in the method described in the foregoing embodiments.

[0148] This application also provides a computer program product including instructions for reading and writing data, which, when run on a computer, causes the computer to perform some or all of the steps in the method described in the foregoing embodiments.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0151] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0153] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, smartphone, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] Although alternative embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0155] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this invention.

Claims

1. An encoding method, characterized in that, The method, applied to artificial intelligence (AI) coding models, includes: Extract image frequency features from at least two frequency domains of the source image; The features selected from the image frequency features whose importance matches the quality level of the source image are used as the features to be encoded. The features to be encoded refer to the necessary feature representations required for encoding based on the quality level. The feature to be encoded is encoded according to a bitrate that matches the image frequency feature to obtain a bitstream corresponding to the image frequency feature, wherein the bitrate is proportional to the frequency component corresponding to the image frequency feature.

2. The method according to claim 1, characterized in that, The step of selecting features from the image frequency features whose importance matches the quality level of the source image as the features to be encoded includes: Based on the image frequency features, the source image, and the quality level, a mask feature for the image frequency features is generated, wherein the mask feature includes mask feature values ​​and non-mask feature values. The feature corresponding to the non-masked feature value in the quantized image frequency features is determined as the feature to be encoded.

3. The method according to claim 2, characterized in that, The step of generating a mask feature for the image frequency features based on the image frequency features, the source image, and the quality level includes: Extract the semantic representation of the frequency domain corresponding to the image frequency features from the image frequency features; An initial importance feature is generated based on the semantic representation and the image features of the source image, wherein any feature value in the initial importance feature represents the degree of influence of the corresponding pixel on the semantics of the source image; The mask features are obtained by adjusting the individual feature values ​​in the initial importance feature according to the quality level.

4. The method according to claim 3, characterized in that, Each feature value in the initial importance feature includes the position parameter and channel parameter of the corresponding pixel, whereby the channel parameter characterizes the parameters of each color channel of the pixel. Adjusting each feature value in the initial importance feature according to the quality level includes: Calculate the weights of the channel parameters corresponding to each feature value based on the quality level; Each feature value is adjusted according to its corresponding weight to obtain the importance feature; Each feature value in the importance feature is binarized, and the binarized feature is used as the mask feature.

5. The method according to claim 2, characterized in that, The step of determining the feature corresponding to the non-masked feature value in the image frequency features as the feature to be encoded includes: Calculate the quantization vector based on the quality level; The image frequency features are quantized based on the quantization vector to obtain the quantized features of the image frequency features; The feature identified by the non-masked feature value from the quantized features is determined as the feature to be encoded.

6. The method according to claim 1, characterized in that, The extraction of at least two image frequency features from the source image includes: Extract image features from the source image; The image features are converted into frequency domain features; Image frequency features of high-frequency components, mid-frequency components, and low-frequency components are extracted from the frequency domain features, respectively. The image frequency features of high-frequency components represent the texture features of the source image, the image frequency features of mid-frequency components represent the contour features of the source image, and the image frequency features of low-frequency components represent the color distribution features of the source image.

7. The method according to claim 1, characterized in that, Before encoding the features to be encoded according to a bitrate matching the image frequency features, the method further includes: Determine the rated bitrate to match the quality level; According to the preset frequency domain bit rate ratio and the rated bit rate, calculate the bit rate matching each image frequency feature in the at least two image frequency features, where the bit rate ratio corresponding to the higher frequency domain is greater than the bit rate ratio corresponding to the lower frequency domain.

8. A training method for an AI encoding model, characterized in that, The method includes: The sample image is input into the network to be trained to obtain at least two predicted frequency features output by the network to be trained, and the at least two predicted frequency features correspond to different frequency domains of the sample image. A loss function is calculated based on the at least two predicted frequency features. The loss function includes at least two loss values, which correspond one-to-one with the at least two predicted frequency features. Each loss value represents the loss between the corresponding predicted frequency feature and the frequency feature in the corresponding frequency domain of the sample image. When the loss function reaches the preset convergence condition, the network to be trained is determined as an encoding model, and the encoding model is used to execute the encoding method of any one of claims 1-7.

9. The method according to claim 8, characterized in that, When the network to be trained includes a high-frequency component feature extraction network, a mid-frequency component feature extraction network, and a low-frequency component feature extraction network, the at least two predicted frequency features include high-frequency component predicted frequency features, mid-frequency component predicted frequency features, and low-frequency component predicted frequency features. The calculation of the loss function based on the at least two predicted frequency features includes: Calculate the perceptual loss value of the predicted frequency features of the high-frequency components relative to the frequency features of the high-frequency components of the sample image; Calculate the mean square error loss value of the predicted frequency features of the intermediate frequency components relative to the intermediate frequency features of the sample image; Calculate the color loss value of the predicted frequency features of the low-frequency components relative to the frequency features of the low-frequency components of the sample image; The perceptual loss value, the mean square error loss value, and the color loss value are weighted and summed, and the result of the weighted summation is used as the loss function.

10. A decoding method, characterized in that, The method, applied to an artificial intelligence (AI) decoding model, includes: At least two bitstreams are acquired, and the at least two bitstreams correspond one-to-one with at least two image frequency features in the frequency domain; each bitstream is obtained by encoding a feature whose importance in the corresponding image frequency feature matches the quality level of the source image, and the feature to be encoded refers to the necessary feature representation required for encoding based on the quality level; For any bitstream, the bitstream is decoded according to a bitrate that matches the bitstream to obtain decoding features. The bitrate that matches the bitstream refers to the bitrate that matches the image frequency features corresponding to the bitstream. The bitrate is proportional to the frequency components corresponding to the image frequency features. After obtaining at least two decoding features, the at least two decoding features are fused to obtain the reconstructed image of the source image, wherein the at least two decoding features correspond one-to-one with the at least two bitstreams.

11. A codec system, characterized in that, The system includes: an artificial intelligence (AI) encoding model and an AI decoding model, wherein, The AI ​​encoding model is used to extract image frequency features from at least two frequency domains of the source image; select features among the image frequency features whose importance matches the quality level of the source image as features to be encoded, wherein the features to be encoded refer to the necessary feature representations required for encoding based on the quality level; encode the features to be encoded according to a bitrate matching the image frequency features to obtain the bitstream corresponding to the image frequency features, wherein the bitrate is proportional to the frequency components corresponding to the image frequency features. The AI ​​decoding model is used to decode any bitstream according to a bitrate that matches the bitstream to obtain decoding features. The bitrate that matches the bitstream refers to the bitrate that matches the image frequency features corresponding to the bitstream. The at least two decoding features obtained by the at least two decoding modules are fused to obtain the reconstructed image of the source image. The at least two decoding features correspond one-to-one with the at least two bitstreams.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to cause the electronic device to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-10.

14. A computer program product, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-10.

Images