Image processing method, training method, system, device, product and storage medium

Abstract

The specification provides an image processing method, a training method, a system, a device, a product and a storage medium, which are applied to the field of image coding. The method comprises: acquiring a source image to be encoded; acquiring complexity information, which represents the image processing time length of the source image; inputting the source image into a trained image coding system for encoding processing to obtain a hyper-prior code stream and an entropy coding code stream and output the same; the image coding system is used for encoding the hyper-prior representation of the source image into a hyper-prior code stream containing a first spatial mask; and encoding the latent representation of the source image into an entropy coding code stream based on a second spatial mask; the first spatial mask is extracted from the hyper-prior representation based on the complexity information and is used for representing whether the spatial positions in the hyper-prior representation need to be subjected to mask convolution; and the second spatial mask is extracted from the latent representation based on the complexity information and is used for representing whether the spatial positions in the latent representation need to be subjected to mask convolution.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and in particular to image processing methods, training methods, systems, devices, products and storage media. Background Technology

[0002] In recent years, with the development of deep learning, machine-oriented artificial intelligence (AI) encoding and decoding technologies have shown great potential in the field of image compression. Currently, machine-oriented AI encoding and decoding offer at least a 60% improvement in compression efficiency compared to traditional codecs. This compression advantage has attracted widespread attention in many application scenarios, such as streaming media transmission and intelligent monitoring.

[0003] However, even though AI encoding and decoding technologies have significant advantages in bit rate, the complex processing of image encoding and decoding results in long image processing times. Due to their complex model structures and intensive computational requirements, the long image processing time is currently a major obstacle to the widespread application of AI encoding and decoding technologies. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this specification provides image processing methods, training methods, systems, devices, products, and storage media.

[0005] According to a first aspect of the embodiments of this specification, an image processing method is provided, the method comprising:

[0006] Obtain the source image to be encoded;

[0007] Obtain complexity information, which represents the image processing time for the source image;

[0008] The source image is input into a trained image coding system for encoding processing to obtain a priori bitstream and an entropy-coded bitstream, which are then output.

[0009] The image coding system is configured to encode the prior representation of the source image into a priori bitstream containing a first spatial mask; and to encode the latent representation of the source image into an entropy-coded bitstream based on a second spatial mask.

[0010] The first spatial mask is extracted from the super-prior representation based on the complexity information and is used to characterize whether each spatial location in the super-prior representation needs to be masked convolution.

[0011] The second spatial mask is extracted from the latent representation based on the complexity information and is used to characterize whether each spatial location in the latent representation needs to be masked convolution.

[0012] According to a second aspect of the embodiments of this specification, a training method for an image encoding and decoding system is provided, the method comprising:

[0013] Obtain a training dataset; wherein, the training image samples in the training dataset correspond to spatial mask samples;

[0014] The training image samples are input into the initial image coding system to obtain the super-prior bitstream and entropy-coded bitstream of the training image samples;

[0015] The super-prior bitstream and entropy-encoded bitstream of the image sample are input into the initial image decoding system. After the initial image decoding system decodes the super-prior bitstream, it obtains the spatial mask and super-prior representation. Then, it decodes the entropy-encoded bitstream to obtain the initial reconstructed image sample. Based on the bit rate of the initial reconstructed image sample and the degree of distortion between the pixels of the initial reconstructed image sample and the pixels of the training image sample, the initial image encoding system and the initial image decoding system are trained to obtain the intermediate image encoding system and the intermediate image decoding system.

[0016] The training image samples and corresponding complexity information samples are input into the intermediate image encoding system to obtain the super-prior bitstream and entropy-encoded bitstream generated by the intermediate image encoding system. Then, the intermediate image decoding system decodes the intermediate images to obtain intermediate reconstructed image samples. Based on the bit rate of the intermediate reconstructed image samples, the degree of distortion between the pixels of the intermediate reconstructed image samples and the pixels of the training image samples, and the decoding time index of the entropy-encoded bitstream, the intermediate image encoding system and the intermediate image decoding system are trained to obtain a trained image encoding system and an image decoding system.

[0017] According to a third aspect of the embodiments of this specification, an image processing method is provided, the method comprising:

[0018] Obtain the super-prior code stream and the entropy-coded code stream; the super-prior code stream and the entropy-coded code stream are obtained through the steps of the method described in the first aspect;

[0019] The super-prior bitstream and the entropy-coded bitstream are input into an image decoding system. The image decoding system is used to decode the super-prior representation and spatial mask from the super-prior bitstream, and decode the entropy-coded bitstream based on the super-prior representation and spatial mask to obtain the reconstructed image.

[0020] According to a fourth aspect of the embodiments of this specification, an image processing system is provided, the image processing system including an image encoding end and an image decoding end, the image encoding end being used to perform the steps of the method described in the first aspect, and the image decoding end being used to perform the steps of the method described in the third aspect.

[0021] According to a fifth aspect of the embodiments of this specification, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method embodiments described in the first, second, or third aspects above.

[0022] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method embodiments described in the first, second, or third aspects above.

[0023] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method embodiments described in the first, second, or third aspects above.

[0024] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0025] In the embodiments of this specification, the image coding system includes an entropy coding function. A spatial mask is used to indicate which spatial locations in the image need to be masked convolution. The spatial mask is generated using complexity information that characterizes the image processing time of the source image. Therefore, the number of spatial locations in the image that need to be masked convolution is controlled by the complexity information. Reducing the number of spatial locations in the image that need to be masked convolution can reduce the processing time of entropy coding and the corresponding decoding time, resulting in a lower image processing time. Thus, the image coding system can adjust the complexity of different entropy codes based on different complexity information, thereby achieving the effect that an image coding system can adjust the image processing time based on different complexities.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] Figure 1A This is a flowchart illustrating an image processing method according to an exemplary embodiment of this specification.

[0028] Figure 1B This is a flowchart illustrating another image processing method according to an exemplary embodiment of this specification.

[0029] Figure 1C This is an application scenario diagram illustrating an image processing method according to an exemplary embodiment of this specification.

[0030] Figure 2AThis is a schematic diagram illustrating an image encoding system according to an exemplary embodiment of this specification.

[0031] Figure 2B This is a schematic diagram illustrating an image decoding system according to an exemplary embodiment of this specification.

[0032] Figure 2C This specification illustrates an overall architecture diagram of an image encoding and decoding system based on an exemplary embodiment.

[0033] Figure 2D This is a schematic diagram illustrating a conditional transformation module according to an exemplary embodiment of this specification.

[0034] Figure 2E This is an overall architecture diagram of another image encoding and decoding method illustrated in this specification according to an exemplary embodiment.

[0035] Figure 2F This is a flowchart illustrating a training method for an image encoding / decoding system according to an exemplary embodiment.

[0036] Figure 3 This is a hardware structure diagram of a computer device containing an image processing apparatus, as illustrated in this specification according to an exemplary embodiment.

[0037] Figure 4 This is a structural diagram of an image processing apparatus illustrated in this specification according to an exemplary embodiment.

[0038] Figure 5 This is a structural diagram of another image processing apparatus illustrated in this specification according to an exemplary embodiment.

[0039] Figure 6 This is a structural diagram of a training apparatus for an image encoding and decoding system according to an exemplary embodiment of this specification. Detailed Implementation

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0041] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0042] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0043] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0044] Because most AI image codecs use complex model structures and have high computational requirements, the decoding process takes significantly longer. This not only increases the consumption of computing resources but also causes response latency issues in real-time applications, making real-time processing impossible and severely limiting the promotion of their practical applications.

[0045] Popular context-based neural network image encoding and decoding systems, while offering approximately 7% bitrate savings, suffer from drawbacks. Their entropy models are autoregressive, relying on previously decoded pixels to predict the current pixel. Although this provides higher compression efficiency, the serial processing significantly increases decoding time, negatively impacting system performance. This high time cost is extremely inconvenient in practical applications, making AI encoding and decoding technologies difficult to use in real-time visual analysis scenarios, thus limiting the widespread adoption and use of such products.

[0046] For example, entropy models based on autoregressive contexts can capture spatial dependencies in latent representations during image compression. Each spatial location in the latent representation can be modeled based on information from other spatial locations it depends on. Its autoregressive nature can be achieved through convolutional layer mask convolution. For instance, by masking the convolutional kernel (often called masked convolution), the values ​​of undecoded points are masked, ensuring that the model predicts the parameters of the current pixel only from previously decoded dependent points, not from undecoded points. Therefore, the problem with the autoregressive nature of entropy models lies in the strict temporal relationship: the information of the current point depends on previous points; the information of the current point can only be obtained after obtaining the previous points, and the previous points can only be obtained from even earlier points. This manifests as a sequential decoding process, resulting in a long decoding time.

[0047] To address this issue, this embodiment differs from most studies that only pursue a better bit rate while ignoring encoding and decoding complexity. It can achieve the effect of adaptively adjusting entropy encoding complexity and adjusting image processing time through an image encoding system.

[0048] like Figure 1A As shown, Figure 1A This is a flowchart illustrating an image processing method according to an exemplary embodiment, comprising the following steps:

[0049] In step 102, the source image to be encoded is obtained.

[0050] In step 104, complexity information is obtained. The complexity information represents the image processing time for the source image.

[0051] In step 106, the source image is input into a trained image coding system for encoding processing to obtain a priori bitstream and an entropy-coded bitstream, which are then output.

[0052] The image coding system is configured to encode the prior representation of the source image into a priori bitstream containing a first spatial mask; and to encode the latent representation of the source image into an entropy-coded bitstream based on a second spatial mask.

[0053] The first spatial mask is extracted from the super-prior representation based on the complexity information and is used to characterize whether each spatial location in the super-prior representation needs to be masked convolution.

[0054] The second spatial mask is extracted from the latent representation based on the complexity information and is used to characterize whether each spatial location in the latent representation needs to be masked convolution.

[0055] The image processing method of this embodiment can be used to encode source images and can run on any computer device that requires image encoding. Computer devices include, but are not limited to, servers, cloud servers, server clusters, tablet computers, personal digital assistants, laptops, desktop computers, or mobile devices. This embodiment of the image processing method can be applied to either a client or a server; this embodiment does not limit its application.

[0056] The hyperprior bitstream and entropy-coded bitstream obtained by the image coding system can be used to construct an image. For example, ... Figure 1B The diagram shown is a flowchart illustrating another image processing method according to an exemplary embodiment of this specification, including the following steps:

[0057] In step 112, the priori bitstream and the entropy-coded bitstream are obtained;

[0058] In step 114, the super-prior bitstream and the entropy-coded bitstream are input to the image decoding system;

[0059] The image decoding system is used to decode the prior representation and spatial mask from the prior bitstream, and decode the entropy-coded bitstream based on the prior representation and spatial mask to obtain the reconstructed image.

[0060] As an example, decoding the entropy-coded bitstream based on the prior representation and the spatial mask may include: determining each spatial location indicated in the spatial mask that does not require mask convolution, and performing parallel decoding on each determined spatial location based on the prior representation.

[0061] The image processing method of this embodiment can reconstruct an image based on a bitstream and can run on any computer device that requires image decoding. Such computer devices include, but are not limited to, servers, cloud servers, server clusters, tablet computers, personal digital assistants, laptops, desktop computers, or mobile devices. This embodiment of the image processing method can be applied to either a client or a server; this embodiment does not limit its application.

[0062] like Figure 1C The diagram shown is a schematic representation of an image processing system according to an exemplary embodiment of this specification. The image processing system of this embodiment may include an image encoding end and an image decoding end. The image encoding end is used to execute the aforementioned embodiment for image encoding, and the image decoding end is used to execute the aforementioned embodiment for image decoding. Figure 1C The example illustrates that the image encoding end and the image decoding end can run on different computer devices. In actual applications, the image encoding end and the image decoding end can also run on the same computer device. This embodiment does not limit this.

[0063] In this embodiment, considering the practical applications of AI encoding and decoding, it is necessary to control the encoding / decoding time. AI encoding and decoding employs entropy coding, a technique where entropy coding models perform masked convolutions on various spatial locations in the latent representation to accurately predict the image feature distribution (i.e., entropy) within the latent representation. Research shows that not all spatial locations in an image require autoregressive masked convolutions. For example, in some cases, when decoding a smooth image filled with sky, the prior bitstream is sufficient to accurately estimate the distribution parameters. In other words, based on image information contained in the prior bitstream (e.g., image context information, image statistics, or image edge information), the time-consuming masked convolutions at certain spatial locations in the latent representation can be avoided. The encoding / decoding complexity is controlled by adjusting the number of spatial locations for masked convolutions.

[0064] Therefore, this embodiment introduces a spatial mask that characterizes whether masked convolution is needed at each spatial location. The spatial mask is generated under the guidance of complexity information, meaning that the input of the image encoding system includes dynamically variable complexity. Through this complexity information, the system can adaptively control the encoding complexity based on the image, thereby controlling the decoding complexity. The resulting image encoding and decoding system is a unified model that can dynamically adjust the encoding and decoding complexity according to different complexity information, eliminating the need to store multiple models for different complexity levels.

[0065] This embodiment designs a two-way coding system for the image coding system, one being a priori coding and the other entropy coding. Entropy coding requires the priori representation y... f This was conducted under the guidance of [unclear], in order to improve compression efficiency and accurately model the probability distribution of the latent representation y.

[0066] Firstly, in the area of ​​super-prior coding, in neural network image compression, the source image x can be converted into a latent representation y through neural network coding. The latent representation y typically contains rich image information, but its distribution is complex and has high redundancy. To perform entropy coding more effectively, a way is needed to capture and utilize the statistical properties of y; therefore, super-prior coding is introduced. Through super-prior coding, the super-prior representation y can be obtained. f The super-prior representation y f It can contain high-level statistical information that may represent y, such as the correlation and distribution characteristics of various spatial locations.

[0067] Accurate entropy encoding relies on a precise estimate of the probability distribution p(y) of y. Directly modeling the high-dimensional distribution of y is difficult and computationally complex. However, by using a priori representations of y... f The provided image context information can transform p(y) into a more accurate conditional probability p(y|y). fThis allows for more efficient bit allocation, reduced redundant information, and improved bit rate and accuracy. However, existing entropy coding has high processing complexity, resulting in longer processing and decoding times.

[0068] In order to control the complexity of the image encoding and decoding process, since it is not desirable to store specific image encoding systems for each level of specific decoding complexity, this embodiment designs a variable complexity technique, that is, the image encoding system can adaptively select the encoding complexity for image encoding based on the input image and complexity information.

[0069] In this embodiment, in addition to acquiring the source image, it can also determine complexity information representing the image processing time of the source image; here, the image processing time represents the image encoding time and / or image decoding time. There are various ways to determine the complexity information. As an example, acquiring the complexity information may include any of the following steps:

[0070] Obtain the preset complexity information;

[0071] Obtain complexity information provided by the user;

[0072] The complexity information is determined based on the resource information of the device where the image encoding system is located; or,

[0073] Complexity information is determined based on the downstream tasks used to process the hyperprior stream and the entropy-coded stream.

[0074] In this embodiment, the complexity information can be user-defined by the source image provider. For example, the user can manually set the complexity information according to specific needs (such as device performance, real-time requirements, etc.). When resources are limited or faster decoding is required, the user can choose lower complexity information. Alternatively, the image encoding system can be configured with fixed complexity information during deployment, allowing different complexity information to be used on different devices. Alternatively, it can be automatically adjusted, either within the image encoding system itself or outside the system, and then provided to the system. The automatic determination of complexity information can be based on the resource information of the device hosting the image encoding system, such as dynamic determination based on various factors like the runtime environment or resource conditions. For example, it can be based on the resources, capabilities, and current operating scenario of the computer device hosting the image encoding system. Alternatively, it can be determined based on downstream tasks, which are tasks that process the priori bitstream and entropy-coded bitstream. Different tasks have different time requirements for bitstream decoding. Other methods of obtaining complexity information may also be included in practical applications, but this embodiment does not limit these methods.

[0075] In this embodiment, a function capable of extracting spatial masks based on complexity information is designed into the image encoding system. The image processing time indicated by the complexity information is positively correlated with the number of spatial locations in the spatial mask that require mask convolution. That is, the complexity information is used to control the number of spatial locations in the spatial mask (first spatial decoding and second spatial decoding) that require mask convolution.

[0076] As an example, the image encoding system is used to perform neural network encoding on the source image to obtain a latent representation of the source image, then perform super-prior encoding to obtain a super-prior representation, extract a first spatial mask based on the complexity information to characterize whether each spatial location in the super-prior representation needs to be masked convolution, encode the super-prior representation to obtain a super-prior bitstream containing the first spatial mask; and after extracting a second spatial mask based on the complexity information to characterize whether each spatial location in the latent representation needs to be masked convolution, entropy encoding is performed on the latent representation based on the second spatial mask and the super-prior bitstream to obtain an entropy-encoded bitstream.

[0077] In the super-prior coding approach, a first spatial mask representing whether each spatial location in the super-prior representation needs masked convolution can be extracted based on complexity information. The super-prior representation can then be encoded so that the encoded super-prior bitstream contains the first spatial mask. The spatial mask can be binary information. To ensure the spatial mask is available at the decoding end, allowing it to decode the entropy-coded bitstream, and considering the relatively small size of the super-prior bitstream, the spatial mask is included within it. Only a small number of additional bits need to be transmitted on top of the super-prior bitstream to achieve the transmission of the spatial mask.

[0078] In entropy coding, a second spatial mask representing whether each spatial location in the latent representation needs mask convolution can be extracted based on complexity information. Then, entropy coding is performed on the latent representation based on the second spatial mask and the super-prior bitstream to obtain the entropy-coded bitstream. The second spatial mask and the first spatial mask can be the same.

[0079] Through the above processing, the image coding system can obtain the super-prior bitstream and the entropy-coded bitstream, which can be output, for example, provided to the decoding side, to construct the reconstructed image.

[0080] Figure 2A This is a schematic diagram illustrating an image encoding system according to an exemplary embodiment. Figure 2B This is a schematic diagram illustrating an image decoding system according to an exemplary embodiment. Figure 2CThis specification illustrates an image encoding / decoding architecture according to an exemplary embodiment; as an example, it can be pre-built. Figure 2C The image encoding / decoding architecture shown is used for training, and the image encoding / decoding architecture is then used to obtain the image encoding / decoding data. Figure 2A The image encoding system shown is deployed on the computer device where image encoding is required, and the image encoding / decoding architecture is obtained from this system. Figure 2B The image decoding system shown is deployed on the computer equipment that needs to perform image decoding.

[0081] like Figure 2A As shown, the image encoding system may include:

[0082] A neural network encoder is used to perform neural network encoding on the source image to obtain a latent representation y of the source image.

[0083] A super-prior coding model is used to perform super-prior coding on the latent representation y to obtain the super-prior representation y. f .

[0084] The first conditional transformation module is used to extract a first spatial mask based on the complexity information, representing whether each spatial location in the super-prior representation needs to undergo mask convolution, to obtain a super-prior representation containing the first spatial mask. Figure 2A The super-submersible representation z1 is shown in the diagram. The condition transformation module will be explained in detail later.

[0085] The first quantization module is used to quantize the prior representation containing the first spatial mask to obtain a quantized prior representation containing the first spatial mask; that is... Figure 2A The quantized hyperturbation representation z1 is shown in the figure.

[0086] A first arithmetic encoder is used to encode the quantized hyperprior representation carrying the first spatial mask to obtain the hyperprior bitstream. The encoded hyperprior bitstream may carry the first spatial mask M. Optionally, the first arithmetic encoder may be configured to... Figure 2C Arithmetic encoding is performed under the guidance of the Factorized Entropy Model (FEAM). For example, the quantized hyperlatent representation z1 can be input into the FEAM and arithmetic encoding is performed on z1 under the guidance of the FEAM. The FEAM is a method for efficiently modeling the probability distribution p(y|y_f) of a latent representation y. This model assumes a certain degree of independence between different elements of the latent representation, thus simplifying the modeling process of the probability distribution. Through the FEAM, y can be used to... f Predict each y jThe probability distribution parameters (such as mean and variance) are then used. Then the predicted probability distribution p(y) is used. j |y f Encode y to obtain the super-prior bitstream.

[0087] The second conditional transformation module is used to extract a second spatial mask based on the complexity information, representing whether each spatial location in the latent representation needs to undergo mask convolution, to obtain a latent representation containing the second spatial mask. Figure 2A The latent representation z2 shown can be understood as containing the latent representation y and the second space mask.

[0088] The second quantization module is used to quantize the latent representation containing the second spatial mask, obtaining a quantized latent representation containing the second spatial mask. Figure 2A The quantized latent representation z2 is shown in the figure.

[0089] A super-prior decoding module, consisting of a first arithmetic decoder, a third conditional transformation module, and a super-prior decoding model, is used to perform super-prior decoding on the super-prior bitstream to obtain the image edge information represented by the decoded super-prior representation. As an example, the super-prior decoding module may include:

[0090] A first arithmetic decoder is used to perform arithmetic decoding on the hyperprior bitstream to obtain a quantized hyperprior representation containing a first spatial mask; that is... Figure 2A The quantized hyperlatent representation z1 is shown in the figure. The arithmetic decoder can be used in, for example... Figure 2C Arithmetic coding is performed under the guidance of the factorization entropy model shown. For example, a priori bitstream can be input into the factorization entropy model, and arithmetic decoding of the priori bitstream is performed under the guidance of the factorization entropy model.

[0091] The third conditional transformation module is used to perform an inverse conditional transformation on the quantized hyper-prior representation carrying the first complexity information to obtain the first spatial mask and the hyper-prior representation y. f .

[0092] A super-prior decoding model is used to decode the super-prior representation to obtain the image edge information represented in the super-prior representation, as well as a first spatial mask M.

[0093] An entropy coding model is used to entropy code the latent representation based on the second spatial mask and the image edge information represented by the decoded super-prior representation, to obtain an entropy-coded bitstream.

[0094] In this way, the image coding system can output both the a priori bitstream and the entropy-coded bitstream.

[0095] like Figure 2BAs shown, the image decoding system may include:

[0096] and Figure 2A The same super-prior decoding module is composed of the first arithmetic decoder, the third conditional transformation module, and the super-prior decoding model; similarly, after the super-prior bitstream is decoded, the decoded image edge information and spatial mask M can be provided to the entropy decoding model.

[0097] It also includes an entropy decoding model for entropy decoding of the entropy-coded bitstream based on image edge information and a spatial mask M, to obtain a quantized latent representation z2. Specifically, the entropy decoding process can be as follows: Figure 2C As shown, the decoding is performed by a second arithmetic decoder guided by a masked autoregressive model. For example, an entropy-coded bitstream can be input into the masked autoregressive model, which combines a spatial mask (MM) and an autoregressive mechanism to decode the latent representation y during the decoding process. This model allows for complex autoregressive processing at specific spatial locations, while employing parallel decoding strategies at other locations to balance decoding efficiency and image quality. As an example, the spatial mask M indicates which locations require autoregressive processing (i.e., depend on previous locations) and which locations can be decoded in parallel. Under the guidance of the spatial mask M, spatial locations requiring masked convolution use an autoregressive method to predict based on previous locations, thereby improving decoding accuracy and image quality. Spatial locations not requiring masked convolution are decoded in parallel, allowing multiple locations to be decoded simultaneously, significantly reducing decoding time and computational resource consumption.

[0098] The fourth conditional transformation module is used to perform an inverse conditional transformation on the quantized latent representation z2 to obtain the second space mask and the latent representation y.

[0099] A neural network decoder is used to decode a reconstructed image based on a latent representation y.

[0100] like Figure 2D The diagram shows the structure of the conditional transformation module. The conditional transformation modules mentioned in the preceding embodiments can adopt the same structure; however, after training, the network parameters of each conditional transformation module will be different. The conditional transformation module is used to fuse the spatial mask and the latent representation y to obtain the fused features. Figure 2D Shown:

[0101] The first splicing layer is used to splice the latent representation y and the data representation corresponding to the spatial mask M (i.e., the spatial mask representation in the figure) to obtain the spliced ​​representation.

[0102] The first convolutional layer is used to perform convolution operations on the latent representation y to obtain the convolutional features of the latent representation y.

[0103] The second concatenation layer is used to concatenate the concatenated representation and the latent representation y with the concatenated features to obtain the concatenated features.

[0104] The first feature fusion layer is used to fuse the spliced ​​features to obtain the first fused feature;

[0105] An activation function layer is used to apply an activation function, such as the ReLU function, to the first fusion feature to obtain the activated first fusion feature.

[0106] The second convolutional layer is used to perform convolution operations on the first fused features of the spliced ​​representation and activation to obtain convolutional features.

[0107] The second feature fusion layer is used to fuse the concatenated representation and the convolutional features output by the second convolutional layer to obtain the final fused features.

[0108] For the source image, the complexity information used in this embodiment is as an aid. The complexity information l can represent the complexity level, which can be represented by a numerical range. In this embodiment, l∈[0,1] is used as an example; in practical applications, other numerical ranges can be used, and this embodiment does not limit this. The complexity level l is used to guide the transformation and probability estimation in the latent representation y. The specific value can be related to the decoding complexity required by the user. In addition, since the distribution of the entropy-encoded bitstream differs under different time constraints, the discrete entropy of the entropy-encoded bitstream is established on condition l. At the decoding end, the spatial binary mask M can be decoded to locate which spatial locations should discard mask convolution. Therefore, y can be decoded using the probability distribution calculated by the super-prior and partial spatial autoregression.

[0109] In this embodiment, at the encoding end, three condition transform modules can be inserted during the analysis transform, synthesis transform, and supercoding processes to adapt to changes in the decoding complexity level. The condition transform contains information about the complexity level l.

[0110] First, the source image x is transformed into y by the master encoder (i.e., the neural network encoder in the diagram). In the hyperprior encoding path, the latent representation z1 is generated by a given complexity level l, and its calculation formula can be:

[0111] z1 = h a (y f ,ψ), ψ=h a,c (y f ,l);

[0112] Among them, h a For the hyperencoder model, ha,c This corresponds to the first conditional transformation module in the diagram. Since different complexities correspond to different amounts of hyper-information to be transmitted, the distribution of the quantized latent representation z1 varies with the complexity level l. The other conditional transformation modules follow the same principle. Here, the parameter ψ represents the output of the conditional transformation module, i.e., the fused feature output by the conditional transformation module.

[0113] After the super-prior bitstream z1' is transmitted to the decoder, in addition to generating features for plotting the entropy parameters (μ,δ), the spatial mask M is also decoded from the super-prior bitstream to guide the entropy coding model in performing spatial autoregressive entropy coding.

[0114] Optionally, an upsampling network f can also be applied. c Features used to extract complexity information, such as complexity information. Figure 2E As shown, it is Figure 2C The diagram illustrates the addition of an upsampling network to the existing structure. Each upsampling network is equivalent to adding a branch to each conditional transformation module. The conditional transformation module is used to extract the spatial mask, and the upsampling network is used to extract complexity information.

[0115] After receiving the bitstream and complexity information, the decoding end performs the corresponding decoding operations and then reconstructs the image. The reconstructed image is used for downstream vision tasks, such as object detection, image classification, and image segmentation.

[0116] The optimization objectives of the image encoding and decoding system in this embodiment will be explained next.

[0117] In practical applications, image codecs are concerned with more than just rate-distortion (RD) performance. Encoding complexity, especially decoding complexity, directly impacts latency in many real-time applications. Therefore, considering decoding complexity alongside bit-distortion optimization is crucial. This embodiment extends rate-distortion (RD) optimization to the following rate-distortion-complexity (RDC) function, aiming to decode time constraints T. t Minimize the RD loss and decoding complexity:

[0118] min(R+λ·D)stT≤T t

[0119] Where T represents the decoding time. For neural image codecs, the decoding time can be divided into two parts:

[0120] T = T e +T n

[0121] Among them, T e T represents the entropy decoding time. n This represents the inference time used for the decoding network. Typically, T... e Usually more than T n The complexity is even greater, especially when using an entropy decoding model based on an autoregressive context (Mask-guilded Autoregressive Model). Due to the nature of autoregressive mask convolution, the decoding of the latent representation must be performed sequentially. Regardless of the architecture of the entropy model, the complexity of entropy decoding can be described by the number of autoregressions, *n*, which is O(n). Specifically, in the initial superoptimal model, *n* = 1. In the checkerboard context model, the number of autoregressions is 2. In the channel-wise autoregressive model, the number of autoregressions is *s*, where *s* is the number of channel groups. Following this definition, it is possible for the first time to quantitatively describe the decoding complexity of the entropy model. The overall complexity *l* is positively correlated with the size of *n*.

[0122] After analyzing the complexity of entropy decoding, a new optimization objective can be formulated to balance the trade-off between rate-distortion-complexity (RDC). Since the decoding time (Tn) of the neural network inference used for decoding is relatively negligible and directly related to hardware computation speed, this part can be ignored in the RDC loss function. The overall RDC optimization objective can be as follows:

[0123] L 总 =R+λ D ·D+λ C ·C e

[0124] C e It is an indicator of entropy decoding time; and it also introduces another Lagrange multiplier, λ. C This is to control the time complexity of entropy decoding. The autoregressive properties of the entropy model are strongly correlated with the time complexity of entropy decoding. Based on a typical spatial autoregressive context model, C can be further calculated. e The value of is calculated as follows.

[0125] This embodiment designs C eA key idea behind this computational approach is that not all spatial locations in an image need to be predicted using autoregressive masked convolution. While masked convolution can more accurately estimate the entropy of the latent representation, in some cases, such as decoding a smooth image filled with sky, the prior variable z is sufficient to accurately estimate the distribution parameters. In other words, depending on the specific image context, the time-consuming masked convolution can sometimes be avoided. In short, this embodiment generates an additional spatial mask to indicate which spatial locations should be predicted sequentially (i.e., sequential predictions based on autoregressive masked convolution).

[0126] In this embodiment, this spatial mask is denoted as M; where M j =1 indicates the potential representation y j The probability distribution is estimated with the help of masked convolution. Conversely, M j =0 means y j It can be decoded in parallel with other potential representations that have the same zero mask value.

[0127] Based on this, the decoding time index of the entropy-encoded bitstream in this embodiment is determined by the number of spatial locations in the spatial mask samples of the training image samples that need to be masked and convolved, as well as the weights corresponding to the complexity information samples. The decoding time index is positively correlated with the number of locations and also positively correlated with the image processing time represented by the complexity information samples.

[0128] Since using masked convolutions in more locations leads to longer decoding times, this embodiment can reduce the decoding time metric C of the entropy-coded bitstream. e Decomposed into:

[0129]

[0130] Where H and W are the height and width of the latent representation y, respectively, and are proportional to the resolution of the source image. j represents the spatial location j in the latent representation y. A mask of all zeros represents a parallel super-prior entropy model, i.e., it uses the super-prior coding model entirely. A mask of all ones is equivalent to a pure masked convolution autoregressive context model, i.e., all positions in the latent representation y require masked convolution. Therefore, C e This represents the complexity of entropy decoding.

[0131] Decoding complexity C is introduced into the optimization objective. e As a term, the mask can be learned to achieve better adaptability and performance. To obtain this spatial mask M at the decoder end, it can be decoded from the super-prior z. Typically, the super-prior bitstream has a small amount of data, and the spatial mask M is also a small binary data, so the spatial mask M can be included in the super-prior bitstream with negligible increase in data volume.

[0132] The spatial mask M can be binary, and this binary mask can be learned using the Gumbel-softmax reparameterization technique and backpropagation performed. By generating such a learnable spatial mask, the entropy model can adaptively control the decoding complexity through end-to-end optimization. If the optimization objective is related to the term C... e Imposing larger constraints, i.e., having a larger λ in the optimization objective. C If the value is zero, then more positions will be covered by the zero-value mask. Therefore, entropy decoding has a lower time complexity.

[0133] Thus, the hyperparameter λ in the optimization objective can be adjusted. C To finely control the RDC (bitrate-distortion-complexity) tradeoff in neural image compression. In practical applications, λ C The larger the initial value of λ, the greater the degree of control over complexity, meaning the model is more inclined to optimize with the goal of reducing complexity. Conversely, the smaller the initial value, the less complex the model. Therefore, λ can be adjusted by setting... C The initial value is used to flexibly adjust the encoding and decoding complexity of the model.

[0134] The training process of the image encoding and decoding system will be explained next.

[0135] This embodiment can employ a two-stage training strategy, such as... Figure 2F The diagram shown is a flowchart illustrating a training method for an image encoding / decoding system according to an exemplary embodiment of this specification, comprising the following steps:

[0136] In step 202, a training dataset is obtained. The training image samples in the training dataset correspond to spatial mask samples.

[0137] In step 204, the training image samples are input into the initial image coding system to obtain the super-prior bitstream and entropy-coded bitstream of the training image samples.

[0138] In step 206, the prior code stream and entropy-encoded code stream of the image sample are input into the initial image decoding system, and the initial image encoding system and the initial image decoding system are trained to obtain the intermediate image encoding system and the intermediate image decoding system.

[0139] Specifically, after obtaining the spatial mask and the prior representation obtained by the initial image decoding system from decoding the prior bitstream, the initial reconstructed image sample is obtained by decoding the entropy-coded bitstream. Based on the bit rate of the initial reconstructed image sample and the degree of distortion between the pixels of the initial reconstructed image sample and the pixels of the training image sample, the initial image encoding system and the initial image decoding system are trained to obtain the intermediate image encoding system and the intermediate image decoding system.

[0140] In step 208, the training image samples and the corresponding complexity information samples are input into the intermediate image encoding system to train the intermediate image encoding system and the intermediate image decoding system, thereby obtaining the trained image encoding system and image decoding system.

[0141] Specifically, after obtaining the prior code stream and entropy-encoded code stream generated by the intermediate image coding system, the intermediate image decoding system decodes them to obtain intermediate reconstructed image samples. Based on the bit rate of the intermediate reconstructed image samples, the degree of distortion between the pixels of the intermediate reconstructed image samples and the pixels of the training image samples, and the decoding time index of the entropy-encoded code stream, the intermediate image coding system and the intermediate image decoding system are trained to obtain a trained image coding system and image decoding system.

[0142] In the first stage, a pre-trained model that can support arbitrary spatial mask inputs will be optimized. For example, a spatial mask M can be randomly generated for each image sample, covering M% to 100%. j =1. In the first stage, the spatial binary mask M is randomly sampled, rather than decoded from z. The proportion of points with a value of 1 in mask M ranges from 0% to 100%.

[0143] For example, the training dataset contains multiple image samples:

[0144] For image sample 1, a mask M can be randomly generated for this image sample. This mask M contains n ones, meaning that there are n positions in image sample 1 that need to undergo mask convolution. n can be any value of 0 or N, where N is the number of pixels in the image sample. For example, after determining the proportion of the image sample size to the positions in image sample 1 that need to undergo mask convolution, the positions in image sample 1 that need to undergo mask convolution are determined based on this proportion, thus generating the mask information for this image sample. For example, M j =1 indicates that this position requires complex autoregressive mask convolution for decoding, improving compression efficiency; M j =0 indicates that parallel decoding is possible, reducing computational complexity and decoding time. Improving efficiency: In smooth regions of the image (such as the sky), more Mj=0 can be set to reduce unnecessary computation; in complex regions (such as building edges), Mj=0 can be set to... j =1, to ensure image quality.

[0145] Similarly, for other image samples, a mask M can be randomly generated for each image sample. The position and number of 1s in the mask M are random. Finally, the proportion of 1s in the mask M of each image sample in the training dataset needs to cover the range of 0% to 100%.

[0146] Alternatively, for image sample 1, multiple masks M can be randomly generated for the image sample, with different masks M containing different proportions or numbers of 1.

[0147] The training loss in the first phase can be as follows:

[0148]

[0149] Where λ1 is the pixel-wise Lagrange multiplier, which can be adjusted by the spatial mask M. Where x i Let x′ be the i-th pixel in the source image. i Let represent the i-th pixel of the reconstructed image, and N be the number of pixels in the source image. The first term in the loss function, -logP(y|M), represents the aforementioned bit rate.

[0150] By optimizing the objective formula, this embodiment distinguishes the use of masked convolution by measuring the distortion level of each pixel. Through this training process, a pre-trained image coding model that is robust to spatial mask variations can be obtained. That is, the first stage can consider only the bit rate R and distortion rate D, making it effective in the face of various random masks.

[0151] As an example, image samples from the training dataset can be input to... Figure 2C In the image encoding and decoding architecture, the encoding process of the image encoding system in the first stage is similar to the aforementioned process:

[0152] The latent representation of the training image samples is obtained by performing neural network encoding on the training image samples;

[0153] The latent representation of the training image samples is encoded using a super-prior encoding to obtain the super-prior representation;

[0154] Obtain the prior representation of the spatial mask sample corresponding to the training image sample; at this time, use the spatial mask sample corresponding to the training image sample instead of the spatial mask extracted by the conditional transformation module.

[0155] The prior representation containing the spatial mask sample is quantized to obtain the quantized prior representation containing the spatial mask sample.

[0156] The quantized prior representation containing the spatial mask sample is encoded to obtain the prior bitstream;

[0157] Performing super-prior decoding on the super-prior bitstream yields the image edge information represented by the decoded super-prior representation;

[0158] Based on the spatial mask samples corresponding to the training image samples and the image edge information represented by the decoded prior representation, the latent representation is entropy encoded to obtain the entropy encoded bitstream.

[0159] The super-prior bitstream and entropy-encoded bitstream obtained by encoding the training image samples will be decoded. Specifically, the super-prior bitstream is decoded to obtain the super-prior representation and spatial mask M. When decoding the entropy-encoded bitstream, the decoding is performed based on the super-prior representation and spatial mask M to obtain the reconstructed image samples.

[0160] After obtaining the reconstructed image samples, the function value can be calculated based on the aforementioned L1 function, and the parameters of each network in the entire network architecture can be adjusted based on the function value.

[0161] In the second stage, training the intermediate image encoding system and the intermediate image decoding system may include:

[0162] For the intermediate image encoding system, the network parameters other than the encoder used for neural network encoding of the training image samples are adjusted;

[0163] For the intermediate image decoding system, the network parameters other than the decoder used for neural network decoding of the latent representations of the training image samples are adjusted.

[0164] As an example, the main encoder and main decoder can be fixed (i.e., the parameters of the neural network encoder and neural network decoder can be fixed, while the parameters of other networks are adjusted through training) to optimize the generation of the spatial mask. By learning a content-adaptive spatial mask M and jointly optimizing bitrate, distortion, and complexity constraints, a better RDC tradeoff can be achieved. This method can find an optimal balance point for different decoding complexity levels, improving the adaptability and performance of the compression model. The objective function becomes:

[0165]

[0166] Where λ c =F(l), where F is the complexity rank function and l is the rank variable. Compared to the loss function in the first stage, the constraint of the complexity indicator Ce forces the model to determine where to use masked convolutions based on local dependencies in the latent representation at a specific decoding complexity level. After the entire training process, a uniform variable complexity model is obtained.

[0167] In this embodiment, the complexity level l is a parameter used to control the decoding complexity. It determines which positions in the generated spatial mask M require complex autoregressive mask convolution (i.e., M... j=1), which positions can be decoded in parallel (i.e., M) j =0).

[0168] The scheme defines λ c =F(l), where F is a complexity ranking function that maps l to the weights λ in the loss function. c It is necessary to ensure that the design of F(l) effectively reflects the impact of complexity level on the loss function. This is used to balance complexity with other optimization objectives. This is achieved by adjusting λ. c The loss function dynamically balances bit rate R, distortion rate D, and complexity Ce. Different values ​​of l correspond to different values ​​of λ. c This, in turn, affects the model's constraint on complexity during training. For example, a higher l value will increase λ. c This prompts the model to generate more M j Setting l to 0 reduces complexity; lower l values ​​have the opposite effect. In practical applications, the F function can be configured as needed, as long as it reflects the mapping of l to λ. c The mapping relationship can be established. This embodiment does not impose any limitations on this.

[0169] As seen in the above embodiments, the goal of the first stage is to train a base model robust to any spatial mask M. The complexity level l is not explicitly introduced; training is performed using a randomly generated M. The goal of the second stage is to learn how to generate a suitable spatial mask M based on a given complexity level l. In this stage: each training image sample and its corresponding l value can be input into the system. l influences λ. c Further optimization of Ce enables the model to generate a reasonable spatial mask M at a specific complexity level. Ultimately, a unified encoder-decoder can be obtained that can dynamically adjust the decoding complexity according to different l values, eliminating the need to store multiple models for different complexity levels. This allows for flexible control of decoding complexity and achieves a better balance between rate-distortion-complexity (RDC).

[0170] Optionally, the loss function can be adjusted as needed. For example, in principle, the encoder is responsible for generating latent representations to provide a compact representation of the original image. The optimization goal of end-to-end codecs has always been quality for visual perception. However, for compression aimed at machine vision, semantic information is more important. This inconsistency can lead to redundant and computationally inefficient compression performance for machine vision, especially for front-end devices such as cameras and mobile phones, where the encoder is deployed in environments with limited computing power.

[0171] Based on this, the reconstructed image is used for downstream tasks; the training process can be based on the bit rate of the intermediate reconstructed image samples, the degree of distortion between the pixels of the intermediate reconstructed image samples and the pixels of the training image samples, the decoding time index of the entropy-encoded bitstream, and the processing accuracy of the downstream task on the intermediate reconstructed image samples, to train the intermediate image encoding system and the intermediate image decoding system.

[0172] In other words, during training, the losses of downstream tasks (such as image classification, object detection, etc.) can also be considered in the loss function. Its expression can be:

[0173]

[0174] Among them, the task loss L task This represents the processing accuracy of downstream tasks. λ2 is a weighting factor used to control the balance between signal-level distortion and accuracy.

[0175] Therefore, this embodiment proposes a machine-oriented image encoding and decoding scheme that balances bit rate, distortion rate, and complexity. In this embodiment, decoding complexity is quantified as an important factor in the optimization objective, enabling the system to effectively control decoding complexity while ensuring bit rate and distortion rate (i.e., image quality) and the accuracy of downstream tasks. This balancing strategy not only improves the practical application performance of AI encoding and decoding technology but also provides users with the flexibility to choose different complexity balance points according to their needs.

[0176] The core concept of this embodiment is to improve the overall performance of image processing in different tasks through an end-to-end intelligent compression mechanism. Specifically, the system not only focuses on the traditional trade-off between bit rate and signal quality (i.e., bit rate-distortion trade-off), but also incorporates considerations of complexity and the effect on downstream tasks, enabling image compression to achieve better results in a wider range of applications.

[0177] In practical implementation, a variable-complexity neural codec was developed. The codec supports fine-grained complexity adjustment, meaning the system can adaptively control spatial dependency modeling in the context model according to specific needs. This capability allows the system to dynamically adjust complexity when processing different images, thereby achieving the goal of saving computational resources while maintaining high performance.

[0178] To achieve this goal, the complexity of entropy decoding is incorporated into the optimization objective. By quantifying the complexity, the optimization process can precisely control the complexity of the codec. This is the first time that fine-grained control of the rate-distortion-complexity tradeoff has been achieved in a neural network image codec. Traditional image compression methods struggle to maintain signal quality while simultaneously ensuring the performance of the analysis task. However, in the system of this embodiment, this novel entropy decoding complexity control method effectively balances signal reconstruction performance with the requirements of the analysis task.

[0179] Specifically, this embodiment presents a unified machine-oriented variable-complexity image compression model. This model enables fine-grained adjustment of decoding complexity within a single framework, while maintaining distortion rate and downstream task performance. Specifically, during model training, in addition to the conventional losses for image compression and reconstruction, losses for specific downstream tasks (such as image classification and object detection) are added. This approach ensures that critical information required for downstream tasks is preserved during compression, thereby improving the model's performance on these tasks.

[0180] Corresponding to the embodiments of the aforementioned image processing methods, this specification also provides embodiments of the apparatus and the computer equipment on which it is applied.

[0181] The embodiments of the image processing apparatus described in this specification can be applied to computer devices, such as servers or terminal devices. The apparatus embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by its processor reading the corresponding computer program instructions from non-volatile memory into memory and executing them. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of a computer device containing the image processing apparatus described in this specification. (Except for...) Figure 3 In addition to the processor 310, memory 330, network interface 320, and non-volatile memory 340 shown, the computer device in which the image processing apparatus is located in the embodiment may also include other hardware depending on the actual function of the computer device, which will not be described in detail here.

[0182] like Figure 4 The diagram shown is a structural diagram of an image processing apparatus according to an exemplary embodiment of this specification, comprising:

[0183] Image acquisition unit 41 is used to: acquire the source image to be encoded;

[0184] Information acquisition unit 42 is used to: acquire complexity information, wherein the complexity information represents the image processing time of the source image;

[0185] Encoding unit 43 is used to: input the source image into a trained image encoding system for encoding processing, obtain a priori bitstream and an entropy-encoded bitstream, and output them;

[0186] The image coding system is configured to encode the prior representation of the source image into a priori bitstream containing a first spatial mask; and to encode the latent representation of the source image into an entropy-coded bitstream based on a second spatial mask.

[0187] The first spatial mask is extracted from the super-prior representation based on the complexity information and is used to characterize whether each spatial location in the super-prior representation needs to be masked convolution.

[0188] The second spatial mask is extracted from the latent representation based on the complexity information and is used to characterize whether each spatial location in the latent representation needs to be masked convolution.

[0189] In some examples, the image encoding system is used to perform neural network encoding on the source image to obtain a latent representation of the source image, then perform super-prior encoding to obtain a super-prior representation, extract a first spatial mask based on the complexity information to characterize whether each spatial location in the super-prior representation needs to be masked convolution, encode the super-prior representation to obtain a super-prior bitstream containing the first spatial mask; and after extracting a second spatial mask based on the complexity information to characterize whether each spatial location in the latent representation needs to be masked convolution, entropy encoding is performed on the latent representation based on the second spatial mask and the super-prior bitstream to obtain an entropy-encoded bitstream.

[0190] In some examples, the entropy encoding of the latent representation based on the second spatial mask and the super-prior codestream includes:

[0191] Decode the super-prior bitstream to obtain the image edge information represented by the decoded super-prior representation;

[0192] Based on the second spatial mask and the image edge information represented by the decoded hyperprior representation, the latent representation is entropy encoded.

[0193] In some examples, the image coding system includes:

[0194] A neural network encoder is used to perform neural network encoding on the source image to obtain a latent representation of the source image;

[0195] A super-prior encoding model is used to perform super-prior encoding on the latent representation to obtain a super-prior representation;

[0196] The first conditional transformation module is used to extract a first spatial mask based on the complexity information, which represents whether each spatial position in the super-prior representation needs to be masked convolution, and to obtain a super-prior representation containing the first spatial mask.

[0197] The first quantization module is used to quantize the prior representation containing the first spatial mask to obtain the quantized prior representation containing the first spatial mask.

[0198] A first arithmetic encoder is used to encode the quantized super-prior representation carrying the first spatial mask to obtain the super-prior bitstream.

[0199] In some examples, the image coding system includes:

[0200] The second conditional transformation module is used to extract a second spatial mask based on the complexity information, which represents whether each spatial position in the latent representation needs to be masked convolution, and to obtain a latent representation containing the second spatial mask.

[0201] The second quantization module is used to quantize the latent representation containing the second space mask to obtain the quantized latent representation containing the second space mask.

[0202] The super-prior decoding module is used to perform super-prior decoding on the super-prior bitstream to obtain the image edge information represented by the decoded super-prior representation;

[0203] An entropy coding model is used to entropy code the latent representation based on the second spatial mask and the image edge information represented by the decoded super-prior representation, to obtain an entropy-coded bitstream.

[0204] In some examples, the priori decoding module includes:

[0205] A first arithmetic decoder is used to perform arithmetic decoding on the super-prior bitstream to obtain a quantized super-prior representation containing a first spatial mask;

[0206] The third conditional transformation module is used to perform an inverse conditional transformation on the quantized super-prior representation containing the first spatial mask to obtain the first spatial mask and the super-prior representation.

[0207] A super-prior decoding model is used to decode the super-prior representation to obtain the image edge information represented in the super-prior representation.

[0208] In some examples, the first condition transformation module, the second condition transformation module, or the third condition transformation module respectively include:

[0209] The first splicing layer is used to splice the data representations corresponding to the latent representation and the spatial mask to obtain the spliced ​​representation;

[0210] The first convolutional layer is used to perform convolution operations on the latent representation to obtain the convolutional features of the latent representation;

[0211] The second concatenation layer is used to concatenate the convolutional features of the concatenated representation and the latent representation to obtain the concatenated features.

[0212] The first feature fusion layer is used to fuse the spliced ​​features to obtain the first fused feature;

[0213] The activation function layer is used to apply activation functions to the first fusion feature to obtain the activated first fusion feature.

[0214] The second convolutional layer is used to perform convolution operations on the first fused features of the spliced ​​representation and activation to obtain convolutional features;

[0215] The second feature fusion layer is used to fuse the concatenated representation and the convolutional features output by the second convolutional layer to obtain the final fused features.

[0216] In some examples, obtaining complexity information includes any of the following steps:

[0217] Obtain the preset complexity information;

[0218] Obtain complexity information provided by the user;

[0219] The complexity information is determined based on the resource information of the device where the image encoding system is located; or,

[0220] Complexity information is determined based on the downstream tasks used to process the hyperprior and entropy-coded bitstreams.

[0221] like Figure 5 The diagram shown is a structural diagram of another image processing apparatus according to an exemplary embodiment of this specification, the apparatus comprising:

[0222] The acquisition unit 51 is used to: acquire the prior code stream and the entropy-coded code stream; the prior code stream and the entropy-coded code stream are obtained through the foregoing embodiments;

[0223] The input unit 52 is configured to: input the super-prior code stream and the entropy-coded code stream to the image decoding system, wherein the image decoding system is configured to decode the super-prior representation and spatial mask from the super-prior code stream, and decode the entropy-coded code stream based on the super-prior representation and spatial mask to obtain the reconstructed image.

[0224] In some examples, decoding the entropy-coded bitstream based on the prior representation and spatial mask includes:

[0225] Determine the spatial locations in the spatial mask that do not require mask convolution, and perform parallel decoding on each determined spatial location based on the prior representation.

[0226] In some examples, the image decoding system includes:

[0227] A priori decoding module is used to decode the priori representation and spatial mask from the priori bitstream;

[0228] The entropy decoding model, based on the aforementioned prior representation and spatial mask, performs entropy decoding on the entropy-coded bitstream to obtain a quantized latent representation;

[0229] The conditional transformation module is used to perform inverse conditional transformation on the quantized latent representation to obtain the spatial mask and the latent representation.

[0230] A neural network decoder is used to decode the reconstructed image based on the latent representation.

[0231] like Figure 6 The diagram shown is a structural diagram of a training apparatus for an image encoding / decoding system according to an exemplary embodiment of this specification, comprising:

[0232] Acquisition unit 61 acquires a training dataset; wherein, the training image samples in the training dataset correspond to spatial mask samples;

[0233] Input unit 62 inputs the training image samples into the initial image coding system to obtain the super-prior bitstream and entropy-coded bitstream of the training image samples;

[0234] The first training unit 63 inputs the super-prior code stream and entropy-coded code stream of the image sample into the initial image decoding system, obtains the spatial mask and super-prior representation obtained by the initial image decoding system decoding the super-prior code stream, and then decodes the entropy-coded code stream to obtain the initial reconstructed image sample. Based on the bit rate of the initial reconstructed image sample and the degree of distortion between the pixels of the initial reconstructed image sample and the pixels of the training image sample, the initial image encoding system and the initial image decoding system are trained to obtain the intermediate image encoding system and the intermediate image decoding system.

[0235] The second training unit 64 inputs the training image samples and corresponding complexity information samples into the intermediate image encoding system. After obtaining the super-prior bitstream and entropy-encoded bitstream generated by the intermediate image encoding system, the intermediate image decoding system decodes them to obtain intermediate reconstructed image samples. Based on the bit rate of the intermediate reconstructed image samples, the degree of distortion between the pixels of the intermediate reconstructed image samples and the pixels of the training image samples, and the decoding time index of the entropy-encoded bitstream, the intermediate image encoding system and the intermediate image decoding system are trained to obtain a trained image encoding system and image decoding system.

[0236] In some examples, the decoding time metric of the entropy-encoded bitstream is determined based on the number of spatial locations in the spatial mask samples of the training image samples that need to be masked and convolved, as well as the weights corresponding to the complexity information samples; the decoding time metric is positively correlated with the number of locations and also positively correlated with the image processing time represented by the complexity information samples.

[0237] In some examples, training the intermediate image encoding system and the intermediate image decoding system includes:

[0238] For the intermediate image encoding system, the network parameters other than the encoder used for neural network encoding of the training image samples are adjusted;

[0239] For the intermediate image decoding system, the network parameters other than the decoder used for neural network decoding of the latent representations of the training image samples are adjusted.

[0240] In some examples, the reconstructed image is used for downstream tasks; training the intermediate image encoding system and the intermediate image decoding system based on the bit rate of the intermediate reconstructed image samples, the degree of distortion between the pixels of the intermediate reconstructed image samples and the pixels of the training image samples, and the decoding time metric of the entropy-coded bitstream, includes:

[0241] The intermediate image encoding system and the intermediate image decoding system are trained based on the bit rate of the intermediate reconstructed image samples, the degree of distortion between the pixels of the intermediate reconstructed image samples and the pixels of the training image samples, the decoding time index of the entropy encoded bitstream, and the processing accuracy of the downstream task on the intermediate reconstructed image samples.

[0242] In some examples, obtaining the training dataset includes:

[0243] After obtaining training image samples and determining the proportion of the training image sample size to the positions where mask convolution needs to be performed, the positions where mask convolution needs to be performed in the training image sample are determined based on the proportion, and a spatial mask for the training image sample is generated.

[0244] A training dataset is obtained based on the spatial mask of each training image sample; wherein, the proportion of the training image sample size to be masked and convolved in each training image sample of the training dataset ranges from 0% to 100%.

[0245] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0246] Accordingly, this specification also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned image processing method embodiments.

[0247] Accordingly, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the image processing method embodiments.

[0248] Accordingly, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image processing method embodiments.

[0249] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0250] The above embodiments can be applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0251] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0252] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0253] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0254] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0255] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0256] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0257] The terms "specific example" or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with the embodiments or examples, which are included in at least one embodiment or example of this specification. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0258] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0259] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0260] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. An image processing method, the method comprising: Obtain the source image to be encoded; Obtain complexity information, which represents the image processing time for the source image; The source image is input into a trained image coding system for encoding processing to obtain a priori bitstream and an entropy-coded bitstream, which are then output. The image coding system is used to encode the prior representation of the source image into a priori bitstream containing a first spatial mask. Furthermore, the latent representation of the source image is encoded into an entropy-coded bitstream based on a second spatial mask; The first spatial mask is extracted from the super-prior representation based on the complexity information and is used to characterize whether each spatial location in the super-prior representation needs to be masked convolution. The second spatial mask is extracted from the latent representation based on the complexity information and is used to characterize whether each spatial location in the latent representation needs to be masked convolution.

2. The method according to claim 1, wherein the image encoding system is specifically used for: After performing neural network encoding on the source image to obtain its latent representation, super-prior encoding is performed to obtain a super-prior representation. Based on the complexity information, a first spatial mask characterizing whether each spatial location in the super-prior representation needs to undergo masked convolution is extracted. The super-prior representation is then encoded to obtain a super-prior bitstream containing the first spatial mask; and... After extracting a second spatial mask based on the complexity information to characterize whether each spatial location in the latent representation needs to be masked convolution, the latent representation is entropy encoded based on the second spatial mask and the super-prior bitstream to obtain an entropy-coded bitstream.

3. The method according to claim 2, wherein the entropy coding of the latent representation based on the second spatial mask and the super-prior codestream comprises: Decode the super-prior bitstream to obtain the image edge information represented by the decoded super-prior representation; Based on the second spatial mask and the image edge information, the latent representation is entropy encoded.

4. The method according to claim 2, wherein the image encoding system comprises: A neural network encoder is used to perform neural network encoding on the source image to obtain a latent representation of the source image; A super-prior encoding model is used to perform super-prior encoding on the latent representation to obtain a super-prior representation; The first conditional transformation module is used to extract a first spatial mask based on the complexity information, which represents whether each spatial position in the super-prior representation needs to be masked convolution, and to obtain a super-prior representation containing the first spatial mask. The first quantization module is used to quantize the prior representation containing the first spatial mask to obtain the quantized prior representation containing the first spatial mask. A first arithmetic encoder is used to encode the quantized super-prior representation carrying the first spatial mask to obtain the super-prior bitstream.

5. The method according to any one of claims 1 to 4, wherein the image coding system comprises: The second conditional transformation module is used to extract a second spatial mask based on the complexity information, which represents whether each spatial position in the latent representation needs to be masked convolution, and to obtain a latent representation containing the second spatial mask. The second quantization module is used to quantize the latent representation containing the second space mask to obtain the quantized latent representation containing the second space mask. The super-prior decoding module is used to perform super-prior decoding on the super-prior bitstream to obtain the image edge information represented by the decoded super-prior representation; An entropy coding model is used to entropy code the latent representation based on the second spatial mask and the image edge information represented by the decoded super-prior representation, to obtain an entropy-coded bitstream.

6. The method according to claim 5, wherein the advanced prior decoding module comprises: A first arithmetic decoder is used to perform arithmetic decoding on the super-prior bitstream to obtain a quantized super-prior representation containing a first spatial mask; The third conditional transformation module is used to perform an inverse conditional transformation on the quantized super-prior representation containing the first spatial mask to obtain the first spatial mask and the super-prior representation. A super-prior decoding model is used to decode the super-prior representation to obtain the image edge information represented in the super-prior representation.

7. The method according to claim 6, wherein the first condition transformation module, the second condition transformation module, or the third condition transformation module each comprises: The first splicing layer is used to splice the data representations corresponding to the latent representation and the spatial mask to obtain the spliced ​​representation; The first convolutional layer is used to perform convolution operations on the latent representation to obtain the convolutional features of the latent representation; The second concatenation layer is used to concatenate the convolutional features of the concatenated representation and the latent representation to obtain the concatenated features. The first feature fusion layer is used to fuse the spliced ​​features to obtain the first fused feature; An activation function layer is used to apply an activation function to the first fusion feature to obtain the activated first fusion feature; The second convolutional layer is used to perform convolution operations on the first fused features of the spliced ​​representation and activation to obtain convolutional features; The second feature fusion layer is used to fuse the concatenated representation and the convolutional features output by the second convolutional layer to obtain the final fused features.

8. The method according to claim 1, wherein obtaining the complexity information includes any one of the following steps: Obtain the preset complexity information; Obtain complexity information provided by the user; The complexity information is determined based on the resource information of the device where the image encoding system is located; or, Complexity information is determined based on downstream tasks that process the aforementioned priori bitstream and entropy-coded bitstream.

9. A training method for an image encoding / decoding system, the method comprising: Obtain a training dataset; wherein, the training image samples in the training dataset correspond to spatial mask samples; The training image samples are input into the initial image coding system to obtain the super-prior bitstream and entropy-coded bitstream of the training image samples; The super-prior bitstream and entropy-encoded bitstream of the image sample are input into the initial image decoding system. After the initial image decoding system decodes the super-prior bitstream, it obtains the spatial mask and super-prior representation. Then, it decodes the entropy-encoded bitstream to obtain the initial reconstructed image sample. Based on the bit rate of the initial reconstructed image sample and the degree of distortion between the pixels of the initial reconstructed image sample and the pixels of the training image sample, the initial image encoding system and the initial image decoding system are trained to obtain the intermediate image encoding system and the intermediate image decoding system. The training image samples and corresponding complexity information samples are input into the intermediate image encoding system to obtain the super-prior bitstream and entropy-encoded bitstream generated by the intermediate image encoding system. Then, the intermediate image decoding system decodes the intermediate images to obtain intermediate reconstructed image samples. Based on the bit rate of the intermediate reconstructed image samples, the degree of distortion between the pixels of the intermediate reconstructed image samples and the pixels of the training image samples, and the decoding time index of the entropy-encoded bitstream, the intermediate image encoding system and the intermediate image decoding system are trained to obtain a trained image encoding system and an image decoding system.

10. The method according to claim 9, wherein the decoding time index of the entropy-encoded bitstream is determined based on the number of spatial locations in the spatial mask samples of the training image samples that need to be masked and convolved, and the weights corresponding to the complexity information samples; the decoding time index is positively correlated with the number and also positively correlated with the image processing time represented by the complexity information samples.

11. The method according to claim 9, wherein training the intermediate image encoding system and the intermediate image decoding system comprises: For the intermediate image encoding system, the network parameters other than the encoder used for neural network encoding of the training image samples are adjusted; For the intermediate image decoding system, the network parameters other than the decoder used for neural network decoding of the latent representations of the training image samples are adjusted.

12. The method according to claim 9, wherein the reconstructed image is used for a downstream task; the step of training the intermediate image encoding system and the intermediate image decoding system based on the bit rate of the intermediate reconstructed image samples, the degree of distortion between the pixels of the intermediate reconstructed image samples and the pixels of the training image samples, and the decoding time index of the entropy-coded bitstream, includes: The intermediate image encoding system and the intermediate image decoding system are trained based on the bit rate of the intermediate reconstructed image samples, the degree of distortion between the pixels of the intermediate reconstructed image samples and the pixels of the training image samples, the decoding time index of the entropy encoded bitstream, and the processing accuracy of the downstream task on the intermediate reconstructed image samples.

13. The method according to claim 9, wherein obtaining the training dataset comprises: After obtaining training image samples and determining the proportion of the training image sample size to the positions where mask convolution needs to be performed, the positions where mask convolution needs to be performed in the training image sample are determined based on the proportion, and a spatial mask for the training image sample is generated. A training dataset is obtained based on the spatial mask of each training image sample; wherein, the proportion of the training image sample size to be masked and convolved in each training image sample of the training dataset ranges from 0% to 100%.

14. An image processing method, the method comprising: Obtain the priori bitstream and the entropy-encoded bitstream; The super-prior code stream and the entropy-coded code stream are obtained through the steps of the method described in any one of claims 1 to 8; The super-prior bitstream and the entropy-coded bitstream are input into an image decoding system. The image decoding system is used to decode the super-prior representation and spatial mask from the super-prior bitstream, and decode the entropy-coded bitstream based on the super-prior representation and spatial mask to obtain the reconstructed image.

15. The method according to claim 14, wherein decoding the entropy-coded bitstream based on the prior representation and spatial mask comprises: Determine the spatial locations in the spatial mask that do not require mask convolution, and perform parallel decoding on each determined spatial location based on the prior representation.

16. The method according to claim 14, wherein the image decoding system comprises: A priori decoding module is used to decode the priori representation and spatial mask from the priori bitstream; The entropy decoding model, based on the aforementioned prior representation and spatial mask, performs entropy decoding on the entropy-coded bitstream to obtain a quantized latent representation; The conditional transformation module is used to perform inverse conditional transformation on the quantized latent representation to obtain the spatial mask and the latent representation. A neural network decoder is used to decode the reconstructed image based on the latent representation.

17. An image processing system, the image processing system comprising an image encoding end and an image decoding end, the image encoding end being configured to perform the steps of the method according to any one of claims 1 to 8, and the image decoding end being configured to perform the steps of the method according to any one of claims 14 to 16.

18. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 16.

19. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method of any one of claims 1 to 16.

20. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any one of claims 1 to 16.

Images